Machine Learning Course 4 Selection of Model

Course 4

Average Error on testing data

Two main errors:

Error due to bias
Error due to variance

Estimator

Only god knows about the best function
$$\hat{f}$$
and we can only get a function from training data called
$$f^{\ast }$$
we say:
$$f^{\ast }isanestimatorof\hat{f}$$
and the difference between f^* and f^hat comes from bias and variance

Bias and Variance of Estimator

Suppose the mean of a variable is μ, and the variance of x is σ²
S a m p l e N p o i n t s : { x 1 , x 2 , . . . , x 3 } Sample \quad N \quad points:\{x^1,x^2,...,x^3\} SampleNpoints:{ x1,x2,...,x3}

$$m=\frac{1}{N}\sum _n{x}^n\ne \mu s^2=\frac{1}{N}\sum _n(x^n-m{)}^2$$

$$E[m]=E[\frac{1}{N}\sum _n{x}^n]=\frac{1}{N}\sum _{n}E[x^n]=\mu E[s^2]=\frac{N-1}{N}{\sigma }^2$$

m is a biased estimator of μ, s² is a biased estimator of σ²
$$Var[m]=\frac{{\sigma }^2}{N}$$
which shows how much m deviates μ and the variance depends on the amount of sample

and the relationship of these parameters is below:

Simple model with small variance, and complicated model with large variance since simpler model is less likely to be influenced by the sampled data.

Diagnosis

• If your model cannot even fit the training data, then you got a large bias, it is Underfitting
• If you can fit training data but got large error on testing data, then you probably got a large variance, it is Overfitting

For bias, redesign your model:

• Add more features as input
• A more complex model maybe needed

For large variance:

• More data is needed(Very effective but not always practical)
• Regularization

Model Selection

There is usually a trade-off between bias and variance

Select a model that balances two kinds of error to minimize total error

Cross Validation could a possible way to make balance:

And an advanced method called N-fold Cross Validation can be used