Chapter 2: Model Evaluation and Selection

Empirical error and generalization error

Overfitting is the core problem.
Overfitting is unavoidable and can only be mitigated as much as possible.
Every algorithm has to think about how to alleviate overfitting ? When do mitigations fail ?

model selection

assessment method

set aside method

Why can't it be too small?
Because Mx is used to approximate the result of M100,
why can't it be too large?
A big problem with the remaining data used to approximate the err100

hold-out method is that even if 100 random divisions are performed, there are still some data that have not been trained and some that have not been tested.

cross-validation method

It can solve the problems existing in the reservation method.
Randomly divide 10 times, called 10 times 10-fold cross-validation.

self-help

Preferences

The parameters of the model cannot be tuned directly with the test set.

performance metrics

PR curve
sorts all the test samples according to the probability of being a positive example from high to low, the ordinate is the precision rate, and the abscissa is the recall rate to draw a curve.
Generally, if the curve of A completely wraps B, then we think that A is better than B.
The balance point of the BEP
PR curve.
The ROC curve
is similar to the PR curve, but the abscissa is replaced with a false positive example, and the ordinate is replaced with a true example.
$$real\; example=\frac{TP}{TP+FN}$$
$$false\; positive=\frac{FP}{TN+FP}$$
The area cost-sensitive error rate
under the AUC ROC curve The FP and FN of the confusion matrix contain weights. Cost curve drawing: calculate FNR for each point (TPR, FPR) on the ROC curve, draw a line segment from (0, FPR) to (1, FNR), and the area under the line segment represents the expected population under this condition As the cost, just take the area of ​​the lower bound of all line segments.

comparative test