Selection and optimization of machine learning notes (six) machine learning algorithms

1, when the processing method when a large deviation of the predicted and actual results:

(1) increase the training sample;

(2) prevent overfitting reduced feature set;

(3) increasing the characteristic feature set or a polynomial (e.g. X _{. 1} ², X ₂ ³, etc.);

(4) decrease / increase lambda.

2, the evaluation function is assumed:

The data set is divided into two parts: the training set (70%) and test set (30%)

specific process:

(1) [theta] is obtained through learning, training error calculating J (θ);

(2) calculating a test error J _Test ([theta])

① For linear regression:

② For logistic regression:

Misclassification error misclassification rate (0/1 misclassification rate):

Test error:

3, model selection:

Consider the following 10 candidate models:

(1) The data set is divided into three parts: the training set (60%), cross-validation set (cross validation set, the CV short, 20%) and test set (20%).

(2) adding a parameter d, indicates the number of polynomials in turn values d = 1,2, ..., 10.

(3) the model training sequence, to obtain [theta] of each model, referred to as [theta] ^{(. 1)} , [theta] ⁽²⁾ , ..., [theta] ⁽¹⁰⁾

(4) The cross-validation set J is calculated _{the CV} ([theta] ^{(. 1)} ), J _{the CV} ([theta] ⁽²⁾ ), ..., J _{the CV} ([theta] ⁽¹⁰⁾ ). Select the model with the least error.

(5) stars d = k (where k th model selected model), the error is calculated using a verification test set, the model evaluation.

4, bias and variance issues:

(1) Example:

Pic.1 underfitting (high bias) d = 1 Pic.2 fit d = 2 Pic.3 overfitting (high variance) d = 4

When d is small, high Deviation: J _Train ([theta]) high, J _{the CV} ([theta]) ≈ J _Test ([theta]) ≈ J _Train ([theta]).

When d is large, high Variance: J _Train ([theta]) is low, J _{the CV} ([theta]) ≈ J _Test ([theta]) >> J _Train ([theta]).

(2) assumed to have been elected d = 4, regularization is introduced to solve the problem of over-fitting:

For λ is too small, there remains overfitting; for λ is too large, it tends to assume a straight line function, underfitting.

① Set λ = 0,0.1,0.2,0.4, ..., 10.24 (last group can be set to 10, a total of 12 groups), were fitted to obtain a different [theta], respectively, corresponding to [theta] ^{(. 1)} , [theta] ⁽²⁾ , ..., [theta] ^{(12 is)} .

② be evaluated by cross-validation set, selects the best evaluation results λ.

5, the learning curve:

(1) a high deviation (less fit):

With the increase in the training set, fit the deviation is still large, the training and validation sets the error difference is small, and tends to level. No amount of explanation to collect data for the result of little help.

The following figure: With the increase in the training set, a fitting result is still straight, an error is still large.