Course 3: Structured Machine Learning Project (Week 1) - Machine Learning Strategies (1)

1. Machine Learning Strategies

How to improve the performance of the model?

• collect more data
• Training set diversity (eg, identifying cats, collecting cats in various poses, and counterexamples)
• longer training time
• Try different optimization algorithms (like Adam optimization)
• Try a larger/smaller neural network
• Try DropOut regularization
• Try adding L2 regularization
• Design a new network structure (modify the activation function, the number of hidden units)

All of the above methods can change the performance of the model, but it is necessary to judge which ones are effective and which ones can be safely discarded .

2. Orthogonalization

It is best to have no coupling relationship between the adjusted variables.

Locate the performance bottleneck of the model and use the corresponding method to improve it.

Early stopping is a less orthogonal method.
Stopping prematurely affects the accuracy of the training set. At the same time, it can improve the accuracy of the development set.
It affects two things at the same time. Try to use other orthogonal control methods.

3. Single Numerical Evaluation Metrics

Precision , recall, F1 value (average of the first two) :
$$F1=\frac{2}{\frac{1}{\text{ precision }}+\frac{1}{\text{ recall }}}=2\ast \frac{\text{ precison }\ast \text{ recall }}{\text{ precison }+\text{ recall }}=\frac{TP}{TP+\frac{FN+FP}{2}}$$

Having a single real number evaluation metric can improve the efficiency of your decision making.

4. Meet and optimize metrics

Considering N metrics, sometimes it is reasonable to choose one of them as the optimization metric .

Try to optimize that metric, and then the remaining N-1 metrics are all satisfying metrics , which means that as long as they reach a certain threshold, you no longer care about the size of the metric within the threshold.

5. Train/dev/test set division

Example: The data of the first 4 regions is used as the development set, and the last 4 regions are used as the test set

Very bad , they most likely come from different distributions ; all data
should be shuffled randomly and repartitioned.

6. Size of Dev and Test Sets

7. When to change dev/test sets and metrics

Change the test index:
$$\begin{array}{rl}& \text{ error }:\frac{1}{\sum w^{(i)}}\sum _{i=1}^{m_{\text{dev }}}{w}^{(i)}\mathcal{L}\left\{\left({\hat{y}}^{(i)}\ne y^{(i)}\right)\right\}\\ & \\ & w^{\left(i\right)}=\{\begin{array}{cl}1& \text{ if }{x}^{\left(i\right)}\text{is a non-pornographic image}\\ 10& \text{ if }{x}^{\left(i\right)}\text{is an erotic}\end{array}\end{array}$$
For the above method, you have to go through the data yourself and mark the pornographic images.

---

Another example: your development/test set are all clear professional pictures, and the final launch of the application is for unprofessional pictures (blur, bad angle, etc.), then you need to change the development/test set and add unprofessional pictures as training data.

8. Human Performance Levels

It is natural to compare the level of machine learning to that of humans. We want machines to do better than humans

For tasks that humans are good at, when machine learning algorithms are worse than humans , people can help you label the data, so that more data can be fed to the learning algorithm to improve the algorithm.

9. Bias can be avoided

10. Understand human performance

11. Beyond human performance

Scenario B: Beyond the 0.5% threshold (lower error than the best doctors), there are no clear options and way forward to further optimize your machine learning problem

12. Improve the performance of your model

Summarize:

The above method is an improved idea of ​​orthogonalization.

• Gap between training set error and Bayesian estimation error: avoidable bias
• Gap between training set error and dev set error: variance

---

Improve Bias:

• Larger scale model
• Longer training and more iterations
• Better optimization algorithms (Momentum, RMSprop, Adam)
• Better new neural network architecture
• better hyperparameters
• Change the activation function, the number of network layers, the number of hidden units
• Other models (recurrent NN, convolutional NN)

---

Improved variance:

• Collect more data to train
• Regularization (L2 regularization, dropout regularization, data augmentation)
• Better new neural network architecture
• better hyperparameters