Machine learning basic (Basic "Recipe" for Machine Learning)
The lesson we are talking about is how to determine whether the algorithm deviation or variance is high, help us to be more systematic application of these methods in machine learning algorithms to optimize performance through training and validation set error error.
The figure is my basic approach used in training the neural network :( Try these methods may be useful, it may be useless)
This is the basic method to use when I train the neural network model after the initial training is completed, I first want to know the deviation algorithm Gaobu Gao, if the deviation is higher, try to evaluate the performance of the training set or training data. If the deviation is high indeed, not even fit the training set, then you have to do is to choose a new network, for example, contains more hidden layers or hide the network elements, or take more time to train the network, or try the more advanced optimization algorithms, we will talk about later in this section. You can also try other methods may be useful, it may be useless.
While we will see many different neural network architectures, maybe you can find a more suitable solution to this problem a new network architecture, parentheses, because one of which is that you have to try, it may be useful, it may be useless, but usually using larger network will help extend the training time is not necessarily useful, but it does not hurt. When training learning algorithm, I will keep trying these methods until the deviation rid of the problem, which is the minimum standard, repeated attempts until you can fit the data, at least be able to fit the training set.
If the network is large enough, it can often be a good fit the training set, as long as you can expand the size of the network, if the picture is fuzzy, the algorithm may not fit the picture, but if someone can tell the picture, if you feel not very basic error high, then train a larger network, you should be able to be a good fit at least ...... training set, or at least can be fitted over-fitting the training set. Once the deviation is reduced to an acceptable value, check the variance there is no problem, in order to assess the variance, we want to see performance validation set, we can learn from a training set performance ideal infer verify performance set also ideal, if the variance is high, the best solution is to use more data, if you can do it, there will be some help, but sometimes we can not get more data, we can also try to reduce the regularization of over-fitting, at this we lesson speaks. Sometimes we have to try again, but if you can find a more appropriate framework neural networks, sometimes it might kill two birds with one stone, while reducing variance and bias. How to achieve it? The system would like to say the practice is difficult, in short, is constantly repeated attempts until you find a low-skew, low-variance framework, then you will be successful.
There are two points to your attention:
First, high bias and high variance are two different cases, the method we follow to try may be completely different, I usually use the validation set training algorithm to diagnose whether there is a deviation or variance problem, then try to select the part based on the results method. For example, if the algorithm is the presence of high deviation problem, in fact, ready for more training data is useless, at least this is not a more effective way, we have to clear the problem is bias or variance, or both questions, it clearly that will help us select the most effective method.
Second, in the early stages of machine learning, discussion of so-called bias variance trade-off is not uncommon, because there are many ways we can try. Can increase the deviation, variance reduction can also reduce the deviation, variance increases, but in the early stages of the depth of learning, there is not much we can do only a tool to reduce the deviation or variance does not affect the other. But in the current era of big data and depth of learning, as long as the continuous training of a larger network, as long as more data is ready, it is not only these two cases, we assume that is the case, then, as long as regular moderate, usually to build a greater network will be able, at the same time reduce the deviation does not affect the variance, while the use of more data can often reduce the variance at the same time affecting too much bias. This two-step practical work to be done is: to train the network, select the network or to prepare more data, we now have the tools can be done to reduce the deviation or variance, while the other does not produce too many adverse effects. I think this is an important reason for deep learning supervised learning of great benefit, but also we do not have too much focus on one important reason for how to balance the bias and variance, but sometimes we have many options to reduce deviation or variance without increasing the other . Eventually, we will get a very standardized network. From the beginning of the next class, we will explain regularization, training a larger network virtually no negative impact, but mainly the cost of training a large-scale neural network computation time only, provided that the network is more standardized.
Today we talk about the basics of how to diagnose bias and variance of machine learning through the organization, and then select the correct action to solve the problem, I hope you understand and know. I mentioned more than once in the course of regularization, it is a very practical method to reduce the variance, variance trade-offs go awry when regularization, the deviation may increase slightly, if the network is large enough, the increase is usually not too high we next lesson and then go into detail, so that we better understand how to implement neural network regularization.
Course PPT
