Training / development / test sets (Train / Dev / Test Sets)
You may already know, so this week, we will continue to learn how effective functioning of the neural network, covering over parameter tuning, how to build data, and how to ensure optimization algorithms running quickly, so that learning self-learning algorithm to complete within a reasonable time .
The first week, we first talk about the problems in machine learning neural network, and then random neural networks, but also learn some neural network operation to ensure that the right skills, with these questions, we begin today's lesson.
Make the right decisions in configuring training, validation and test data sets will help you create efficient neural network to a large extent. When training the neural network, we need to make many decisions, such as:
- How many layers of artificial neural network
- Each containing a number of hidden units
- What is the learning rate
- Which activation function using layers
Creating a new application, it is impossible for us to predict these super parameters and other information from the outset. In fact, application of machine learning is a highly iterative process, usually at the start of the project, we will first have a preliminary idea, such as building contains a specific number of layers, the number of hidden units or the number of data sets and so on neural networks, then coding, and try to run the code, run the test and get the results of these neural network configuration or run through, you may re-refine their ideas based on output, change tactics, or to find a better neural networks continue iteration update their program.
Now, already in deep learning natural language processing, in many areas of computer vision, speech recognition, and structured data application and achieved great success. Structured data is all-encompassing, from advertising to the search network. Which includes not only Web search Web search engine, also includes shopping sites, according to the search bar entries from all transmission result of the site. And then to computer security, logistics, such as determining the driver where to take delivery, wide range, the list goes on.
I have found that there may be people in natural language processing would like to set foot in the field of computer vision, speech recognition or experienced experts want to join the advertising industry, or, some people want to jump to the logistics industry from the computer security field, in my opinion, from a field or fields of application have to be intuitive experience, usually can not be transferred to other applications, the best decision on how much data you have, the number of input features of computer configurations, with GPU training or CPU , GPU and CPU 's specific configuration and many other factors.
So far, I think, for many applications, even seasoned connoisseur of deep learning is unlikely to start pre-set parameters that best matches the super, so that the depth of learning is a typical application of an iterative process that requires more than times the cycle in order to find a Heart for the application of neural networks, the efficiency of the recycling process is a key factor in determining the rate of progress the project, and create high-quality training data set, validation and test sets also help to improve circulation effectiveness.
It is assumed that the training data I represented by a rectangle, we usually use these data into parts, as part of the training set, as part of the simple cross-validation set, sometimes called validation set, for convenience, I call it validation set ( dev the sET ), are in fact the same concept, as the last part of the test set.
Next, we started training algorithm, by a simple cross-validation validation set or set to select the best model, well-proven, we selected the final model, then it can be evaluated in the test set, for unbiased assessment the health of the algorithm.
In machine learning and development of small data era, the common practice is to put all the data three seven, is often said that 70% of the validation set, 30% of the test set, if not explicitly set the validation set, can follow the 60% training, 20 % and 20% of the test set validation divided. This is the best machine learning practices generally accepted in previous years.
If only 100, 1000 or 10,000 data, then divide the above ratio is very reasonable.
But in the era of big data, the amount of data we may be one million level, then the validation set and test set percentage of the total amount of data will tend to become smaller. Because the purpose of verification is to verify different set of algorithms, which algorithm test is more powerful, and therefore, the validation set to be large enough to evaluate, such as two or even 10 different algorithms, and quickly determine which algorithm is more effective. We may not need to devote 20% of the data as a validation set.
For example, we have one million data, then take 10000 data sufficient to assess identify best performing algorithms 1-2. Similarly, according to the final selection of the classifier, the main purpose of the test set is correctly assess classifier performance, so if you have a million data points we need only 1000 data, would be sufficient to evaluate individual classifier, and an accurate assessment of the classification performance's. Suppose we have 100 million data, wherein the validation set as 10000, 10000 as the test set, taking 100 Miles 10,000, the ratio is 1%, that is: 98% training set, validation and test sets each accounted 1 %. For the amount of data than a million applications, the training set can be accounted for 99.5%, validation and test sets each accounted for 0.25%, or 0.4% validation set, the test set 0.1%.
To summarize, in machine learning, we usually divide the sample into training set, validation and test sets of three parts, the relatively small size of the data set for the traditional division ratio, the larger the data set, validation set and test set to less than 20% or 10% of the total data. Later I will give specific guidance on how to divide the validation set and test set.
Another trend in modern depth study is that more and more people in the training and test sets match the distribution without training, if you want to build a large number of users can upload pictures of application, the purpose is to identify and present all cats picture, may your users are cat lovers, training sets may be downloaded from the Internet cat pictures, and validation and test sets are uploaded by the user on the application of cat pictures, that is, the training set could be from the network caught down on the picture. The validation and test sets are user uploaded images. The results cat many high resolution images on web pages, very professional, post-production well, and users upload pictures may be taken using a mobile phone casual, low-resolution, rather vague, these two types of data differently for this situation , based on experience, I suggest that you make sure to verify and test sets of data from the same distribution, on this issue I will speak more. Because you use the test set to evaluate the different models to optimize performance as much as possible. If the verification and test sets from the same distribution will be very good.
However, due to the depth of the learning algorithm requires a lot of training data, in order to obtain larger training data set, we can use the popular variety of creative strategies, such as web crawl, the price is set training and validation data set and test data set there may not be from the same distribution. But as long as follow this rule of thumb, you will find a machine learning algorithm becomes faster. I will explain this rule of thumb in more detail in a later lesson.
Finally, even if there is no test set does not matter, the purpose of the test is the final set of the selected neural network system to make unbiased estimation, if not unbiased estimates may not be provided the test set. So if only validation set, there is no test set, we have to do is, in the training on the training set, try a different model framework for assessing these models in the validation set, then iterate and selecting the applicable model. Because the verification test already covered centralized data set, it can no longer provide unbiased performance evaluation. Of course, if you do not need an unbiased estimate, that's fine.
In machine learning, if only a training set and a validation set, and no independent test set, in which case, the training set is also known as the training set and validation set is called the test set, but in practical applications, it is just as simple cross-validation test set set used, does not fully implement the functions of the term, because they are fitted over the data set to verify a test set. If a team tell you they just set up a training set and a test set, I will be very cautious, thinking that they are not really training validation set, because they set the validation data over-fitting to the test set, so that these change the name for the team, renamed it as "training validation set" rather than "training test set", it may not be easy. Even if I think "Training validation set" is more accurate in professional terms. In fact, if you do not need an unbiased assessment of the performance of the algorithm, then this is possible.
So, build a training validation and test sets to accelerate the integration of neural networks can be more effectively measure the bias and variance algorithm to help us more efficiently select the appropriate method to optimize the algorithm.
Course PPT
