1.8 Why is human performance-Deep Learning Lesson 3 "Structured Machine Learning Project" -Professor Stanford Wu Enda

Why is human performance (Why human-level performance?)

In the past few years, more machine learning teams have been discussing how to compare the performance of machine learning systems and humans. Why?

I think there are two main reasons. First, because of the advancement of deep learning systems, machine learning algorithms have suddenly become better. In many application areas of machine learning, algorithms have begun to be seen to threaten human performance. Secondly, it turns out that when you try to make machines do what humans can do, you can carefully design the workflow of the machine learning system to make the workflow more efficient, so in these situations, it is natural to compare humans and machines, or You want the machine to imitate human behavior.

Let ’s take a look at a few such examples. I see that in many machine learning tasks, when you spend a lot of time on a problem, so $x$ axis is time, which may be many months or even many years. During these times, some teams or some research groups are studying a problem. When you start to work towards the human level, the progress is very fast. But after a while, when this algorithm performs better than humans, the progress and accuracy become slower. Maybe it will get better and better, but after surpassing the human level, it can still get better, but the slope of the rate of performance increase and accuracy increase will become more and more gentle, we all hope to achieve Theoretical optimal performance level. Over time, when you continue to train the algorithm, the model may be larger and more data, but the performance cannot exceed a certain theoretical upper limit, which is called the Bayes optimal error rate (Bayes optimal errorrate)). Therefore, the Bayesian optimal error rate is generally considered to be the theoretically possible optimal error rate, that is, there is no way to design a $x$ to $and$The function of$y$ allows it to exceed a certain accuracy.

For example, for speech recognition, if $x$ is an audio clip, some audio is so noisy, it is basically impossible to know what is said, so the perfect accuracy may not be 100%. Or for cat image recognition, maybe some images are very blurry, whether it is a human or a machine, it is impossible to judge whether there is a cat in the picture. Therefore, the perfect accuracy may not be 100%.

The Bayesian optimal error rate is sometimes written as Bayesian , that is, omitting optimal , is from $x$ to $and$The theoretical optimal function of$y$ mapping will never be surpassed. So you should not be surprised. This purple line, no matter how many years you work on a problem, you will never exceed the Bayesian error rate, the Bayesian best error rate.

Facts have proved that machine learning tends to progress quite quickly until you surpass human performance. Sometimes when you surpass human performance, progress sometimes slows down. I think there are two reasons why the progress will slow down when you surpass human performance. One reason is that the human level is not far from the Bayesian optimal error rate in many tasks. People are very good at looking at images and distinguishing whether there are cats or dictation audio. Therefore, when you surpass human performance, there may not be much room for improvement. But the second reason is that as long as your performance is worse than human performance, you can actually use some tools to improve performance. Once you surpass human performance, these tools are not so easy to use.

What I mean is that for tasks that humans are quite good at, including recognizing things by looking at pictures, dictating audio, or reading languages, humans are generally good at processing these natural data. For tasks that humans are good at, as long as your machine learning algorithm is inferior to humans, you can help people tag data for you. You can let people help or spend money to tag examples for you, so you have more data Can be fed to the learning algorithm. We will discuss artificial error rate analysis next week, but as long as human performance is better than any other algorithm, you can let humans see examples of your algorithm processing, know where the error is, and try to understand why people can do it right , The algorithm is wrong. As we will see next week, this will help improve the performance of the algorithm. You can also analyze deviations and variances better, we will talk about them later. But as long as your algorithm is still worse than humans, you have these important strategies to improve the algorithm. Once your algorithm is better than humans, these three strategies are difficult to use. So this may be another benefit compared to human performance, especially on tasks that humans do well.

Why machine learning algorithms are often good at imitating what humans can do, and then catching up even beyond human performance. In particular, even if you know what the deviation is, what is the variance. Knowing how well humans can do on specific tasks can help you better understand whether you should focus on trying to reduce bias or reduce variance, I want to give you an example in the next video.

Course blackboard