Source: Scholar
Jun Wu, born in 1967, graduated from Tsinghua University and Johns Hopkins University, holds a Ph.D. in computer science, former senior researcher at Google, former vice president of Tencent, and venture capitalist in Silicon Valley.
On the evening of April 3rd, Wu Jun, a computer scientist and natural language model expert, was invited to the live broadcast room to share a live broadcast on hot topics such as artificial intelligence and ChatGPT.
Q1：
Why does the emergence of ChatGPT cause panic?
I know that ChatGPT is very popular in China recently, and many people are discussing it, but it is very interesting that in fact, in the United States, not many people talk about this topic. In fact, it’s not just ChatGPT. Looking back ten years, when many new technologies emerged, I found that the discussion in the Chinese media was much more heated than in the United States. Although that technology actually mainly appeared in the United States, the Chinese people are more concerned about it. I think that's a good thing, but it's also a bad thing.
The "bad" is that these technologies are actually overhyped, and in the process, many people who fish in troubled waters make money from it. For example, the blockchain was so hot at the time, but now it is rarely discussed, right? This is the first one. The second is the Metaverse. At present, only Facebook in the United States is still insisting on doing it. When we arrived in China, many people were discussing whether we will live in a completely virtual world in the future. In the end, from the end of last year to the beginning of this year, Facebook invested tens of billions of dollars in this field without listening to a single sound, and finally began large-scale layoffs. Up to now, one of the heated topics is ChatGPT. Some people are excited, some people are afraid, and I also see that there are still many people in China who are fishing in troubled waters, trying to cut everyone's leeks again.
Before I talk about what ChatGPT is, let me tell you a historical story. This historical story will make you laugh when you hear it, but when you look back, many people behave the same way today.
In 1503, Columbus's son wrote down such an incident. Columbus sailed westward to the New World, but halfway through the voyage, when he arrived at Jamaica, there was no food on board. Therefore, Columbus and the crew can only hope that the locals will provide food and beverages. However, after a few days of providing, the crew had conflicts with the locals-some crew members stole things from the locals, so the locals cut off the supply of food.
In order to get out of this predicament, Columbus thought of a coup. Columbus carried a perpetual calendar with him at the time, and marked on the calendar that there would be solar eclipses, lunar eclipses, and all this information on certain years, months, and days. Columbus called the local tribal leaders and said that you had offended God by not providing me with food. God would be angry, the moon would turn red, and God would take the moon away.
Of course, we basically all know now that when a total lunar eclipse occurs, that is, when the earth has not completely blocked the moon, the moon is indeed red, which is what we call a "blood moon". However, Jamaicans at the time did not know it. As a result, at night, the Jamaicans discovered that the moon turned red and then slowly disappeared. The local people fell into panic, and everyone said that God was going to punish them.
The tribal leader hurriedly begged Columbus and promised to agree to all the conditions of Columbus. Columbus said, well, I will go to the tent and pray to God so that he will not punish you, but I need a little time, and then Columbus walked into the tent. In fact, after entering the tent, Columbus was holding an hourglass and watching the time.
Today we have knowledge of astronomy, so we must know the time of the total lunar eclipse, and it will last for about 48 minutes, when the moon will reappear. However, these Jamaicans do not know. All they saw was that Columbus came out of the tent and the moon came out. Then Columbus said, God has listened to my persuasion and promised to forgive you, but you must provide us with good food. Therefore, the locals are grateful and continue to provide them with food.
What does this story illustrate? There are reasons behind the total lunar eclipse, but when people don't know the reason, they can only attribute this natural phenomenon to the action of a god. And this god itself was created by man. That is to say, after man himself created a god, he then fell at the feet of the god and became his slave.
That's why I want to give you the course "History of World Civilization".
In fact, the development process of this civilization is the process of human beings constantly understanding the laws of nature. We have made little progress so that we are no longer like the local indigenous people who blindly believe that praying to God can really prevent the moon from disappearing. We now know that behind the solar and lunar eclipses, it is actually Kepler's three laws of planets that are at work, and then behind Kepler's three laws of planets is Newton's law of universal gravitation. After humans understand this reason, they are no longer just afraid of nature. We can use the laws of nature to do many things.
Q2：
What is the technical basis of ChatGPT?
From history back to the present, in fact, the situation of ChatGPT is similar, and behind it is a mathematical model called a language model. In other words, behind ChatGPT is a mathematical model. Today, the technology appears powerful for three main reasons:
First, it uses a lot of calculations;
Second, it has a large amount of data;
Third, the methods of training language models today are much better than before.
So, what is a language model? Or is it a product of what era?
It was developed in 1972 by a team led by my mentor, Fred Jelinek. Specifically, it was a technology that he led people to complete at IBM at the time, which was used to measure how likely a sentence or a language phenomenon was to occur. What's the use of it then? It was originally used for speech recognition, then for machine translation, and then for computer question and answer, which is the answer to questions we are familiar with today.
At that time, it could make a summary. For example, if there is an article of 10,000 words, how can you summarize the content of this article in ten sentences? For those who do this natural language processing, it is a math problem. That is, what are your terms? The condition is these 10,000 words, and then what is the result you want? The result may be ten sentences, one hundred words. Then there are many combinations here. You can pick a few sentences at will, or split some sentences into two paragraphs, and remove the less important modifiers or descriptions. Then, you can also combine two sentences into one sentence, then when you synthesize a piece of text, the computer will calculate a probability, which sentences have a higher probability of being combined, and it will help you synthesize them according to the probability.
And the ChatGPT we see today is this big language model, and it will pick a text with the highest probability and the most likely occurrence to show you. So generally speaking, the process of ChatGPT generating results is a process that uses a lot of computing resources to calculate. It requires a very large amount of data to support, and there are many, many GPUs (computer processors). Without these things, ChatGPT cannot do it.
And today's ChatGPT is not only technology, but also a lot of manpower behind it. They also hired a company to audit the results generated by ChatGPT. For example, ChatGPT has generated a hundred abstracts, all of which are good, and I can't tell them apart, so these people are responsible for helping me distinguish which one is more like an accurate abstract.
In fact, you can see that there is a language model behind ChatGPT, and the technology of this language model was already available in 1972. Up to now, after fifty years, people in the industry actually don't think it is a big thing. Before this, this language model has actually done a lot of things.
The term "language model" was first coined by my mentor Jalinek. He came to Johns Hopkins University in about 1993, and I came to this university in 1996 and became his student. Then the Chinese of this word, that is, the four words "language model" you see, was created when I published a paper in the 1990s. At that time, only those of us in the circle knew that it could do a lot of things, but you didn't think to say, hey, this thing will be hyped later.
You can understand that the "language model" to ChatGPT is equivalent to Kepler's three planetary laws to lunar eclipses.
Q3：
What was the situation at the beginning of the birth of the "language model"?
So what was the language model at the time of its invention?
In fact, in the 1990s, the models obtained by simple statistical methods were very inaccurate. This is equivalent to, let me make an analogy, you observe the planets, but use Ptolemy's geocentric theory to predict, which is very inaccurate. So, at that time we started to introduce a lot of information about syntax, topics, and semantics. Then, this language model becomes very complicated. After the complexity, it brought a big problem.
what is the problem?
For example, I made a very complex language model at that time. How many parameters did this language model have at that time? 6 million parameters, that is to say, the size of the language model is basically determined according to this parameter. What I did at that time was already the largest and most complex language model that could be done at that time. I didn't use a PC at the time, but 20 super servers, and then it took about three months to train such a language model. So you see, its calculation is very large. So, what are the language model parameters used in the first version of ChatGPT? There are about 200 billion parameters, and you can see the changes over the years.
Therefore, many people ask today, ChatGPT has appeared in the United States, when will Chinese research institutions be able to do ChatGPT? In fact, most research institutions in China can’t do it, not because of the level of research, but because ChatGPT consumes too much resources. Today's ChatGPT may cost about 1 billion US dollars for the hardware alone, and this does not include the electricity cost, so the cost and expense are very huge. So, if you are kidding and ask what is the biggest contribution of ChatGPT, I think it has a great contribution to global warming.
So, what I want to say is that the principle of ChatGPT is very simple, but it is actually quite difficult to do it in engineering.
Q4:
What questions are computers good at answering?
Around 2010, that is, 13 years ago, how far could the language model be? I will show you two examples. Both of these examples were made before I left Google in 2014. At that time, I was in charge of Google's automatic question answering system, which was to let the computer answer questions. But because this product is in English, it basically doesn't show much in the Chinese world.
Let me show you a question answered by Google - why is the sky blue, why is the sky blue?
Its answer is this: sunlight will be refracted when it reaches the earth through the atmosphere, and the gas in the air will scatter the light of different colors to various places. blue.
This was an answer generated by a computer at the time. To be fair, this answer is better than writing a paragraph answer myself, because you need to know a lot of physics to explain this phenomenon, and the sentence seems to make sense. One of the purposes of people using ChatGPT today is to let him answer questions.
Here, I will break it down for you.
In fact, the questions we ask the computer can be divided into two categories, the first category is called simple questions, and the second category is called complex questions. Simple questions are questions about facts, such as where a certain celebrity is from and what year he was born. These are easy questions because it is a fact and has clear answers.
The second category is complex questions, which is what everyone thinks ChatGPT is very amazing. It can integrate information and answer why the sky is blue, as if it has its own logic. Another question is to ask about the process, for example, how do I bake a cake, can you write down the steps? Today we asked ChatGPT how to bake a cake. It can write you the process in detail, how many cups of water, how many eggs to add, how much flour to add, etc., it can tell you. Then you can actually bake a cake based on the answers it provides, and it might be pretty good.
This is where everyone finds it amazing. But you have to know that in 2014, computers have already done this, and they have done it very well. So, there's not much mystery about the technology itself.
Q5:
Who is better at writing, a computer or a human?
Now, everyone is talking about ChatGPT, and another reason is that they think it can write. For example, writing a work briefing, this is where Americans use ChatGPT the most today. I did 1234567 this week, these seven things, hey, you see, I don’t have to write it myself, I let ChatGPT generate one, and then edit it for a while.
However, computer writing is actually difficult or easy to say, and I can give you an example.
After I left Google in 2014, I didn't do much programming at that time, but I still had some computing resources at that time, so I would write some programs in my spare time and do it for fun. At that time, I let the computer write two poems, and everyone can read these two poems.
The first poem is a five-character poem. In my own words, it is a poem in the style of Li Bai. You can read it. The poem is written by the computer itself. In fact, if you read it, there are some characteristics of Li Bai in this poem.
For the second poem, I also put the picture below, you can take a look.
Let me tell you first, because ancient poems all have the word flat and narrow, but our current pronunciation is different from the pronunciation at that time, so we don’t care whether the level is in line with ancient times, but we only look at it from its content and artistic conception. Reading will feel very smooth.
OK, then again. How did you do the first poem?
In fact, it couldn't be simpler, you just put Li Bai's poems into the computer. There are more than 1,000 Li Bai poems, and there are only about 10,000 sentences. This is too simple for a computer. When it is written, it splits the sentences into groups of two characters and three characters, such as "kongchou" is a group, and "recalling Chang'an" is a group of three characters. Then it puts together the language model I just mentioned, and calculates the probability, which one has the highest probability; after dismantling it, I make a request to him, saying that I want to write a poem recalling Chang'an, and it arranges and combines them to generate this " "Recalling Chang'an" is actually pieced together like this. The second poem is a little more complicated.
But do you know how long it took me to write these two programs? two days. What is this indicating? Explain that it is not very difficult for you to let the computer write some decent things. It is not as mysterious as you think, or computer writing itself is not as mysterious as you think.
So why do these two poems look so good? Because this is Tang poetry, the format of Tang poetry is fixed. In the same way, why is it good to write weekly reports with ChatGPT? Because the format of the weekly report is basically a list, it is also a fixed format. Including, if you read the Chinese version of the "Wall Street Journal", let me tell you here that 90% of the content is written by computers, but you don't know it. After finishing writing, of course people have to give it a theme, then write an introduction for the first paragraph it writes, and then give a summary and a title. This is what people have to do.
Why is it better to write financial articles? Because it has a lot of facts in it, and the format is fixed, so it does this very well.
I spent so long talking about the background of ChatGPT, in fact, I want to say that it is not mysterious, not a very deep machine behind it. On the one hand, ChatGPT relies on a mathematical model, and this mathematical model existed in 1972, but today it has very strong computing power and relies on brute force calculations.
So, how much power does ChatGPT consume once for training? Probably 3,000 Tesla electric cars, each running 200,000 miles, running it to death, such a large power consumption is enough for one training session, this is a very expensive thing.
Q6:
How does ChatGPT affect us?
Then let’s talk about the impact of ChatGPT on people.
This is going to go back to history. Every technological revolution will actually have some impact on people. However, ChatGPT is not a new technological revolution, because I just said that this process is very long. From the 1970s to the 1990s, we did a lot of things, and from the 1990s to now, many people have done it A lot of things. The biggest progress here is actually not the language model itself, but the deep learning that was produced around 2000, which made the training language model more accurate than before, not simply doing statistics.
Today, training language models is no longer simply doing statistics. This is one of the reasons why ChatGPT can produce better results.
As for what kind of influence ChatGPT can have on people, I will not answer you directly for this question. Let me ask you first. Did you find any characteristics of the two Tang poems I showed you just now? By the way, these two poems are well written, but if you knew the Tang Dynasty before, you will not have a new understanding because of these two poems. Because, ChatGPT is a bit like a parrot to a certain extent, you have to say a word before it can learn. It might sound nice, but it doesn't provide much information.
90% of the content on the Internet today falls into this category—it does not provide more new information, nor is it original content, nor is it my own perception, it is nothing more than copying from east to west. At present, short videos such as Douyin and Kuaishou, I think 99% of the content belongs to this category, which is not nutritious. You may find it interesting after reading it, but in fact, no matter how much you read on it, it is actually useless to you. any help.
If ChatGPT really threatens anyone, I think it is the work of this type of people that is threatened. That is to say, those who make short videos or post some content on Douyin, ChatGPT will do a lot better than them. You just think about such a thing, assuming that a group of people turn over and over the sentences in the 300 Tang poems every day, and they can also make some poems, then ChatGPT must be much faster than people. So this technology will have an impact on this group of people.
So, who won't be affected? That is, people who create content will not be affected.
Why do I say that? Remember the question "Why is the sky blue" that I said just now? Why can Google answer this question?
Because when Google answered, it probably analyzed almost all decent sentences in English at that time, and there were about 100 billion English sentences. Well, in fact, you will find that on the websites of some universities and NASA, it has this answer, but we piece it together, delete and delete, and pick it out. But the earliest physicists did this research and figured out the truth. This work is meaningful and cannot be replaced by ChatCPT.
So, what is the working equivalent of ChatGPT? For example, after Ptolemy created this model, every once in a while, they would compile a calendar of about several decades in Europe, and then mark the day on which there will be a solar eclipse, how the planets will move, etc. . Then people printed many copies of this book according to these rules. This ChatGPT is equivalent to having many books. After you read it after holding it, you say, oh, a lunar eclipse will occur on a certain day, and the answer will be very clear. However, the real meaningful work behind it is not to print this book, but to do the research of Ptolemy.
So I think that ChatGPT is not actually a technological revolution from a historical point of view. It affects those who are lazy, who are too lazy to use their brains to create new things. Those who truly explore the mysteries of human knowledge will not be replaced.
Q7:
What new opportunities can ChatGPT bring?
Many people ask, what new opportunities does ChatGPT have? Frankly speaking, you have no chance, because it consumes too many resources and you can't afford it. So who can benefit? That is these people who sell resources.
I can make an analogy, that is, during the California Gold Rush, many people flocked to seek gold, but we still don’t know which gold diggers really made money, and no one left their names. down. But who made the money in the end? It's the guy who sells water and the guy who sells jeans. The same is true for ChatGPT. Everyone goes to pan for gold together. In fact, you can’t earn money, but in the process, you still have to buy water and jeans to wear. In the end, these two groups of people earn money. Levi's, a company born at that time, made jeans.
Then in the end you may be paying money to several large cloud computing companies, which may be a result.
Well, after talking about the history of ChatGPT, I will give you a brief summary.
First, don't be afraid.
Today, many people are afraid of ChatGPT, just like the Jamaican natives who Columbus encountered back then feared the lunar eclipse, the same reason.
Second, don't force yourself to find so-called opportunities, work as you do.
I saw some students asked me why Apple didn’t do ChatGPT, and I said that’s right! That's why Apple is the richest company in the world, with the most profits and the most market capitalization. At present, many so-called companies that do this kind of artificial intelligence are losing money until now. Therefore, this is why when many students sometimes ask a lot of questions that are too inconspicuous, I jokingly ask him, have you paid off your mortgage? If you don't pay it off, you should go back to work and do your job well. This is the most meaningful thing for everyone, and it is also true from a historical point of view.
Third, you have to see through the tricks of these so-called conspirators or those who want to cut your leeks.
That is to say, if another person pretends that Columbus said that he is the representative of God, and then he can pray that the moon will come out, don't believe it. So you need to understand some of the science behind ChatGPT. Some of the simplest principles, like those I talked about today, you still need to understand.