Source | The Robot Brains Podcast
OneFlow compilation
and translation｜Wan Zilin, Yang Ting
On September 26, OpenAI announced that ChatGPT has added image recognition and speech capabilities, allowing ChatGPT to not only communicate through text, but also to display images and interact with it. This is a major upgrade in ChatGPT's multi-modal evolution.
John Schulman , co-founder of OpenAI and architect of ChatGPT , previously believed that adding multi-modal features would bring great performance improvements to large models. “If there are diminishing returns to scaling, then adding multimodality allows the model to acquire knowledge not available in text and potentially master tasks that pure language models cannot. For example, interacting with the physical world or even a computer screen by looking at it of video, the model can gain huge benefits.”
Before taking charge of ChatGPT, Schulman was one of the early pioneers of deep reinforcement learning and invented the widely used proximal policy optimization algorithm (PPO), which is actually part of the training of ChatGPT. He also invented Trust Region Policy Optimization (TRPO) and made important contributions to OpenAI Gym, OpenAI Benchmark, and many meta-learning algorithms in the modern deep learning era.
Before founding OpenAI and before joining OpenAI, Schulman studied for a PhD at the University of California, Berkeley. At first, he mainly studied robotics. With the rise of deep learning, he turned to reinforcement learning. His mentor was Pieter Abbeel, a leader in the field of reinforcement learning. .
Recently, John Schulman and Pieter Abbeel conducted a detailed discussion on the construction process and methods, capabilities, limitations and other model details of ChatGPT, and looked forward to the development direction of the evolution of large language models to multi-modal models. In addition, Schulman also shared his research process and experience along the way.
(The following content is compiled and published by OneFlow with authorization. Please contact us for authorization for reprinting. Video: https://www.youtube.com/watch?v=nM_3d37lmcM)
1
Construction of ChatGPT
Pieter Abbeel: I am very impressed by ChatGPT. I once entered a speech about a company into ChatGPT, and then asked ChatGPT to describe the company in a paragraph. After completing it, I further asked ChatGPT to describe the company in the style of Snoop Dogg (West Coast rap star Snoop Lion (formerly known as Snoop Dogg)). company. Even though Snoop Dogg has never written a rap about the company, ChatGPT gets the job done well. I was blown away by the capabilities of ChatGPT and I couldn't help but wonder how this model was built.
John Schulman: Training ChatGPT has the following steps. First, starting with a pre-trained language model, the model is trained to imitate large amounts of text written by humans. We want the model to be able to use language like humans, and for this we need a lot of text data. We collect a lot of text from the Internet and train the model to generate similar text. During the training process, the model predicts the next word based on the previous text, and a pre-trained language model is obtained through extensive training, but it can only generate random text similar to that on the Internet, some of which meet the requirements and some of which do not. In order to train the model to generate more consistent and professional content, we need to fine-tune the model.
There are two steps to fine-tuning. The first step is supervised learning, where we train the model using high-quality responses written by hired people. The second step is reinforcement learning to further improve the model. In this step, we train a reward model to identify good responses and use this model for reinforcement learning training.
Pieter Abbeel: Reinforcement learning with a reward model is to evaluate the text generated by the robot through rewards and try to maximize the rewards obtained. So is it possible to create a chatbot with text-generating abilities that exceed those of humans?
John Schulman: Absolutely. In some respects, models already have superhuman capabilities, but in other respects they may still need to improve. For example, if speed is a measure of ability, these models are much faster than humans at things like writing poetry or creating Snoop Dogg-style lyrics. In short, we cannot use a single indicator to measure the intelligence of the model. They have surpassed humans in some aspects, such as the breadth of knowledge and the diversity of writing styles, but the model's capabilities in other aspects still need to be improved.
Pieter Abbeel: In what areas can the model still be improved?
John Schulman: While models perform well on some tasks, they perform far less well than skilled humans on tasks such as mathematical reasoning. For example, in tasks that require long calculations, chatbots often have difficulty performing tasks effectively. Even if the input prompts are very detailed and careful, and tell the model that it can take multiple steps, they often get stuck during the calculation process and cannot successfully complete the task. .
Pieter Abbeel: To be honest, it also takes humans a while to become proficient in mathematics, and most people cannot really reach the level of top mathematicians. So, is the current application of models in mathematics just the beginning, and will their performance gradually improve?
John Schulman: Models are getting better and better at math. At present, it is difficult for us to determine the fundamental limitations faced by the model. We only know that there are indeed some limiting factors, such as insufficient execution mechanism, which can only generate text and cannot actually perform tasks, etc. But these are superficial limitations that can be overcome, not the key point. Current models are not smart enough for high-quality creative thinking or for things like math and science. Achieving these goals will take some time, and at this time, we are not sure exactly how long it will take and where the model will go.
Pieter Abbeel: Language models have been around for a long time, but it wasn’t until ChatGPT became popular that language models really came into the public eye. The intensity of interaction between people and ChatGPT was beyond the reach of previous language models. Did you anticipate that this would happen when you developed ChatGPT and its predecessor InstructGPT and introduced reinforcement learning components into it?
John Schulman: I do think that ChatGPT's chat interface is more convenient and easier to use than the previous model, and has greater potential. However, I did not expect it to be so popular. Initially, I thought that ChatGPT might only be suitable for some specific fields, and that InstructGPT can be used to make chatbots. We just need to enter the correct prompts into InstructGPT, and then it will run like a chatbot and get good results.
When initially released, ChatGPT performed similarly to InstructGPT, but ChatGPT was more self-aware, understood its own limitations, and had fewer illusions. The Instruct model is mainly used to generate continuous text and conduct writing tests. It has a certain hallucination component, which is one of the characteristics of InstructGPT. In some ways, ChatGPT may be slightly better than the previous model, but it's not a significant improvement. So I didn't expect it to be so successful either.
2
Causes and solutions of hallucinations
Pieter Abbeel: You mentioned that ChatGPT may cause hallucinations, so what exactly are hallucinations? What is its cause? How to avoid hallucinations in the model?
John Schulman: Hallucination is when a model makes up and outputs some plausible text that may contain fictitious content, numbers, or quotes. We can understand the illusion of the model this way: the model has a certain degree of agency, and it is more concerned with correct expression or writing and outputting content in an appropriate style, followed by accuracy.
This choice of model is obvious considering the maximum likelihood goal. In this case, the model will not pay too much attention to whether the output content is correct or not, but more to sound correct or look reasonable. Therefore, with Models trained in simple ways often produce hallucinations. Through fine-tuning and human feedback, we can significantly reduce the output of hallucinations, but cannot completely eliminate them. The free model has more hallucinations, and the GPT-4-based model has less hallucination output, but still occurs occasionally, especially when it comes to specific limitations that the model is not trained to be aware of.
Pieter Abbeel: Does “the model is untrained and unaware of specific limitations” mean that we can train the model to recognize these limitations?
John Schulman: We can train the model to be aware of certain constraints. For example, an early model had no idea what it was capable of. When you asked it if it could send an email to someone, it might answer "Yes, I just sent that email." This particular type of query continues training so that it learns to answer "No, I can't send the email." That is, we can use a staged process to teach the model some specific constraints that it cannot do, and the model will generalize to this.
GPT-4 is a very smart model with strong generalization capabilities. If you tell it something it can't do, it can usually infer many other things that are beyond its capabilities. But this method is not perfect. For example, in terms of citations, the model has a lot of knowledge about specific books and important papers. If the model is asked to provide citations in this area, it can sometimes give useful correct answers. So during the writing process, we prefer to let the model provide the answer.
Therefore, the model will think that it is able to provide references to some extent, but it does not have a good internal sense of how confident it is in these references, which leads the model to sometimes make up references. The model may or may not know that it made up the reference, so sometimes you can ask it if it's sure about it. We don't yet know much about how these capabilities of models and the model's perception of its own limitations generalize, which is an interesting research topic .
Pieter Abbeel: Regarding citations, rather than having the model provide citations when the model has read the entire Internet beforehand, it is better to have the model obtain relevant information in real-time retrieval. How do you weigh the two approaches of retrieving and training model weights?
John Schulman: Both methods have their applicable scenarios, and currently, we are combining the two methods . We use a browsing model in ChatGPT that can find information from the web (although we have recently temporarily turned off this feature, but will restore it later). I think that models can store a lot of information in the weights, including very detailed factual knowledge. If information is stored in weights, the model can use this information flexibly, such as using this information to establish connections between things. At this time, even if you ask a question vaguely related to something, the model may successfully build on the search Connections that are difficult to make in inquiries. Therefore, storing information in weights makes the model more intelligent and flexible.
Retrieval also has important advantages of its own. First, retrieval provides real-time access to information and can provide more detail than storing information in weights. Second, the retrieval output is easier to verify manually. During the model training and use case testing stages, it is very important to ensure the detectability of the model output. Because the model's knowledge range is so broad, people rating model outputs may face the rater's lack of understanding of the topic, which can lead to difficulty in scoring. Therefore, if the model can provide citation sources, this will greatly improve the accuracy of supervision evaluation. Because models sometimes output hallucinations, it is helpful for the end user to verify the output of the language model to deal with hallucinations.
Pieter Abbeel: Most large language models only perform single iteration training (single Epoch training), that is, they only train the data once. How does the model remember this specific information in just one pass? How can I remember a specific citation and store it while doing a pass by doing only one gradient update?
John Schulman: It’s really surprising how well language models absorb information from pre-training data. Many established facts often appear multiple times in different documents on the Internet. If a fact only appears in one document, the model may not be able to recall this fact (at least this is the current model). There’s no exact answer yet as to how many times a model needs to see a particular fact before it can actually remember it and internalize it, and it could be dozens of times.
3
New ways to explore LLM capabilities
Pieter Abbeel: Obviously these models are trained on a lot of data. Recently, Sam Altman mentioned that as the scale of the model continues to expand, it is no longer possible to continue to increase training data and expand the scale of the model to continue to improve model performance, which means that there are not many opportunities that data and scale can bring. What do you think of it?
John Schulman: Indeed, the performance improvement brought by existing data and model scale expansion methods may reach its limit after a period of time, and the improvements brought by algorithms, data sets, data set sizes, and computing power will gradually decrease, but we are still far away from it. It will take some time before this situation occurs, and there is still a lot that can be done before then.
Pieter Abbeel: In the future, will we continue to rely on human feedback to filter text data? Do other elements need to be introduced, such as having the model watch a video to better understand the physical world, or having the model use a simulator to experience what the physical world feels like? Can introducing new elements push the model to the next level? Is the introduction of these new elements less relevant to improving the performance of the model than adding more data of existing types?
John Schulman: I think adding multimodal capabilities will bring huge performance improvements. If there are diminishing returns to scaling, adding multimodality allows the model to acquire knowledge not available in text and potentially master tasks that pure language models cannot. For example, models can benefit greatly from watching videos of them interacting with the physical world or even with a computer screen. All software is designed for humans, and if the model can see pixels and understand video, we can use a variety of existing software or help people use it. Giving the model new capabilities and allowing the model to interact with new things will greatly enhance the model's actual capabilities.
In addition, through more intelligent fine-tuning, pure language models can also complete many tasks. Reinforcement learning based on human feedback also has a lot of room for improvement. Rather than relying on reward models trained on human data, it is better to use the model itself for evaluation.
Pieter Abbeel: This reminds me of Generative Adversarial Networks (GAN): Generative Adversarial Networks have two models, one for training and another for generating more realistic tasks. But in the original GAN, the model usually generates images, which sounds very similar to fine-tuning, but fine-tuning may reduce the model's generalization ability and the breadth of demonstrated knowledge. Is this true? If true, how does this process occur?
John Schulman: That does exist. When a model is fine-tuned, it reduces the diversity of its output styles and content types, leading to mode collapse or entropy collapse. In some cases, the model outputs a very narrow or singular set of answers. .
If you ask a language model, such as ChatGPT, to tell a joke, the model is likely to keep outputting the same joke, such as "Why don't scientists believe in atoms? Because atoms "make up" everything (original text is "because atoms make up everything", where "make up ” (meaning both “composition” and “fabrication”, “fiction”)” and other corny jokes, or some similar unfunny jokes, the model will continue to be used. In short, this mode collapse effect does occur.
Pieter Abbeel: When fine-tuning is performed, the capabilities of the model may also decrease.
John Schulman : This is due to the use of larger batches of data in pre-training, while ensuring that the full capabilities of the model are retained across a large number of inputs of various types. However, when fine-tuning, you only see a much smaller data set, so you may have lost some capabilities that were not fully represented in the fine-tuning data set. At the same time, more noise may appear during the fine-tuning process, and the model performance may be slightly degraded due to the noise.
Therefore, fine-tuning does result in a slight decrease in model performance. We conducted various benchmark tests on the model and compared it with the pre-trained base model to try to suppress the decline in model ability. But what is certain is that under the latest configuration, the performance degradation of the model is not serious.
4
Closed source or open source?
Pieter Abbeel: Currently in the field of AI, in addition to ChatGPT, there are many other models including open source models. What do you think about open source and closed source models?
John Schulman: Open source models are extremely valuable for academic research. Academic researchers can fine-tune the model, change the architecture, and improve reinforcement learning through human feedback. Powerful open source models make this type of research possible. At present, the closed-source model has more technical advantages and is the best model currently known. Obviously, it's hard to incentivize people to develop truly good models without commercial incentives and without being closed source, so I would expect the best performing models to be closed source. However, open source models are more conducive to activities such as academic research, and may also lead to many excellent commercial use cases, because people can fine-tune the models based on their own data, which is something that current business model vendors cannot provide.
Pieter Abbeel: If you want to build an open source model, it can be very difficult to obtain high-quality data and large computing resources?
John Schulman: Exactly. The open source model is not profitable, making it difficult to secure large investments. Therefore, although the open source model has great public welfare value, it will not be a SOTA model. In addition, open source also faces some security concerns. For example, some people use the open source model to spread large-scale spam. This behavior is not allowed by API providers. In view of the above problems, open source model companies such as Meta need to treat open source issues carefully, but the specific development direction of the open source model in the future remains to be seen.
5
LLM dominance and prospects
Pieter Abbeel: Now, language models have taken a dominant position in conversations. Compared with other fields, language models have made many major breakthroughs, and their rapidly growing capabilities have triggered widespread discussions. Will similar major leaps occur in other fields in the future? If so, in what areas will it appear?
John Schulman: I can’t predict exactly what areas will see major breakthroughs. But I think that core technologies similar to language large models will become the cornerstone of many fields to build associations between different modalities, such as combining large language models with vision, video, etc., to achieve more functions. This is a huge advantage of language models, because language has a higher information density than other types of data (such as video) and contains less noise. Therefore, language will be a good medium for a long time to come, capable of carrying a large amount of intelligence with limited computing resources.
However, I think there is a tendency for language to interpenetrate with other modalities. We may see people blend language with multiple modalities such as video, or even language with robotics. I predict that future robotics will employ multimodal models that will be jointly trained on language, video, and control. I believe this type of combination will become one of the important development trends in the field of artificial intelligence in the future.
There are also areas where scalable super-supervision or improving the quality of supervision is more important, although independent of the type of model currently being trained. The key lies in how to collect data in certain areas, especially those where it is difficult for humans to generate high-quality labels or examples.
So, how do we combine humans and machines to create data of a higher quality than humans can produce on their own and enable oversight of models in areas where real difficulty exists? This is an interesting and important question that is likely to gain more attention in the field of machine learning research.
Pieter Abbeel: The example that comes to mind is an AI that can perform efficient scientific research, which can read biological data that is difficult for humans to interpret, such as protein sequences, RNA and DNA sequences, and then analyze the experimental results and somehow convert them into They combine to propose new hypotheses and even draw conclusions that are difficult for humans to reach.
John Schulman: Exactly. This is an exciting area of research, and there may be some areas of research similar to biology that are too complex for humans, but with artificial intelligence, even if they are no smarter than humans, they can get the job done more efficiently. Therefore, we may be able to let models sift through large amounts of complex data in biology and draw valuable conclusions from it.
6
Research career and experience
Pieter Abbeel: I am very interested in your career development trajectory. Initially in my laboratory, you mainly worked on imitation learning and robotics. Later, you realized the development potential of reinforcement learning, so you spent a lot of time on OpenAI. It conducts research and introduces it into the field of language models. So, looking back now, when did you start paying attention to language models? what is the reason?
John Schulman: Around the time of GPT-2, people gradually realized the huge potential of language models and thought this area was worthy of attention. In fact, I did not immediately shift my focus to language models at that time. I thought that unsupervised learning was starting to pay off to a certain extent, and that training generative models was a very effective way to create models with general capabilities that could be fine-tuned for a variety of downstream tasks.
At that time, I was very interested in sample efficiency and reinforcement learning, that is, studying how quickly a model learns new tasks. This issue is to some extent the core of reinforcement learning, and can even be said to be one of the core problems of artificial intelligence.
After GPT-2 was released, it performed well on many tasks, such as understanding contextual meaning with a small number of examples. At the same time, fine-tuning also achieved many good results and was able to complete various tasks, such as natural language benchmarks. In short, I thought at the time that we should apply model training in the field of reinforcement learning to the field of games and robotics.
Maybe we should train the video model and then fine-tune it through reinforcement learning, so I started working on it, and although I achieved some results, I didn't achieve exciting enough results in the GPT-3 era. The performance of GPT-3 surprised me even more, so I decided to shift the focus of my work and the team's work to reinforcement learning. Although we are not the first team at OpenAI to develop a language model on our own, we decided to move in this direction as a reinforcement learning team.
At that time, we were working on two projects, one focused on solving mathematical problems, and the other focused on retrieval and web browsing, using reinforcement learning to better learn how to use these tools. This was my opportunity to enter the field of language models. I also remember that the exact time should be 2020 or mid-2019.
Pieter Abbeel: During your PhD, you changed your research direction once, from the original imitation learning (teaching robots to learn from examples) to reinforcement learning. At that time, you had already made some achievements in imitation learning. Why did you turn to reinforcement learning? For most researchers, whether the research direction they are engaged in is correct is a key issue, especially if they have invested a lot of time on a certain topic in the early stage. At this time, switching to a new related field is a high-cost undertaking. decisions, there may be an adaptation period before achieving the same productivity in the new area, during which progress may be slow.
John Schulman: Yes. Aside from my initial decision to get into machine learning, switching from robotics to reinforcement learning was the biggest change in research direction I've ever made. After the transition, I just did some simple examples for a long time, such as implementing carpooling functions. This state lasted for about six months. Soon I realized that I needed to make up my mind, and then the transition gradually became smoother. This transition became more like a continuous transition, such as moving from one reinforcement learning domain to another, and from solving one problem to solving another. question. These transitions have always come quite naturally to me, and I consider myself lucky to have been able to switch my research topic to reinforcement learning at the right time.
Pieter Abbeel: At least at the moment when deep learning is really coming into play, you were involved in some of the earliest work on combining planning with reinforcement learning, which is still the basis for many people to build and use today, including proximal policy optimization (PPO) , which may still be the most widely used reinforcement learning algorithm to date.
Although seven years have passed now, you must still remember your doctoral days. Now many doctoral students are thinking about an issue, that is, the impact of industry on scientific research, such as OpenAI with a huge budget, the latest up to 100 from Microsoft A $100 million investment, the funding appears to be primarily used for computing resources and possibly data governance. Such a large-scale budget is obviously impossible to achieve in a doctoral program. Therefore, working at OpenAI may provide opportunities that are not possible in a doctoral program, and may be able to show off one's talents in the field of AI. From your perspective, as a member of OpenAI, do you think there are some opportunities that don't require large-scale computing and data budgets, but can still produce exciting results?
John Schulman: According to some academic papers I happened to see, the above goal is achievable, but it is not easy to achieve this goal. You need to find a direction that won't be overtaken by industry and won't be replaced by some newly released model. I think there are actually many directions that meet the above conditions, such as fine-tuning, scientific investigation, in-depth understanding of how the model generalizes, or research on better supervision methods. There are a lot of opportunities to do high-quality scientific research, and we need to really dig deep and understand things, and industrial labs may be more focused on results and creating better products. Regardless, PhD students in the field of AI need to conduct careful research, stay curious, and try to understand the field as deeply as possible.
Pieter Abbeel: Do you study any issues that interest you in your spare time? If I were a computer science student, is there a process I could use to identify these valuable questions?
John Schulman: I think it's exciting to find out what capabilities the model has, but the way to achieve those capabilities isn't entirely clear. Currently, we have some limitations in training models, and it may not be necessary to introduce new capabilities, such as allowing the model to perform surgeries, etc., but it is confusing that we cannot understand where in the data set the source of the model's capabilities lies.
I think attributing model behavior to data sets is an interesting area and there has been some interesting research in this area recently . However, the current problem is that we pre-train the model and then fine-tune it on different datasets, and end up with a model, but we are not sure where all the behavior of the model comes from. For this case, we might be able to combine the future capabilities of interest or the shortcomings of current methods with solutions and avoid those that can be solved without the help of a mentor.
Pieter Abbee: Looking back at the origins of deep learning, at the beginning, no one except a few people studied deep learning, and then breakthroughs were made in the field, which of course included the efforts of many collaborators. Now, everyone is using big models trained on large data sets, but in the future it may be micro data sets, and although the models will not be too tiny, there may be some as-yet-undiscovered methods that are very different from what we are doing today.
John Schulman: This is very likely to become a reality, and greater breakthroughs may be achieved in the future with smaller but richer data sets. Humans learn from a lot of data, and the bandwidth of data we receive through our eyes is very high. Although human infants are exposed to very limited data compared to pre-training datasets, mostly confined to a single room, infants see much richer data.
In summary, it would be an amazing achievement if you could learn a very good vision system from just a tiny dataset. So I think it's likely that there are some new architectures and loss functions that can achieve this goal. We often tend to continue to pursue methods that have achieved good results and are highly scalable, but in fact there is still a lot that has not yet been discovered, and we may still be stuck in a specific local optimal solution, and it is difficult to predict the next one. Where is the big breaking point?
When people talk about the collaboration between Geoffrey Hinton, Yoshua Bengio, and Yann Lecun (the Big Three of deep learning) at the origins of deep learning, there may be survivorship bias. The three of them may have been working in the right fields at the time, but there were many others working in other, less well-known fields, and the importance of their work was never noticed or recognized.
Pieter Abbeel: I personally don’t entirely agree with this, but it’s really difficult in academia to do a PhD while doing the same interesting research as in industry. If you can directly enter the industry, you will immediately have access to richer resources, allowing you to conduct larger-scale experiments, or at least produce more significant results. Taking your personal experience as an example, you completed your undergraduate studies at Caltech and later pursued a PhD at Berkeley. Assume that you have just graduated from Caltech. Do you think you will choose to continue your PhD or directly join industry? Research lab?
John Schulman: Either pursuing a PhD or joining a research program directly is probably a good place to start, both options have their own advantages and disadvantages. For example, a PhD requires a longer investment, which means you can truly become an expert in a certain field. At the same time, as a PhD student, you can also participate in internships, etc. I think there is nothing wrong with choosing to study for a PhD. But if you choose to join an internship program directly, you may have less freedom and may be somewhat restricted in exploring different areas, so there may be some trade-offs between the two. I'm not sure which way I would choose, I think both are good.
Pieter Abbeel: As clearly one of the most successful PhD students in my lab, many new students ask you how you conduct your research. As a doctoral candidate, what research methods do you follow, what is your schedule, what is your development trajectory in the project, and how do you promote the project?
John Schulman: I happened to have the right time and place conditions, so even if I were to do it all over again, it would be difficult to completely reproduce the exact initial conditions at that time. But my suggestion is that if I want to study a certain field, I will fully read relevant materials, such as reading papers in this field in detail. In addition, I will also study a lot of basic content in depth, such as related textbooks on optimization and information theory.
In terms of specific problems, in the first few years, I basically followed the research direction of the laboratory. We created some good results, but we did not conduct research randomly, and we also tried to use some methods that seemed relatively reasonable. Overall, I try to find a natural or reasonable balance there, where I think inspiring questions are really valuable, but they don't necessarily need to be solved without building a product. In the end, you may not actually end up with a useful product, but just hope that these problems will lead to a good solution.
In the first few years, I worked hard to achieve some exciting results in various fields. With the rise of deep learning, I began to have a sense of crisis similar to a mid-life crisis, and began to reflect on whether everything I had done in the field of robotics was Deep enough that I wasn't sure if these methods would have sustained success, so I started exploring deeper and decided to move into deep reinforcement learning research.
This development was quite natural. In the beginning, you will work goal-oriented, and at this stage you may not have a clear understanding of the method. After continuing to study for a period of time, you will gradually realize the limitations of the current paradigm, which will provide useful ideas for the next paradigm or the next method-oriented research.
Everyone else is watching
Welcome to Star and try OneFlow:
github.com/Oneflow-Inc/oneflow/
http://github.com/Oneflow-Inc/oneflow/