Recently, OpenAI's pre-trained model ChatGPT has impressed and inspired researchers in the field of artificial intelligence. It's strong and smart, no doubt, and it's fun to talk to and write code. Its capabilities in many ways far exceed the expectations of natural language processing researchers. So we naturally have a question: How did ChatGPT become so strong? Where did its various powerful abilities come from? In this article, we try to analyze ChatGPT's Emergent Ability, trace the source of these capabilities, and hope to give a comprehensive technical roadmap to illustrate how the GPT-3.5 model series and related large-scale language models are Evolved step by step into the current powerful form.
We hope this post promotes the transparency of large language models and serves as a roadmap for the open source community to work together to reproduce GPT-3.5.
-
In the eyes of the international academic community, ChatGPT / GPT-3.5 is an epoch-making product. The difference between it and the previous common language model (Bert/ Bart/ T5) is almost the difference between missiles and bows and arrows. Pay attention to.
-
In my communication with international colleagues, international mainstream academic institutions (such as Stanford University, University of California, Berkeley) and mainstream industry research institutes (such as Google Brain, Microsoft Research) have fully embraced large models.
-
At the current stage, the gap between the domestic technical level, academic vision, academic philosophy and the international frontier does not seem to be decreasing, but is expanding. If the status quo continues, there is a high possibility of technological dynasties.
-
This is the autumn of critical life and death.
Years later, facing the firing squad, Colonel Aureliano Buendía would recall that distant afternoon when his father took him to discover ice. - García Márquez, One Hundred Years of Solitude