“Technology is there for humanity, for our benefit, for our prosperity, but it’s easy for us to forget that.”
Kenneth Stanley (left), Joel Lehman (right)
Text | Zheng Keshu
Editor｜Liu Yiqin
Kenneth Stanley is not a consulting practitioner, but in recent months his inbox has been flooded with inquiries. Confused people asked him in emails: "What does it all mean?"
“All this” refers to the emergence of ChatGPT and the AI craze it triggered. Since its release last November, people around the world have been using it, talking about it, and trying to understand the threats and opportunities it represents.
OpenAI, the company that developed ChatGPT, has also become a hot topic. Kenneth worked there. Investors who have missed ChatGPT and are worried about missing new opportunities again have contacted him and want to know what he is doing, because "anyone who has a relationship with artificial intelligence and OpenAI may be doing important things."
Another former OpenAI employee, Joel Lehman, has received inquiries from headhunters around the world, including China. He and Kenneth joined OpenAI in 2020 and left a few months before the release of ChatGPT. At OpenAI, the Open-Endedness team they co-lead is dedicated to allowing algorithms to learn and innovate on their own without predetermined goals.
Previously, they worked at universities, AI startup Geometric Intelligence, and Uber AI Labs. After leaving OpenAI in 2022, Kenneth founded a company called Maven to build an open, accidental social network, while Joel led the open research team at Carper, a subsidiary of AI unicorn Stability.
They have been working in the artificial intelligence industry for many years, but the technology is advancing so fast and the number of papers is so large that sometimes it is difficult for them to keep up.
At this moment, China’s artificial intelligence industry faces similar anxieties. Worried about missing opportunities, industry and academic institutions have released large models one after another. On May 28, the China Institute of Scientific and Technological Information, directly under the Ministry of Science and Technology, released the "China Artificial Intelligence Large Model Map Research Report", stating that there are at least 79 large models with more than 1 billion parameters in China. The numbers continue to grow, competition is getting fiercer and players are riding the wave. Many Chinese AI entrepreneurs told "Financial Eleven" that the AI issues they are most concerned about right now are business models and monetization capabilities.
Kenneth and Joel, who have gone through the struggle for money phase, gave completely different answers. Kenneth's biggest concern is that AI will drive people away from humanity - machines will become more and more interesting, so that people will spend more time communicating with machines instead of interacting with people. Over time, social bonds will break and governments will collapse. The reason why he initially became interested in artificial intelligence was that he wanted to understand human nature by understanding "intelligence".
Joel is concerned about how machine learning can make a positive contribution to the open growth of individuals and society. He said that current machine learning treats humans in a very narrow way and regards satisfying preferences as the only way to promote human prosperity. This has given rise to information cocoons and increased barriers between people. Human beings are rich and complex. "We may be addicted to certain things and may do things that are against our own interests; we have willpower, but sometimes we only do what we really want to do after struggling." In February this year, he published a paper "Machines" "Machine Love" explores the possibility of machines "loving" people from a richer and more complex dimension.
They use AI tools to assist their work, but refuse AI participation in some things. Kenneth is very happy when playing with his children without the help of any electronic devices; Joel rarely uses ChatGPT in his writing because he "doesn't want to automate the things he really cares about."
Innovation is another thing they care about. In 2015, the two co-authored "Why Greatness Cannot Be Planned" was published. In the book, they record their findings during many years of AI research: setting goals can sometimes become an obstacle; following the guidance of curiosity can solve problems and achieve innovation. The Chinese version of the book was published this year, which sparked discussions among many Chinese science and technology and education practitioners.
In May and June this year, "Financial Eleven" conducted a written communication and a video communication with Kenneth and Joel respectively. In addition to the AI issues they are most concerned about, they also talked about the working experience of OpenAI, the venture capital environment of the US AI industry, as well as technical issues such as the flaws of Transformer and the arrival of AGI (artificial general intelligence).
The following is a conversation between Kenneth, Joel and Finance Eleven. The dialogue has been edited and deleted:
The strength of the U.S. investment community is its diversity
Finance Eleven: What is it like working at OpenAI?
Kenneth: It’s exciting that we get to experience new technology. I have worked in both academic and commercial laboratories. The obvious difference between OpenAI and other research institutions is: other institutions have different research groups, and group members set their own agendas and publish papers in their respective fields; OpenAI also has groups, but they are different The team has the same goal: we share a mission and work toward it together.
Joel: OpenAI is an exciting, fast-growing, pragmatic, and rigorous company. Employees can try out the model in advance, making it feel like they are living in the future. In addition, OpenAI has sufficient computing resources, which is not the case with other research institutions I have worked for. We share the same vision, which is different from other AI organizations or what most people thought at the time - OpenAI believes in scale and that using more data and computing power can enhance the capabilities of the model.
Finance Eleven: What changes have taken place in your work and life after the release of ChatGPT?
Kenneth: My inbox was suddenly flooded with requests for advice. It was like the world was interested in this field, and even fearful - they were worried they were being left behind, so they were looking for authority figures to ask what all this meant. There are also many investors contacting me, wanting to find the next outlet. The success of ChatGPT made them think that anyone related to artificial intelligence and OpenAI might be doing important things.
The thing about GPT is, in a way, it's scary because it captures certain aspects of human cognition quite accurately. This was not only a scientific advance, it was also philosophically important, because these perceptions were what separated humans from everything else in the universe, and they began to function in machines.
Joel: I’ve been contacted by many headhunters, including from China. Also, OpenAI was in the news so frequently that for the first time, my parents were so aware of the company I worked for. This feeling is amazing.
Finance Eleven: Why did you leave OpenAI in the second half of 2022, a few months before the release of ChatGPT?
Joel: I didn't expect ChatGPT to have such a huge impact. OpenAI is a great place to do research, but its research is always pushed in a specific direction; as researchers, Kenneth and I hope to participate in more basic and open (without limiting the direction) exploratory research. . Right then, we both had other suitable job offers, so we left.
Finance Eleven: Some people believe that China’s investment institutions pay too much attention to the business model and profitability of the invested companies and are not as willing to take risks as OpenAI’s investors, so they have stifled innovation, resulting in China’s failure to develop OpenAI and ChatGPT. What do you think?
Kenneth: It is wrong to interpret OpenAI's financing process as "investors should not care about money." The founding team of OpenAI is a world-class talent, so investors are willing to place bets here.
Finance Eleven: What is the venture capital environment in the US AI industry like?
Kenneth: In the United States, there are enough investors who focus on ideas and visions and are willing to take risks, while others only care about profits. The same person's thoughts will also change. For example, if I invested in a company with unclear goals and failed and lost money, then next time I will care more about the business model and profits. In my opinion, the advantage of the American investment community lies in the diversity of viewpoints. This is a healthy environment.
I don't have a deep understanding of China's investment community, but diversity is good everywhere. The venture capital industry in Silicon Valley is very mature, and in a relatively immature environment like Chinese VC—and I don’t mean that in a disparaging way, because few places are as mature as Silicon Valley—I can imagine that people are very focused on profits and business. change. This can also lead to good companies, but they won't be as revolutionary as OpenAI.
In addition, it is not just the investors who decide the investment direction, but also the various institutions and funds behind them. National agencies that provide funding for scientific research should not be afraid of losing money, because scientific research is always full of risks; if there are no losses, it means that they are not actually doing research. But I found that these institutions are more conservative. It's kind of ironic, they ask, "What is the research goal?" and then evaluate the feasibility of the goal and then decide whether to give money. On the contrary, market-oriented investors will have different views. They regard investment as a portfolio and do not care about individual failures, as long as the overall rise can make up for the decline. They sometimes prefer to say, "I don't know what the point of this is, but it sounds cool, I'm going to vote for it."
Transformer has flaws, AGI is still far away
Finance Eleven: Will there be a new architecture after Transformer? (Note: Transformer is a natural language processing model proposed by Google in 2017, and is also the basic architecture of today’s large AI models.)
Kenneth: I don’t believe that Transformer is the final architecture we have reached. This view seems too optimistic from a research perspective. But I can't rule out this possibility. Maybe you can not change the architecture, but only change the training method, the length of the prompt (prompt) and other aspects.
The text in the prompt words is in order, and the neural network does see this order. However, the prompt words that can be input currently are too short, so the Transformer cannot learn in chronological order. Maybe there will be some breakthroughs in the future so that the prompt words can be long enough to cover the entire human history, and then the model can learn from the prompt words. It's a very strange idea, but if it can be implemented, maybe the architecture will remain the same.
Another factor is hardware. Scientists will think of very interesting new architectures, but current hardware may not be able to support them, which limits our scope to try certain solutions.
Joel: Transformer is indeed amazing and can accomplish many tasks, but from historical experience, there will be ways to surpass it.
A basic flaw of the current Transformer is that it does not have the ability to "recurrence". When faced with a problem, humans will review experience to learn. A certain psychological state in the past can be repeated indefinitely, so you can reflect on something for as long as you want. Transformer also has explicit memory of the past, but this memory only lasts for a specific number of tokens (note: the data unit used by AI to process text). This paradigm is powerful enough to equip the model with the ability to learn from input contextual information. But it still can't preserve memories permanently like humans can.
I want to emphasize that I am not saying that machine learning models must function like the human brain, but that is the current mainstream path of research.
Finance Eleven: Some people believe that large models do not have the ability to think like humans, they just appear to be thinking.
Joel: That’s an interesting point, but from a technical perspective, I don’t agree with it. Transformer's ability to perform arithmetic operations without help is very unhuman, and it makes very basic mistakes; but when you talk to it about complex topics that it seems to have no exposure to (such as trying to combine 22 strange philosophical ideas Combined), it can give a pretty impressive response. So I think the reality is probably somewhere in the middle.
Finance Eleven: How to define AGI (artificial general intelligence)? Sam Altman (founder of OpenAI) mentioned in his speech in China that very powerful artificial intelligence systems will appear within ten years, and we need to prepare for it from now on. What do you think?
Kenneth: I'm not too concerned about the precise definition of AGI. We'll know it when we see it. I feel like the word AGI distracts us. The real question is not whether we reach AGI, but whether what we achieve in ten years will have a significant impact on society; and if so, then we need to be prepared whether it is called AGI or something else.
Joel: I agree. The process of scientific development is always non-linear. I don’t know when AGI will arrive. It may be very soon or very slowly. AI will soon surpass humans in some tasks, and AGI in some aspects may soon arrive.
This creates two problems. "Intelligence" is the basis for defining human beings. When something smarter than humans appears, humans are in danger. Another scary thing is that many people get meaning from work. When AI automates work, we need to shift the meaning of life from work to finding pleasure and doing what we like. That's great, but the transition can be tough. Current evidence shows that in the United States, even with social security, people are depressed by unemployment and even abuse drugs. We are not ready for this possible future.
Finance Eleven: When will AGI be realized?
Kenneth: There are many different opinions on this issue. But the reality is, no one can know. There are still some gaps in the field of intelligence that do not yet have clear solutions.
The more mainstream approach to AGI now is to continue to increase data and computing power based on Transformer. Because this scaling method has filled the holes of previous models very well in the past, such as the upgrade from GPT-2 to GPT-3 and then to GPT-4. Based on this, some believe we are at a tipping point for AGI—it just requires further scaling to solve the problem.
But there are some things that cannot be improved through scale, such as innovation. The current model cannot invent a new paradigm the way humans invented rock music. This is a serious deficiency, because the essence of civilization is innovation.
The reason for this flaw is that Transformer learns from data, and things like "innovation" are not in the data. Currently, the data that is fed into the model is not chronologically arranged from beginning to end, but rather a single block of data. This results in a model that lacks the concept of time, and innovation is closely related to time order.
We humans are always at a specific point in time. What happened before this point and what has not happened yet are all default, so we can know what is new and cutting-edge; but the model cannot judge the point in time, it will all These things are all treated as one giant hybrid: it'll see data about cars, it'll see data about spaceships, it'll see data about large language models, but they'll all be there at the same time, in no particular order. Therefore, the model cannot tell what is cutting edge.
Another example is the problem of hallucination (note: "hallucination" refers to false or wrong information generated by AI), that is, how does the language model know what it knows and what it does not know? "Knowing what you remember" is not a linguistic process. If I ask you, what did you eat three weeks ago? You will say you don’t remember. But how do you know you don’t remember? The process can't be expressed in words, you can't say, I checked this and then I checked that so I know I don't remember. This is more of an implicit, innate process that does not involve language; without language, it is not present in the data, and the model cannot obtain this concept from the data.
Now, due to the application of RLHF (Reinforcement Learning from Human Feedback, reinforcement learning from human feedback, that is, humans train the model, the model is rewarded if it does well, and punished if it does not do well), these shortcomings may be corrected. But I think this approach only works if the model truly understands what it should and shouldn't know; and it can only truly understand if that knowledge is implicit in the data. But so far, it appears that no such knowledge exists in the data. For example, to teach a model to be honest, we can punish it when it is dishonest, and then it will become honest. But that doesn’t mean it understands the importance of honesty.
These problems are difficult to solve through scale. There are solutions, but they require more complexity, require new insights and technological breakthroughs, and will not come easily. So AGI is still very far away.
Love, Death and Artificial Intelligence
Finance Eleven: Nowadays, many people in the industry are warning about the threat of AI and emphasizing the control and supervision of AI. The openness of AI you study is to allow AI to produce more new and unexpected results without having a goal. It is about relaxing control. How do you view the contradictions?
Kenneth: Good question. As a science, AI itself is seeking further discovery and innovation. We should embrace openness in research and rely on it to find a balance that allows for continued discovery but is sufficiently constrained.
Joel: The study of openness is important in part because it allows us to scientifically address a question that runs throughout the history of technology: How do we get the most benefit from openness processes while mitigating risks? The same open scientific process that gave us vaccines to cure disease and planes to fly around the world also gave us nuclear weapons. The tension between creativity and control runs deep and requires a lot of thought and research.
Finance Eleven: Are you already an AI-native living state?
Kenneth: No. I do use AI at work, but many things in life, like playing with my kids, would be great without any electronics. I use AI in moderation and life is life.
Joel: I use ChatGPT (with GPT-4) to understand new knowledge domains and Copilot to write code.
But so far, I haven't used ChatGPT in my writing (i.e. I haven't used it to answer these questions of yours). Part of the reason is that it hasn't integrated into my workflow yet. For example, I will use a specific text editor, and it is very troublesome to integrate GPT. Also, I love the artistic nature of writing. Maybe in the future it will become easy enough to integrate into workflows; but I'm also a little resistant because sometimes, you don't want to automate the things you really care about.
Financial Eleven: At present, what is the issue about AI that you are most concerned about?
Kenneth: I worry that we are becoming more and more alienated from our humanity. I became interested in AI when I was 8 years old. What fascinates me the most about it is that it allows me to understand and connect with people better. This may be hard to understand, after all, we seem to only deal with machines all day long, and have no interest in humans. But the process of understanding "intelligence" is actually understanding the essence of human nature. Psychology is also understanding intelligence, but it only studies how it works, not how to build it; and I've always felt that if I can't build something, I can't really understand it.
The case of machines replacing humans confuses me. This is what I call alienation from humanity—the machine becomes rich and interesting enough that you find time to interact with the machine instead of with people. The images we see, the music we hear, and the stories we read were once the product of the rich imagination of human beings, but are now more and more produced by machines.
The specific moment when I was shocked was when I saw DELL-E (Note: OpenAI's image generation product) for the first time. Suddenly, I realized that art can be anything other than self-expression, it can just be art. And I have always believed that the beauty of art lies in the presentation of human nature. At that time, my child was about 7 years old. He loved to draw, and the quality of the paintings produced by this machine far surpassed his own. This leaves me very confused about the future – this cold thing with no real experience will prevent people from appreciating my child’s self-expression. I don't like that kind of world.
Technology like DELL-E can also help humans express themselves, but if it goes in the wrong direction, we will become addicted to inhuman things and stop interacting with humans. At that time, social bonds will break down and governments will collapse. A big challenge before us is how to make artificial intelligence support meaningful human connections instead of encouraging isolation and loneliness.
Joel: Right now I'm most concerned about how quickly language models can be deployed into the world. I worry that institutions (such as legal, educational, economic, news media, and political institutions) are slow to update and cannot adapt as fast as the development of language models.
In the United States, systems such as social media and recommendation engines take up a lot of people's time and energy. These systems have the power to connect us, but they often only pursue user engagement and dwell time, rather than helping us improve our lives.
It seems like all of our institutions turn into gambling in this way. For example, in the United States, in order to win elections, some political candidates no longer conduct honest debates but create ads that target the psychological characteristics of their audiences and arouse their hatred of other parties. Machine learning is involved in many different ways, such as personalized recommendations and information cocoons. This is also one of the main reasons why I shifted my research direction to the intersection of machine learning and philosophy. I hope we can use machine learning in a more beneficial way.
Finance Eleven: Is there any way to prevent these situations from getting worse?
Kenneth: One way is to encourage the development of AI tools that augment human capabilities—turning songs that people sing casually into complete works, promoting high-quality human interaction, and so on. But human nature is hard to control. If communicating with AI is pleasant, people will do so. It's a bit like eating sugar, the sweetness is pleasant, but you have to work hard to control it and not overdo it.
We also need international cooperation to establish review and legal frameworks and identify those responsible. This is also difficult, involving economic issues and international competition - if the independent development of AI can be achieved, a country can completely reform the global economy and gain huge advantages. Therefore, it is very tempting not to cooperate.
Finance Eleven: Many Chinese AI entrepreneurs are troubled by business models and financing. They want to gather resources to create China’s OpenAI. Don't you care about money?
Kenneth: If I were in a startup period where I was desperate for money, I would also value financing; but I seem to have passed that stage - I now believe that as long as I do the right things, I can get financing. So I was more focused on the prospects and worrying about not having evidence to support my ideas than on the money.
I personally feel that "becoming the next OpenAI" is not a wise path, because usually, you can't be the same thing again. What makes this thing great is that it is the first to do it and is unique. Now, the competition in the field of conversational robots is very fierce, and I don’t think I am good enough to win, so I am more inclined to think, is there anything new that is completely different from OpenAI?
Joel: I like money. I was lucky enough to work in a laboratory that paid well. But now, more money doesn't make my life feel more meaningful.
Finance Eleven: What issues about AI are the American public concerned about?
Kenneth: There are all kinds of concerns. Some people focus on long-term risks, such as the threat of human extinction; some people focus on short-term risks, such as employment issues. Then there are those who worry about everything at the same time. Some people are angry at others who believe that their unimportant concerns are attracting so much attention that they distract from what really matters. There is currently a lack of consensus. I think people are sorting their minds and deciding what are the most important issues that deserve a lot of our time and energy.
Finance Eleven: What is the main content of the paper "Love of Machines"?
Joel: "Artificial intelligence" is to implant our understanding of "intelligence" into machines and try to understand the nature of intelligence; "artificial life" is to abstract "life" and simulate biological evolution in computers. I try to apply the same idea to "love", and abstract the concept of "love", so that the machine can express "love". In practice, this means combining methods from machine learning with those from fields that study love (such as philosophy, spirituality, psychotherapy, etc.).
One of the problems with machine learning is that it treats humans in a very narrow way, seeing satisfying preferences as the only way to promote human flourishing. From this perspective, it becomes understandable that social media values user engagement time: In the eyes of machine learning, you are a completely rational individual, so the more time you spend on social media, it means that it provides something of value.
However, humans are rich and complex. We can be addicted to things and do things that are against our own interests; we have willpower and sometimes it takes a struggle to do what we really want to do.
In this paper, I adopt a richer human behavior model (similar to Maslow's hierarchy of needs theory) and try to use the language model to make machines respect and promote human development instead of just giving narrow satisfaction.
Finance Eleven: This sounds closely related to the AI issue you are most concerned about today. Are you looking for a solution to a problem?
Joel: I don’t claim to have a solution, but I hope to find a positive way forward. I'm very interested in humans, human psychology, and how the human realm and the machine realm can be effectively combined. Technology is here for humanity, for our benefit, for our prosperity, but sometimes, it’s easy to forget that.