Heart of the Machine Report
Editors: Chen Ping, Du Wei
As for the reason, on the one hand, the large model needs to jump out of the scope of expanding the parameter scale, and on the other hand, it is limited by the high training cost.
The powerful generative conversational capabilities of OpenAI ChatGPT have sparked renewed interest and investment in AI in recent months. With the upsurge of ChatGPT-like research and development at home and abroad, conversational AI and the large model behind it are favored by more and more people.
But in a video speech at MIT last week, OpenAI CEO Sam Altman made another statement. He warns that the research strategy that gave birth to ChatGPT is over . It's unclear where future progress will take place.
Over the past few years, OpenAI has made a series of impressive advances in the field of language-related artificial intelligence by taking existing machine learning algorithms and scaling them to previously unimaginable scales. Among the latest advances, GPT-4 may be trained using trillions of text words and thousands of powerful computer chips.
GPT-4 performs well on tasks as diverse as language, mathematics, programming, vision, medicine, law, and psychology. Sam Altman said that making the model bigger (such as a giant model) will not bring new progress .
Sam Altman's comments suggest an unexpected turn in the race to develop and deploy new AI algorithms . Since the launch of ChatGPT in November last year, Microsoft has used the underlying technology to add a chatbot based on it to Bing search, and Google has also launched a Bard competitor. Many people are eager to try new types of chatbots to help with work or personal tasks.
Meanwhile, a host of well-funded startups, including Anthropic, AI21, Cohere, and Character.AI, are pouring resources into building ever-larger algorithms in an effort to catch up to OpenAI.
However, Sam Altman's words seem to indicate that GPT-4 may be the last major development in OpenAI's strategy of developing larger models and feeding them more data. This can also be seen from the rumors that he personally refuted GPT-5 . He said that he has not trained GPT-5 at present , and he will not train in the short term .
In the paper describing GPT-4, OpenAI predicts diminishing returns in scaling model size. There are also physical limitations on how many data centers OpenAI can build and how fast they can build them.
In addition to the reasons for the large model design and architecture itself, the huge cost is also a major reason. According to reports, ChatGPT needs at least 10,000 GPUs for training, and more resources are needed to ensure continuous operation. And Nvidia's latest H100 GPU designed for AI and high-performance computing has a unit price of $30,603, which is undoubtedly expensive. By Altman's own admission, GPT-4 cost more than $100 million to train .
Nick Frosst, co-founder of Cohere (who previously worked on AI at Google), said Altman's notion that scaling doesn't always work sounds right . He also believes that the progress of Transformer (the core machine learning model type of GPT-4 and competing products) has exceeded the scope of expansion. Besides adding more parameters to the model, there are many other ways to make Transformer better and more useful.
New AI model designs or architectures and further fine-tuning based on human feedback are promising directions already being explored by many researchers.
Musk named the AI project TruthGPT
On Musk’s side, although he said he was worried that GPT-4 would cause social problems, he signed a moratorium on the development of a system more powerful than GPT-4 for at least six months. But the reality is that Musk is working on creating a new artificial intelligence startup X.AI, which will compete directly with OpenAI.
According to reports, Musk has obtained tens of thousands of GPUs from Nvidia, and these systems power the calculations required for intensive tasks such as artificial intelligence and high-end graphics processing.
Musk confirmed reports that he plans to create an artificial intelligence startup to compete with ChatGPT. Musk announced the plan in an interview with Fox, saying, “I’m going to create an AI platform called TruthGPT, or a maximally truth-seeking AI that can understand the nature of the universe. I Thinking that might be the best path to (AI) safety, in the sense that an AI that cares about understanding the universe is less likely to exterminate humanity, because we're an interesting part of the universe."
Musk also said in the interview that he believes artificial intelligence has the potential to destroy civilization, no matter how small.
Previously, Musk co-founded OpenAI with current OpenAI CEO Sam Altman and others in 2015, but Musk left the company's board of directors in 2018. It is understood that the reason Musk stated at the time was to avoid a potential conflict of interest with Tesla, but Musk later said there were other reasons, and another reason was that he disagreed with some of the things the OpenAI team was doing.
Elsewhere, Musk tried to take over as the company's CEO in 2018, but left after Altman and others declined, Semafor? reported in March.
Since resigning from the OpenAI board, Musk has repeatedly criticized OpenAI. Just in February of this year, he also stated that the current OpenAI is not at all what he hoped for, and called OpenAI a Microsoft-controlled, closed-source, profit-maximizing company.”
Faced with Musk's repeated attacks, Altman said that even so, he still regards Musk as one of his idols, and said, "I believe Musk does feel pressure on AGI security, which is understandable."
It is worth mentioning that this is not the first time Musk has considered building TruthGPT. He tweeted in February that what we need is TruthGPT.
Regarding more details about TruthGPT, Musk did not disclose more information, and we can look forward to a follow-up wave.
Reference link: