Source | Heart of the Machine ID | almosthuman2014
Google's internal documents have been leaked again, this time the training details of Google's new generation of large model PaLM 2: the amount of training data is nearly 5 times that of the previous generation, and the amount of parameters is about two-thirds of the previous generation.
Last Thursday, at the 2023 Google I/O conference, Google CEO Pichai announced the launch of PaLM 2, a large model that benchmarks GPT-4 , and officially released a preview version that improves mathematics, code, reasoning, multilingual translation and natural language generation ability.
The PaLM 2 model provides four versions of different sizes, from small to large in order of Gecko, Otter, Bison, and Unicorn, which are easier to deploy for various use cases. Among them, the lightweight Gecko model can run on mobile devices at a very fast speed, and can run excellent interactive applications on devices without networking.
However, at the meeting, Google did not give specific technical details about PaLM 2, only stating that it is built on top of Google's latest JAX and TPU v4.
Yesterday, according to internal documents seen by foreign media CNBC, PaLM 2 was trained on 3.6 trillion tokens . For comparison, the previous generation PaLM was trained with 780 billion tokens.
In addition, Google previously stated that PaLM 2 is smaller than previous LLMs, which means it can be more efficient while completing more complex tasks. This point has also been verified in internal documents. The number of training parameters of PaLM 2 is 340 billion , which is much smaller than PaLM's 540 billion.
How does the training token and parameter amount of PaLM 2 compare with other LLMs? For comparison, LLaMA released by Meta in February was trained on 1.4 trillion tokens. OpenAI's 175 billion parameter GPT-3 is trained on 300 billion tokens.
While Google has been eager to demonstrate the capabilities of its AI technology and how it can be embedded in search, email, document processing and spreadsheets, it has also been reluctant to release the size of its training data or other details. In fact, it is not only Google that does this, OpenAI also keeps silent about the details of its latest multi-modal large model GPT-4. They all said the non-disclosure of details stemmed from the competitive nature of the business.
Still, as the AI arms race continues to heat up, the research community is increasingly demanding more transparency. And in a Google internal document leaked some time ago, Google's internal researchers expressed such a point of view: Although it seems that OpenAI and Google are chasing after each other on the AI model, the real winner may not come from the two. Home, because the third-party force "open source" is quietly rising.
At present, the authenticity of this internal document has not been verified, and Google has not commented on the relevant content.
Reviews
At the beginning of the official announcement of PaLM 2, some netizens predicted its parameters according to Chinchilla's law. Ta predicted that the parameters of the PaLM 2 model family ranged from 80B / 90B / 100B, which is still far from the 340B that was revealed this time.
Someone also made a wave of predictions on the training cost of PaLM 2. According to the development of large models in the past, this netizen said that it will cost 100 million US dollars to build PaLM 2.
PaLM 2 parameters are leaked, you can try to guess Bard, this netizen said:
With the leak of the number of PaLM 2 tokens, netizens can't help but wonder, how many tokens can usher in a big turning point before the arrival of AGI?
Reference link: https://www.cnbc.com/2023/05/16/googles-palm-2-uses-nearly-five-times-more-text-data-than-predecessor.html