Microsoft founder and former CEO Steve Ballmer once said: "Linux is a cancer."
"But it creates good competition, it forces us to innovate, it forces us to justify the price and the value that we offer."
Does the era of large models still need open source?
A few days ago, a document from within Google leaked on the Internet.
The author of the document is a researcher at Google. He believes that the closed strategy of Google and OpenAI will inevitably lead to falling behind open source models (such as Meta's LLaMA, etc.), and even OpenAI cannot match the power of open source.
He explained in detail the reasons for making this judgment in the document, and these views have aroused a lot of discussion in the technical community and the AI industry.
We put the original translation at the end of this article.
01
Discussions in the technical community
After the document was released, it quickly triggered a lot of discussions in the technology and technical communities. We organize some of our views as follows:
I think Satya Nadella said it well in his interview: Advertising revenue, especially from search, is incremental for Microsoft; for Google, it's everything .
So while Microsoft is willing to command lower margins on search ads to win Google's market share, Google has to defend all of their margins -- or they'll be far less profitable in their core business.
The cost of LLM is much higher than traditional search, and Google can't just replace their existing product line with LLM: it actually improves their bottom line.
Microsoft is willing to replace the existing Bing with a "new Bing" based on OpenAI technology, because they make relatively little money on search, and winning market share will make up for the small profit in market share.
Google is in a dilemma: either they dramatically increase their cost of revenue to defend market share, or they risk losing market share in their core business. Just like Google once ate Yahoo with PageRank, it may in the future be eaten by disruptive technologies like LLM.The fastest way for anyone to bring themselves to the state of the art is to buy OpenAI's services. The existential risk for OpenAI is whether they continue to be (semi)independent, not whether they shut down.
LLM is better than Google at search (for open-ended questions) and that's where most of Google's revenue comes from. So it actually gives new companies like OpenAI the opportunity to redefine where consumers go.
OpenAI is at existential risk, not Google. Google will catch up and will have the coverage/subsidy advantage. This so-called "competition" from open source will be free labor. Any winning ideas are ported to Google's products within a short period of time.
Google could bet on megamodels as it did YouTube. At the time, the cost of running YouTube was staggering, costing billions of dollars. But Google can see where things are headed: a triple trend of falling storage costs, bandwidth and transport costs. If there is enough demand, dedicated, low-cost hardware dedicated to LLM will emerge.
Google will not fail, they will only provide the computing infrastructure for the people who build AI products.
When Google complains about not having a moat, they are complaining about not having a moat big enough to sustain a company as big as Google.
Honestly, I don't see a chance for Google to fail. Like every other tech giant, they sit on a ridiculously large war chest. Worst-case scenario, they can wait for the space to stabilize a bit, and then spend billions buying up the market leader. If AI does indeed pose an existential threat to their business prospects, it's a no-brainer to spend their reserves on it.
OpenAI is more of a laboratory than a company. In a sense, they are a bit like the lab department that invented the computer mouse, and commercialization is not on their radar.
If OpenAI can win the developer market with cheap API access and better products, then distribution becomes through third parties, and everyone else will be products that send training data back to the model.
No "open source" model is open source in the classic sense. They're free, but they're not source code; they're closer to freely distributable compiled binaries, where the compiler and raw input haven't been released yet. A true open source AI model requires specifying the training data and code from training data to the model. Of course, for others, it is very expensive to get this information, build the model again and verify that the same results are obtained, maybe we don't really need it.
02
Google must open source
Closure is not a barrier
Former Google scientist, Li Zhifei, the founder and CEO of Wenmeng, who just released the large-scale model "Serial Monkey", has recently criticized and analyzed Google's large-scale model strategy. He also expressed his own views on this leaked document .
Google's open source imperative. The reasons are as follows:
01. If Google doesn’t do it, others will
Every company has what it thinks is the most precious and core thing, and it is unwilling to show it to everyone, but these things are not necessarily so precious to competitors. For example, the large model may be Google's treasure, but the Meta next door is not so important-anyway, the Metaverse has suffered such a loss, so it is better to take this opportunity to open source the large model, so LLaMA has the limelight. Even if Meta quits, or does not insist on doing it, there must be other latecomers (such as Amazon) who are eyeing it. This is the law and fate of Internet technology.
02. No matter how good Google is, it can’t match the collective wisdom
Technology in the United States is very diverse. Although giants such as Google have gathered a large number of AI talents, universities and other non-giant companies (including start-up companies) have also gathered many capable and smart talents. The current situation is that large models have attracted the attention of the whole society. Not only the AI industry, but also elite talents from all walks of life have poured into the large model industry. With such a high degree of social attention, talent density and strength, the result is bound to be extremely prosperous innovation and extremely fast ecological evolution. If Google does not open source, it will not be able to use more powerful collective intelligence to polish its own large model and build its own ecology.
03. Open source is a must for latecomers
Obviously, in the field of large models, Google has lagged far behind OpenAI in terms of momentum. In terms of open source ecology, if Google hesitates again, even if it wants to open source later, it will lag behind Meta's LLaMA by a large margin. On the contrary, if it is open source, the barriers of OpenAI can be further lowered, and it will attract (or retain) many senior talents who support open source. If Google is open source, it will definitely make OpenAI extremely painful. On the one hand, its business model of relying on model API fees will be greatly reduced. On the other hand, it does not have Google's high-intensity investment in supporting large-scale model development through other business models. To put it vulgarly, the pain of the enemy is one's own happiness, which can also make Google feel better, and don't be fooled.
04. Open source does not mean that you cannot build barriers
Some people may say, isn't there no barriers for Google to open source? First of all, Google is rich in resources. I once suggested that the internal forces can be divided into three forces: the internal upgrade phalanx, the open source phalanx, and the exploration phalanx. So open source is not the whole of Google, but it can use open source to leverage the collective wisdom of society to attack OpenAI.
Secondly, the variety of large models does not lie in the costly pre-training model (this is a part suitable for open source), but in the fine-tuning of various models and the optimization of engineering deployment for application implementation. Google has its own rich and colorful application scenarios. Many barriers can be built at the product level including data flywheels.
Finally, in an era when technology is becoming more and more popular, the biggest barrier is not established by being the first to make a big move and then self-enclosed. Building barriers requires long-term thinking, continuous investment and iteration on a difficult matter. Google actually has a tradition of open source, and is also a beneficiary of open source, such as its Android and Tensorflow ecology, so Google should maintain this tradition, use its own advantages, and reappear in the AGI era.
The following is the original SemiAnalysis of the leaked document, translated by AI and proofread by Founder Park.
This article is a document leaked on a public Discord forum, from a researcher within Google. The website that published the article, SemiAnalysis, has verified the authenticity of the document.
"This document represents the personal views of Google employees, not the company as a whole. We disagree with the following, and neither do other researchers we have consulted with."
01
OpenAI doesn't work either
We've been closely monitoring what OpenAI is doing. Who will pass the next milestone? What's next?
But the uncomfortable truth is that we didn't take the lead, and neither did OpenAI. While we bickered, a third party quietly ate our lunch.
Of course, I am referring to open source. Frankly, they are outrunning us. What we consider "significant open problems" have already been solved and applied in some hands. Just to name a few:
LLMs on phones : People are running base models on Pixel 6 at 5 tokens per second.
Scalable Personal AI : You can fine-tune your personal AI overnight on your laptop.
Responsible Release : This one is not "solved", but "obvious". There are sites full of painted models with no limits, and LLM is not far behind.
Multimodal : The current multimodal ScienceQA SOTA trains in under an hour.
Although our model still has a slight edge in terms of quality, the gap is closing at an impressive rate. The open source model is faster, more customizable, more private, and more powerful. They're doing things with $100 and 13B parameters, and we're still struggling with $10 million and 540B parameters. Also, they did it in weeks, not months. This has profound implications for us:
We have no secret formula . Our best hope is to learn and collaborate from what others other than Google are doing. We should prioritize enabling third-party integrations.
People wouldn't pay for a restricted model if the free, unlimited alternatives were of similar quality . We should think about where our real added value lies.
Giant megamodels are slowing us down . In the long run, the best models are those that can be iterated quickly. We should let the small variants be the focus, now we know what is likely to happen in the < 20B parameter range.
02
What happened to the open source community?
In early March, Meta's LLaMA model appeared in the open source community. It's a very powerful base model with no instruction or session tuning and no RLHF. Nonetheless, the open source community immediately recognized the importance of this stuff.
What followed was a flurry of innovations, with major developments every few days (see timeline for full details). Now, just a month later, there are variants with instruction tuning, quantization, quality improvement, human evaluation, multimodality, RLHF, many of which build upon each other.
Most importantly, they solve the scaling problem so that anyone can experiment. Many new ideas come from ordinary people. The bar for training and experimentation has been lowered from the total output of a major research institution to one person, one night, and one powerful laptop.
03
LLM's Stable Diffusion Moment
In many ways, it shouldn't surprise people.
The renaissance of open source LLMs has followed the renaissance of image generation. These parallels have not been overlooked by the community, with many calling it LLMs' "Stable Diffusion moment".
In both cases, the public has the opportunity to participate at low cost, thanks to a much-lower-cost fine-tuning mechanism called low-rank adaptation (LoRA), coupled with a major breakthrough in scale ( Latent Diffusion for Image Synthesis, Chinchilla for LLMs). In both cases, access to high-quality models sparked ideas and iterations from individuals and institutions around the world. In both cases, this quickly surpassed the larger players.
These contributions are crucial in the field of image generation and set Stable Diffusion on a different path than Dall-E. The open model led to product integration, marketing, user interface and innovation, none of which happened with Dall-E.
The effect is clear: rapid dominance in terms of cultural impact, making OpenAI's solutions increasingly irrelevant by comparison. Whether the same will be the case for LLMs remains to be seen, but the broad structural elements are the same.
04
What Google missed
The recent success of open source software has been inseparable from leading innovations in technologies that directly address problems we are still struggling with. Paying more attention to their work can help us avoid reinventing the wheel.
LoRA is a very powerful technology that we should pay more attention to.
LoRA works by representing model updates as a low-rank decomposition, which reduces the size of the update matrix by a factor of thousands. This greatly reduces the cost and time of model fine-tuning. Being able to personalize language models in a matter of hours on consumer-grade hardware is important, especially for ambitions that involve incorporating new and diverse knowledge in near real-time. Even though this technology directly impacts our most ambitious projects, it remains underutilized within Google.
Retraining a model from scratch is a tough road.
LoRA works in part because, like other forms of fine-tuning, it is stackable. Improvements like instruction fine-tuning can be applied and then added to dialogue, reasoning, or tool usage as other contributors. While individual fine-tuning is low-rank, their sum need not be low-rank, allowing full rank updates of the model to accumulate over time.
This means that, as new and better datasets and tasks become available, models can be kept up to date cheaply without paying the cost of a full run.
In contrast, training a giant model from scratch discards not only pre-training, but any iterative improvements made on top of it. In an open-source world, these improvements can quickly take over, making a complete retrain extremely expensive.
We should carefully consider whether each new application or idea really requires an entirely new model. If we do have significant architectural improvements where model weights cannot be reused directly, then we should invest in a more aggressive form of distillation to preserve as much as possible the capabilities of the previous generation.
If we can iterate on small models faster, there is no advantage to large models in the long run.
The cost of LoRA updates is very cheap (~$100) for the most popular model sizes. This means that almost anyone with an idea can generate and distribute a model. Training times within a day are normal. At this pace, it doesn't take long before the cumulative effect of all these fine-tuning overcomes the disadvantage of starting small. In fact, in terms of engineers' time, these models improve far faster than we can do with the largest variants, and the best models are already indistinguishable from ChatGPT. Focusing on maintaining some of the largest models in the world can actually put us at a disadvantage.
05
Data quality is more important than data size
Many projects save time by training on small, highly curated datasets. This suggests some flexibility in the data scaling laws. These datasets exist out of ideas in the book Data Isn't What You Think It Is, and they're fast becoming the standard way of doing training outside of Google. These datasets were built using synthetic methods (e.g., filtering out the best responses from existing models) and recycled from other projects, neither of which had an advantage at Google. Fortunately, these high-quality datasets are open source and thus freely available.
Competing directly with open source is doomed.
This latest development has direct, immediate implications for our commercial strategy. Who would pay for a Google product if there was a free, high-quality alternative with no usage limits?
We should not expect to be able to catch up. There's a reason the modern internet works because of open source. Open source has some significant advantages that we cannot replicate.
06
We need open source more than open source models
Keeping our technology secret has always been a shaky proposition. Google researchers often leave for other companies, so we can assume they know everything we know and will continue to do so as long as the channel is open.
However, maintaining a technological competitive advantage has become more difficult, as cutting-edge research at LLM is now more affordable. Research institutions around the world are building on each other's work to explore solution spaces in a breadth-first fashion far beyond our own capabilities. We can try to hold on to our secrets while outside innovations diminish their value, or we can try to learn from each other.
07
Individuals are not subject to the same restrictions as businesses
Many innovations are based on leaked meta model weights. While that will inevitably change as truly open models get better, the point is they don't have to wait. Individuals were able to acquire these technologies as they became popular, thanks to legal guarantees of "personal use" rights and the difficulty of prosecuting individuals.
08
Becoming your own customer means you understand the user
There is a wide range of creativity when browsing the models people have created in the field of image generation, from animation generators to HDR landscapes. These models are used and created by those deeply immersed in their particular sub-genre, with a depth of knowledge and empathy that we cannot match.
09
Owning the Ecosystem: Making Open Source Work for Us
However, the only beneficiary is Meta Corporation. Since the leaked model belongs to them, they have effectively captured the value of free labor worldwide. Since most open source innovations are built on top of their architecture, they can incorporate it directly into their products and no one can stop them.
The value of owning an ecosystem is self-evident. Google itself has successfully applied this model to its open source products such as Chrome and Android. By owning the platform on which innovation happens, Google cements itself as a thought leader and direction setter, gaining the ability to shape narratives of ideas larger than itself.
The tighter we control the model, the more interest will be drawn in opting for open source alternatives. Both Google and OpenAI have adopted a defensive release model that allows them to maintain tight control over how their models are used. However, this control is fictional. Anyone who wants to use LLM for unauthorized purposes is free to choose freely available models.
Google should be a leader in the open source community, taking the lead by collaborating rather than ignoring the broader conversation. This might mean taking uncomfortable steps like publishing model weights for small ULM variants. This necessarily means giving up some control over our model. But such compromises are inevitable. We cannot hope to both drive innovation and control it.
10
Conclusion: What should be the attitude towards OpenAI?
All this talk of open source might feel unfair given OpenAI's current closed policy. If they are not willing to share, why should we share? But the truth is, we've shared everything with them through a steady attrition of senior researchers. Secrecy is pointless until we stop this drain.
Babbitt Park is open for cooperation!
Chinese Twitter: https://twitter.com/8BTC_OFFICIAL
English Twitter: https://twitter.com/btcinchina
Discord community: https://discord.gg/defidao
Telegram channel: https://t.me/Mute_8btc
Telegram community: https://t.me/news_8btc
