"Our large AI model, trained on the Wanka cluster, makes an error once every three hours. Don't laugh, this is already the world's advanced level." At an industry summit, an academician scientist from Tsinghua University said this The "big truth" of AI large model training.
The large AI models that are popular all over the world are undoubtedly the hot topic this year, and their number continues to grow, reaching an astonishing level. Amidst the “hundreds of rivals vying for power”, everyone often overlooks a key issue: the torrent of data brought by large AI models is more turbulent than imagined.
"An error occurs once every three hours" sounds like an incredible failure rate, but it is the norm faced by practitioners of large models, even "top students". The current common practice in the industry is to write fault-tolerant checkpoints. Since an error is reported within three hours, we should stop every 2.5 hours, write checkpoints, save the data, and then start training again. Once a failure occurs, you can recover from the written checkpoints to avoid "starting from scratch" and doing everything in vain. The checkpoint needs to store a lot of data and will consume a lot of time. The academician team developed a large model based on the llama 2 architecture. It takes ten hours to store the data in the hardware once. The storage efficiency directly affects the development progress.
If large-scale heterogeneous data is a torrent that surges wantonly, the storage system is a river carrying data flow. Its width and solidity directly determines whether the data will be blocked or even stagnant, thus blocking the lifeline of the large AI model. It can be said that the productivity and efficiency of the entire large model industry are "upper-limited" by storage.
This is why storage, as an AI data infrastructure, has received more and more attention.
On November 29, the "Digital Innovation AI Future" 2023 China Data and Storage Summit was held in Beijing. Sugon Storage has released a storage solution for large AI models.
Let’s take this opportunity to learn about the load-carrying challenges brought to storage by the wave of AI large models, and how Sugon Storage is leading the way for the smart industry and boosting the success of hundreds of AI large models.
AI large models enter the deep water area of the industry
The Pain of Traditionally Stored Data
I recently went to Yunnan and found that not only are large-scale model construction in full swing in science and technology hubs such as Beijing, Shanghai, and Guangzhou, but also in second- and third-tier cities such as Kunming and Dali, and even in border areas, large-scale model industry applications are actively being explored.
As all walks of life move towards intelligence, almost all of them have ignited a burning interest in large models. At this time, a key issue also emerged: the industrialization trend of large AI models requires upgrading the storage infrastructure.
Every time model developers train, the data poses various challenges to the storage system:
1. The impact of the data flood. With the industrial implementation of large models, many industries have begun to train exclusive models. A large amount of industry data, proprietary data, and new annotated data are fed to large models. The huge amount of data has a huge impact on the storage system. A challenge was posed. A data technology company in Yunnan mentioned that large industry models need to be trained with high-quality data sets, documents, and customer private data. A separate annotation group is established for each project. The scale of data continues to increase, and storage requirements and costs also increase. Increase.
2. The shackles of data congestion. Very large-scale data preprocessing is slow and time-consuming. The collection, classification, relocation and other processes are time-consuming and laborious. Once the storage performance cannot keep up, the throughput of massive files is slow, more reading and less writing, check Clicking Checkpoint and waiting for a long time will delay the development progress and increase development costs.
3. The undercurrent of complex data. In addition, large AI models use a large amount of heterogeneous data, with complex file formats and diverse data set types. The amount of data increases sharply. Traditional storage is difficult to cope with the challenge of data complexity and is prone to indigestion. Problems result in low data access efficiency, resulting in reduced model operating efficiency, increased training computing power consumption, and the inability to fully "squeeze" expensive GPU computing resources. For example, the local solar observatory in Yunnan uses AI scientific computing models to learn massive images to present the true appearance of the sun, generating 2TB of image data every day. The current storage throughput efficiency is low, which will lead to slow loading of training sets and long data processing cycles. Slow down the research process.
4. Concerns about data security. At present, large AI models have penetrated deeply into various industries, requiring massive data support in the process of training, development and application implementation, including data containing sensitive industry or personal information. If there is no reasonable Data desensitization and data hosting mechanisms may cause data leakage and cause losses to industries and individuals. At the same time, model security risks also need to be taken seriously. For example, plug-ins may be implanted with harmful content and become a tool for criminals to commit fraud and "poisoning", endangering social and industrial security.
AI large-scale models are heading into the deep water zone of the industry. What is gratifying is that this technological innovation is being highly integrated into thousands of industries, meeting the needs for intelligence, and has strong vitality. What is worrying is that data engineering runs through the entire life cycle of large models, requiring the use of large amounts of data in various stages from collection, cleaning, training, inference deployment, feedback tuning, etc. Storage has become a bottleneck, which means that all stages of large AI models need to be consumed by a large amount of data congestion, failures, and inefficiencies. This will make the development cycle and comprehensive cost of large models extremely high, which is unbearable for the industry.
Dredging the storage "river" to avoid data siltation and provide support and nourishment for the big model industry. The new solutions brought by Sugon Storage have allowed us to discover valuable reference cases.
High-quality data “channel”
Sugon Storage gives the large model industry an answer
After communicating with developers of large AI models, I came to a clear conclusion: building a new storage system that adapts to large AI models is no longer an issue that needs to be discussed. The key is who can take the lead in completing the solution upgrade and providing Come up with practical solutions.
Having insight into the storage needs of the industry, Sugon Storage created an AI large model storage solution based on ParaStor large model dedicated storage, and wrote its own answer.
Sugon Storage AI large model storage cluster has three leading capabilities: heterogeneous fusion, ultimate performance and native security.
First of all, it can provide hundreds of billions of file storage services, with nearly unlimited expansion scale. In response to the issue of data access protocol diversity, it supports multiple storage protocols such as files and objects at the same time to avoid data duplication across storage systems.
Secondly, in response to the high demand for data processing efficiency during the development of AI large models, Sugon Storage AI large model storage cluster can provide multiple data IO performance optimization capabilities such as multi-level cache acceleration, XDS data acceleration and intelligent high-speed routing.
Finally, in order to ensure data security throughout the entire process, Sugon storage nodes also provide chip-level security capabilities and support the national secret instruction set. Through multi-level reliability, the storage cluster is guaranteed to operate stably throughout the training and development cycle, in line with policies and future security trends. .
Someone may ask, there are so many storage solutions on the market, and some also advertise to provide professional support for model development. What are the differentiated values of Sugon Storage’s solutions?
If you are confused about the technical terms and product details of each company, you may wish to use a few words to remember the differentiated value of Sugon Storage AI large model storage cluster:
1. Advanced. Heterogeneous fusion, ultimate performance, and chip-level native security demonstrate the technological advancement of Sugon Storage, and also specifically solve the problems of large data volume, complex and diverse data forms, and throughput in large model development. There are real pain points such as low efficiency and long storage and calculation time.
2. Reliable. The high-performance AI data infrastructure is based on Sugon Storage’s self-developed innovation and is more reliable and secure. It is in line with the information innovation policy and future security trends. It can help domestic large-model service providers avoid overseas supply chain risks, thus Supply chain security, data security, model security and other perspectives protect the development of the large model industry.
3. Comprehensive. Sugon Storage has created a full-dimensional AI solution covering network, computing and platform, supporting stable operation throughout the training and development cycle, reducing overall costs, and making it hassle-free for large model developers and industry customers. Move forward with worries.
To sum up, on the high-quality "channel" built by Sugon Storage, large-scale data can be efficiently processed and the development of large AI models can be accelerated. Therefore, industries and enterprises can be one step ahead, deeply integrate large models with vertical scenarios and businesses, and be the first to obtain A ticket to the intelligent age.
A new starting point for the fifth paradigm
Watch hundreds of companies quarreling and thousands of businesses setting sail
Turing Award winner Jim Gray once proposed the fourth paradigm, whose core is data-driven. With the "emergence of intelligence" in large language models, the fifth paradigm of "intelligence-driven" focuses more on the organic combination of data and intelligence, becoming the new underlying logic supporting scientific revolution and industrial revolution.
All the past is a prologue. This is true for AI, and so is storage.
At this conference, Hui Runhai, president of Sugon Storage Company, was awarded the title of "Storage Pioneer" with 20 years of industry experience and leading practices in AI storage technology breakthroughs, liquid cooling storage research and development and other fields. Under his leadership, Sugon distributed file storage has continued to lead the market for many years, ranking among the top in market share. Data storage solutions for AI large models have once again brought Sugon Storage to the forefront of the times.
Sugon Storage's AI large model storage cluster is actively practicing paradigm shift, corresponding to the new data paradigm, and using the leap in data infrastructure to promote the rising tide of large model industrialization.
Next, in the new paradigm and new starting point of the storage industry, on the high-quality data "river" of Sugon Storage, we will see hundreds of large industry models competing for the stream, and thousands of AI applications racing, accelerating towards smart China.
