On October 30, Kunlun Wanwei announced the open source of the tens of billions-level large language model "Tiangong" Skywork-13B series, and rare open source of a large high-quality open source Chinese data set of 600GB and 150B Tokens.
Kunlun Wanwei's "Tiangong" Skywork-13B series currently includes two major models with 13 billion parameters : Skywork-13B-Base model and Skywork-13B-Math model . They have performed well in many authoritative evaluations and benchmark tests such as CEVAL and GSM8K. It shows the best effect of models of the same scale , and its Chinese ability is particularly outstanding. Its performance in Chinese technology, finance, government affairs and other fields is higher than other open source models.
Skywork-13B download address (Model Scope): https://modelscope.cn/organization/skywork
Skywork-13B download address (Github): https://github.com/SkyworkAI/Skywork
In addition to open source models, the Skywork-13B series of large models will also open source the high-quality Chinese corpus data set Skypile/Chinese-Web-Text-150B with 600GB and 150B Tokens, which is currently one of the largest open source Chinese data sets.
At the same time, Kunlun Wanwei's "Tiangong" Skywork-13B series of large models will soon be fully open for commercial use - developers do not need to apply for commercial use.
13 billion parameters, two major models, one of the largest Chinese data sets, fully open for commercial use. Kunlun Wanwei's "Tiangong" Skywork-13B series of large models can be called the industry's most thorough open source high-quality commercial models worth tens of billions.
The open source of Skywork-13B series large models will provide the best technical support for the scene application of large models and the development of open source communities, lower the commercial threshold of large models, promote the implementation of artificial intelligence technology in thousands of industries, contribute to the construction of artificial intelligence ecology, and join hands with the open source community Explore the unknown world and create a better future.
Two major models leading the industry
Kunlun Wanwei's "Tiangong" Skywork-13B series includes two major models and a 150B high-quality Chinese data set.
- The Skywork-13B-Base model is the basic model of Skywork-13B. It has been trained with 3.2 trillion multi-language high-quality data. It has demonstrated the best performance of models of the same size in CEVAL, CMMLU, MMLU, GSM8K and other evaluation and benchmark tests. Effect.
- The Skywork-13B-Math model has undergone specialized mathematical ability-enhanced training and achieved the best results of models of the same size on data sets such as GSM8K.
- Skypile/Chinese-Web-Text-150B dataset. This dataset is high-quality data filtered from Chinese web pages according to our carefully filtered data processing process. The size of the open source data set this time is about 600GB, and the total number of tokens is about 150B. It is currently one of the largest open source Chinese data sets.
In addition, the Skywork-13B series also discloses the evaluation methods used in the model, data ratio research and training infrastructure tuning solutions. It is hoped that these open source contents can further inspire the community's understanding of large-scale model pre-training and promote the realization of artificial general intelligence (AGI).
Five Characteristics Comprehensive Transcendence
Kunlun Wanwei's "Tiangong" Skywork-13B series of large models have demonstrated the best results of models of the same size in many authoritative evaluations and benchmark tests such as CEVAL and GSM8K. Its Chinese ability is particularly outstanding, and it is widely used in Chinese technology, finance, and government affairs. The performance in other fields is higher than other open source models.
Five major features of Skywork-13B series models:
- Strongest parameter performance: comprehensively surpassing large models of the same size
This open source Skywork-13B series model comprehensively surpasses large open source models such as LLaMA2-13B in several authoritative evaluation benchmarks such as CEVAL, CMMLU, MMLU, GSM8K, etc., and achieves the best results among large models of the same scale. (Data as of October 25)
- Maximum training data: 3.2T high-quality multi-language training data
The Skywork-13B series large models have 13 billion parameters and 3.2 trillion high-quality multi-language training data. The model generation ability, creative ability and mathematical reasoning ability have been significantly improved.
- The strongest Chinese language modeling capability: Chinese language modeling perplexity evaluation, surpassing all Chinese open source models
The Skywork-13B series large models perform well in Chinese language modeling capabilities and have excellent Chinese cultural creative capabilities. In the evaluation in the field of Chinese text creation, the Skywork-13B series large models have demonstrated outstanding capabilities, especially in fields such as technology, finance, government affairs, corporate services, cultural creation, games, etc., which have performed higher than other open source models in the industry.
- One of the largest Chinese open source data sets: 150B Tokens high-quality Chinese corpus
The Skywork-13B series will be equipped with the open source 600GB, 150B Tokens high-quality Chinese corpus data set Skypile/Chinese-Web-Text-150B, which is currently one of the largest open source Chinese data sets. Developers can draw on the large model pre-training process and experience in the technical report to the greatest extent, deeply customize model parameters, and perform targeted training and optimization.
- The most sincere open source commercial use: no need to apply, you can achieve commercial use
At present, most of the Chinese large models in the open source community are not fully commercially available. Generally, users in the open source community usually need to go through a complex commercial authorization application process. In some cases, there are even clear regulations on company size, industry, number of users and other dimensions. No commercial license granted.
Kunlun Wanwei attaches great importance to the openness and commercialization of the Skywork-13B series open source. It has simplified the authorization process and removed restrictions on industry, company size, users, etc., with the purpose of helping more people who are familiar with Chinese large models Interested users and enterprises continue to explore and progress in the industry.
This time, the Skywork-13B series of large models will be fully licensed for commercial use. After users download the model and agree to and abide by the "Skywork Model Community License Agreement", they can use the large model for commercial purposes without applying for authorization again. It is hoped that users can more conveniently explore the technical capabilities of the Skywork-13B series large models and explore commercial applications in different scenarios.
Promote the prosperity of the open source ecosystem, allow more developers to participate in AIGC's technological development, and promote technological improvement through co-creation and sharing.
In the era of AI, the construction of a booming open source ecosystem is an important part of building the integration of AI and applications. Reduce the research and development threshold and usage cost of the model, maximize the sharing of technical capabilities and experience, and allow more companies and developers to participate in this technological change led by AI. Kunlun Wanwei Chairman and CEO Fang Han is the first open source veteran to participate in the construction of the open source ecosystem, and is also one of the earliest promoters of Chinese Linux open source. The spirit of open source and the development of AIGC technology will be perfectly integrated in Kunlun Wanwei's strategy.
All in AGI与AIGC
All in AGI and AIGC are Kunlun Wanwei’s strategies.
On April 17, Kunlun Wanwei released China's first domestic large-scale language model that truly realizes the emergence of intelligence - "Tiangong 3.5" and launched invitation testing.
On May 19, the Beijing Municipal Bureau of Economy and Information Technology announced the first batch of "Beijing General Artificial Intelligence Industry Innovation Partnership Program Member List". Kunlun Wanwei has become the first batch of model partners and investment partners with its cutting-edge exploration and investment layout in the AIGC field.
On August 23, Kunlun Wanwei released Tiangong AI Search, China’s first AI search product.
On September 1, Professor Yan Shuicheng, a top international expert in the field of computer vision and machine learning, officially joined Kunlun Technology. Together with Kunlun Technology founder Zhou Yahui, he served as the co-CEO of Tiangong Intelligence and concurrently served as the director of Kunlun Technology 2050 Global Research Institute. Responsible for research on cutting-edge technologies.
On September 5, the Kunlun Wanwei Tiangong Large Model ranked first in the comprehensive score in the multi-modal large language model evaluation conducted by Tencent Youtu Lab and Xiamen University.
On September 25, Kunlun Wanwei officially took control of Aijie Core and laid out AI chips.
Today, the open source of the Skywork-13B series of large models marks Kunlun Wanwei’s determination to continue investing in the AGI ecosystem.
Introduction to Kunlun Wanwei Group
Kunlun Wanwei was established in 2008 and listed on the Shenzhen Stock Exchange in 2015. From gaming to AII, AGI and AIGC, we have comprehensively built a diversified business ecosystem. With more than ten years of development, we have always been committed to providing leading Internet services to global users. products and services. Today, Kunlun Wanwei is still exploring the infinite possibilities in the field of AI. At present, Kunlun Wanwei has gradually built three major business segments: AGI and AIGC, overseas information distribution and metaverse, and investment. Its business covers more than 100 countries and regions around the world, and the global average monthly active users are nearly 400 million.
With its advanced prediction of technological development trends, Kunlun Technology has begun to deploy in the AIGC field as early as 2020. So far, it has accumulated nearly three years of relevant engineering research and development experience and established industry-leading in-depth processing capabilities for pre-training data. Kunlun Wanwei has also made major breakthroughs in the field of artificial intelligence. It has now formed AI large models, AI search, AI With six major AI business matrices: games, AI music, AI animation, and AI social networking, it is one of the domestic companies with the strongest model technology and engineering capabilities, the most comprehensive layout, and is fully committed to the construction of open source communities.
Alibaba Cloud suffered a serious failure and all products were affected (restored). Tumblr cooled down the Russian operating system Aurora OS 5.0. New UI unveiled Delphi 12 & C++ Builder 12, RAD Studio 12. Many Internet companies urgently recruit Hongmeng programmers. UNIX time is about to enter the 1.7 billion era (already entered). Meituan recruits troops and plans to develop the Hongmeng system App. Amazon develops a Linux-based operating system to get rid of Android's dependence on .NET 8 on Linux. The independent size is reduced by 50%. FFmpeg 6.1 "Heaviside" is released