On September 20, Shanghai Artificial Intelligence Laboratory (Shanghai AI Laboratory) and SenseTime, together with the Chinese University of Hong Kong and Fudan University, officially launched the 20 billion parameter version InternLM-20B of the Scholar·Puyu Large Model (InternLM), and launched it on Alibaba Cloud ModelScope is released as open source for the first time. At the same time, Shusheng Puyu’s entire tool chain for large model R&D and application has been upgraded across the board, and will continue to be fully open together with InternLM-20B, providing free commercial authorization to enterprises and developers.
According to reports, the InternLM-20B large model with middle-weight parameters has advanced performance and is easy to apply. With less than one-third of the parameters, it has reached the capability level of Llama2-70B, which is currently regarded as the benchmark for open source models.
Code library link: https://github.com/InternLM/InternLM
Magic scope community link: https://modelscope.cn/organization/Shanghai_AI_Laboratory
Compared with the 7B and 13B models that have been open sourced by the domestic community before, the 20B model has more powerful comprehensive capabilities, especially in complex reasoning and reflection capabilities, so it can bring more powerful performance support to practical applications. ; At the same time, the 20B-level model can be inferred on a single card. After low-bit quantization, it can be run on a single consumer-grade GPU, making it more convenient in practical applications.
InternLM-20B is a large medium-weight language model trained from scratch based on 2.3T token pre-training corpus. Compared with InternLM-7B, the training corpus has undergone a higher level of multi-level cleaning, supplemented with high knowledge density and training data to strengthen understanding and reasoning capabilities. Therefore, InternLM-20B has significantly improved in aspects such as understanding ability, reasoning ability, mathematical ability, and programming ability that test the technical level of language models.
Compared with previous open source models, the capability advantages of InternLM-20B are mainly reflected in:
- Excellent overall performance. InternLM-20B has excellent comprehensive performance, not only leading the open source models of similar magnitude (including Llama-33B, Llama2-13B and the domestic mainstream 7B and 13B open source models), but also with less than one-third of the parameters, it has achieved the best performance in the evaluation. The result reached the level of Llama2-70B.
- Powerful tool calling capabilities. InternLM-20B expands the capability boundaries of the model and achieves effective connection between large models and real scenes. InternLM-20B supports dozens of types of plug-ins and tens of thousands of API functions. It achieved the best results in the ToolBench evaluation set. In the competition with ChatGPT, the winning rate reached 63.5%. InternLM-20B also has code interpretation and reflection correction capabilities, providing a good technical foundation for the construction of agents.
- Longer context. Through multi-stage training expansion, InternLM-20B supports 16K context length, thereby more effectively supporting long text understanding, long text generation and ultra-long conversations. Safer value alignment. Compared with previous versions, InternLM-20B is more secure and reliable in value alignment. During the development and training process, the research team greatly improved its security through two-stage value alignment based on SFT (supervised fine-tuning) and RLHF (reinforcement learning based on human feedback), as well as adversarial training by an expert red team. When users ask questions with biases, the model can give positive guidance.
- Fully upgraded open source tools and data systems. The Shusheng Puyu open source tool chain has been upgraded across the board, forming a more complete tool system, including the pre-training framework InternLM-Train, the low-cost fine-tuning framework XTuner, the deployment inference framework LMDeploy, the evaluation framework OpenCompass, and the agent framework for scenario applications. Lagent. The Shusheng·Puyu tool chain will form a powerful open source tool and data system with the open source data platform OpenDataLab, jointly providing full-chain R&D and application support for academia and industry.
Based on the OpenCompass large model evaluation platform, researchers conducted a comprehensive test and comparison of InternLM-20B and open source models of similar magnitude on 50 mainstream evaluation sets covering five dimensions: language, knowledge, understanding, reasoning and subject ability. The evaluation results show that InternLM-20B is ahead of the open source 13B model in all dimensions. The average score not only significantly surpasses Llama-33B, but is even better than the benchmark Llama2-70B, which is called an open source model.
In ToolBench, a large model tool calling evaluation set jointly released by Tsinghua University and other institutions, InternLM-20B achieved a winning rate of 63.5% compared with ChatGPT, achieving the best results on the list and showing strong tool calling capabilities. .
