This article is a series of LLM articles, focusing on
the translation of "Baichuan 2: Open Large-scale Language Models".
Baichuan2: an open large-scale language model
Summary
Large language models (LLMs) have demonstrated remarkable performance on a variety of natural language tasks based on just a few examples of natural language instructions, reducing the need for extensive feature engineering. However, most powerful LLMs are closed source or have limited capabilities for languages other than English. In this technical report, we present Baichuan 2, a series of large-scale multi-language models containing 7 billion and 13 billion parameters, trained from scratch on 2.6 trillion tokens. Baichuan 2 matches or outperforms other open source models of similar scale on public benchmarks such as MMLU, CMMLU, GSM8K and HumanEval. In addition, Baichuan 2 has outstanding performance in vertical fields such as medicine and law. We will publish all pre-training model checkpoints to help the research community better understand the training dynamics of Baichuan-2.
1 Introduction
2 Pre-training
3 Alignment
4 Security
5 evaluation
6 related work
7 Limitations and ethical considerations
Like other large language models, Baichuan 2 also faces ethical challenges. It is prone to bias and toxicity, especially given that much of its training data comes from the internet. Despite our best efforts to mitigate these issues using benchmarks such as Toxigen, the risks cannot be eliminated and toxicity tends to increase with model size. In addition, the knowledge of the Baichuan No. 2 model is static and may be outdated or incorrect, which poses a challenge to fields such as medicine or law that require the latest information. While the model is optimized for Chinese and English for safety reasons, there are limitations in other languages and may not fully capture biases associated with non-Chinese cultures.
There is also the potential for abuse, as the model could be used to generate harmful or misleading content. Although we do our best to balance safety and practicality, some safety measures may appear to be overly cautious and affect the usability of the model for certain tasks. We encourage users to use Baichuan 2 models responsibly and ethically. At the same time, we will continue to optimize these issues and release updated versions in the future.