Hello everyone, I am Wei Xue AI. Today I will introduce to you the practical application of large models, the 12-GPT4 framework introduction and detailed training process, as well as parallelism strategies, expert trade-off mechanisms, reasoning trade-offs, etc. On March 14, 2023, OpenAI released GPT-4. However, the framework of GPT-4 was not disclosed. The reason why OpenAI did not disclose the architecture of GPT-4 was not because of potential threats to humans, but because of the Models can be copied. Now, GPT4-turbo has been released, which can be called the most powerful model in history. In addition to GPT, other companies around the world are not to be outdone. For example, companies such as Google, Meta, Anthropic, Inflection, Character, Tencent, Alibaba, Baidu and other companies may have GPT in the future. -4 equally or even more powerful models. Sure, OpenAI has amazing engineering capabilities and the stuff they build is incredible, but the solutions they employ are not magical. This is a pragmatic scenario with many complex trade-offs. OpenAI's biggest advantage is that they have the most real-world use cases, leading engineering talent, and can continue to lead other companies with future models.
GPT-4 status quo
We have gathered a lot of information about GPT-4 from multiple sources, and today we want to share some of it. This includes model architecture, training infrastructure, inference infrastructure, number of parameters, training dataset composition, number of markers, number of layers, parallelization strategies, multi-modal visual adaptability, thought process behind different engineering trade-offs, unique techniques implemented , and how they alleviate some of the biggest bottlenecks associated with inference on large models.
The most interesting aspect of GPT-4 is understanding why they made certain architectural decisions. Additionally, we will outline the cost of GPT-4 training and inference on the A100 and describe the scale compared to next-generation model architectures using the H100.
First, let's look at the problem statement. From GPT-3 to GPT-4, OpenAI hopes to expand the scale 100 times, but the key to the problem is cost. Dense Transformer models cannot be scaled further. DenseTransf