【CS324】LLM (large model capabilities, data, architecture, distributed training, fine-tuning, etc.)

note

• This article is the study notes of Stanford University’s CS324 course, and also refers to some LLM related materials.

I. Introduction

• Language models were originally studied in the context of information theory and can be used to estimate the entropy of English.
  • Entropy is a measure of probability distribution: $$H(p)=\sum _{x}p(x)\log \frac{1}{p\left(x\right)}.$$
  • Entropy is actually a measure of how many samples $x\sim A\; measure\; of\; the\; expected\; number\; of\; bits\; required\; to\; encode\; (i.e.\; compress)\; p$ into a bit string. For example, "the mouse ate the cheese" might be encoded as "0001110101". The smaller the value of entropy, the stronger the structure of the sequence and the shorter the length of the code. To understand intuitively,$\log \frac{1}{p(x)}$It can be regarded as used to express the probability of occurrence as p ( x ) p(x)elementxx$of\; p$$($$x$$)$The length of the encoding of$x\; .$
  • The upper bound of cross entropy H(p,q) is entropy H§: $$H(p,q)=\sum _{x}p\left(x\right)\log \frac{1}{q(x)}.$$ , so it can be achieved by constructing a distributionpp(Language) model qqfor samples of$p$$q$ to estimate$H(p,q)$
• N-gram models are computationally extremely efficient, but statistically inefficient.
• Neural language models are statistically efficient but computationally inefficient.
• Parameter development of large models: With the rise of deep learning in the 2010s and major hardware advancements (e.g., GPUs), the size of neural language models has increased significantly. The table below shows that in the past 4 years, the size of the model has increased 5000 times.

Model	Organization	Date	Size (# params)
ELMo	AI2	Feb 2018	94,000,000
GPT	OpenAI	Jun 2018	110,000,000
BERT	Google	Oct 2018	340,000,000
XLM	Facebook	Jan 2019	655,000,000
GPT-2	OpenAI	Mar 2019	1,500,000,000
RoBERTa	Facebook	Jul 2019	355,000,000
Megatron-LM	NVIDIA	Sep 2019	8,300,000,000
T5	Google	Oct 2019	11,000,000,000
Turing-NLG	Microsoft	Feb 2020	17,000,000,000
GPT-3	OpenAI	May 2020	175,000,000,000
Megatron-Turing NLG	Microsoft, NVIDIA	Oct 2021	530,000,000,000
Gopher	DeepMind	Dec 2021	280,000,000,000

2. The ability of large models

1. From language model to task model

• Create a new model and use the language model as features (probe method)
• Model fine-tuning

2. Task evaluation

• Language modeling
  • Perplexity: The average "branching factor" per token. The "branching factor" here can be understood as the average number of words or tags that the language model predicts may appear next after each specific word or tag appears. So it's actually a way of measuring the diversity and uncertainty of a model's predictions.
  • Two types of errors:
    • recall error
    • accuracy error
• Question answering: question and answer
• Translation: translation
• Arithmetic: abstract reasoning, such as doing math problems
• News article generation: Given a title, generate an article
• Novel tasks: Given a custom word, generate sentences using that word

Related benchmarks:

• SWORDS : Lexical replacement, the goal is to predict synonyms in the context of a sentence.
• Massive Multitask Language Understanding : Includes 57 multiple-choice questions on math, U.S. history, computer science, and more.
• TruthfulQA : A dataset of questions and answers answered incorrectly by humans due to misunderstanding.

3. Harmfulness of large models

Harmful (Part 1)

• Performance Difference: Automatic Speech Recognition (ASR) systems perform worse on Black speakers than White speakers ( Koenecke et al., 2020 )
• Social prejudice: sexism, etc.

Harmful (Part 2)

• Toxicity: Toxicity, such as rude, disrespectful, or unreasonable behavior, may make someone want to leave a conversation. As in "Trans women are not women."
• False information: disinformation
• Content moderation: Facebook (or Meta) has long combated harmful content and has recently begun leveraging language models to automatically detect such content. For example, RoBERTa has been used for several years.

4. Data of large models

• Common Crawl is a non-profit organization that crawls the web and provides snapshots free to the public. Due to its convenience, it has become the standard data source for many models such as T5, GPT-3 and Gopher.
  • Although Internet data is abundant, it is biased. For example, there are more young, male, and more users. Only 8.8% of Wikipedia writers are female.
• Although OpenAI does not publicly release the WebText dataset, the OpenWebText dataset conceptually replicates the construction method of WebText.
• The GPT-3 data set is mainly derived from Common Crawl, and Common Crawl is similar to a reference data set-WebText. GPT-3 downloaded Common Crawl data (2016-2019) for 41 shards. By training a binary classifier to predict the difference between WebText and Common Crawl, if the classifier believes that the document is closer to WebText, then the document has a greater probability of being retained. When processing the data, GPT-3 adopted a fuzzy deduplication method (detecting 13-gram overlap and removing the window or document if it occurs in less than 10 training documents) and removed the data from the benchmark data set . In addition, GPT-3 also expands the diversity of data sources (including WebText2, Books1, Books2, and Wikipedia). During the training process, Common Crawl is downsampled, it accounts for 82% of the dataset, but only contributes 60% of the data.
• Corpus used for large model pretrain:

• Overview "A Survey of Large Language Models" summarizes the proportions of each part of the pretrain corpus of mainstream large models:
  
• Typical pretrain data preprocessing pipeline: abnormal word filtering, deduplication, private information processing, tokenization, etc.:

5. Law issues

• legal question:
  • Data: such as data privacy issues
  • Model application: used for downstream tasks and cannot do bad things (such as fraud, fake news, fraud, etc.)
  • Copyright issue
• Three major stages of information technology:
  • 1. The first stage: text data mining (search engine), based on simple pattern matching.
  • 2. Phase 2: Classification (e.g., classification stop signs or sentiment analysis), recommendation systems.
  • 3. The third stage: learning a generative model that imitates expressions.

6. Model Architecture

1. tokenization

Good participle:

• There cannot be too many tokens after word segmentation, otherwise it will be difficult to model; there cannot be too few tokens, otherwise parameters cannot be shared between words.
• Each token is a linguistically or statistically meaningful unit

Word segmentation method:

• Space-based word segmentation: text.split(' '), the simplest method, is feasible if it is English, but there are no spaces between Chinese words, and even in English, there are hyphenated words (such as father-in-law) and abbreviations (such as don't ), they need to be split correctly. For example, Penn Treebank splits don't into do and n't, which is a linguistically information-based choice but is less obvious. Therefore, dividing words simply by spaces creates a lot of problems.
• Other methods:
  • Byte-Pair Encoding (BPE) tokenization
  • WordPiece tokenization
  • Unigram tokenization
  • You can use the library: SentencePiece
  • The tokenizer used by common models is as shown below:

[Chestnut]
Suppose we have an English corpus containing the following two sentences:

1. “I like playing soccer.”
2. “I like playing basketball.”

Now, we want to use these sentences to build a subword vocabulary for word segmentation.

BPE（Byte Pair Encoding）：

When using BPE for word segmentation, we first divide each sentence into individual characters, resulting in the following character sequence:

1. “I like playing soccer.”

   • Characters: [‘I’, ’ ', ‘l’, ‘i’, ‘k’, ‘e’, ’ ', ‘p’, ‘l’, ‘a’, ‘y’, ‘i’, ‘n’, ‘g’, ’ ', ‘s’, ‘o’, ‘c’, ‘c’, ‘e’, ‘r’, ‘.’]

2. “I like playing basketball.”

   • Characters: [‘I’, ’ ', ‘l’, ‘i’, ‘k’, ‘e’, ’ ', ‘p’, ‘l’, ‘a’, ‘y’, ‘i’, ‘n’, ‘g’, ’ ', ‘b’, ‘a’, ‘s’, ‘k’, ‘e’, ‘t’, ‘b’, ‘a’, ‘l’, ‘l’, ‘.’]

We then calculate the most frequently occurring character combinations in the character sequence and merge them into a new subword. In this example, "pl" is the most frequently occurring character combination, and we merge it into a new subword, resulting in the following sequence:

1. “I like playing soccer.”

   • Tokens: [‘I’, ’ ', ‘like’, ’ ', ‘playing’, ’ ', ‘s’, ‘o’, ‘c’, ‘e’, ‘r’, ‘.’]

2. “I like playing basketball.”

   • Tokens: [‘I’, ’ ', ‘like’, ’ ', ‘playing’, ’ ', ‘b’, ‘a’, ‘s’, ‘k’, ‘e’, ‘t’, ‘b’, ‘a’, ‘l’, ‘l’, ‘.’]

Through BPE, we treat "pl" as a subword and successfully segment "playing" into "play" and "ing".

WordPiece：

For WordPiece word segmentation, we also first segment each sentence into individual characters. We then select the merging operation based on the frequency of the character sequence and the language model score.

For the above example, we count the frequency of character sequences and select the most frequently occurring merge operation. Let's say we choose to merge "pl":

1. “I like playing soccer.”

   • Tokens: [‘I’, ’ ', ‘like’, ’ ', ‘p’, ‘lay’, ‘ing’, ’ ', ‘soccer’, ‘.’]

2. “I like playing basketball.”

   • Tokens: [‘I’, ’ ', ‘like’, ’ ', ‘p’, ‘lay’, ‘ing’, ’ ', ‘basketball’, ‘.’]

Through WordPiece, we successfully split "playing" into "p", "lay" and "ing".

Summarize:

• BPE and WordPiece are subword segmentation methods based on statistics, which can merge frequently occurring character combinations into subwords.
• BPE combines character combinations through simple frequency statistics, while WordPiece also takes into account language model scores.

2. Model architecture

The contextual vector representation (Contextual Embedding) is as follows. The contextual vector representation of a mark depends on its context (surrounding words); for example, considering the vector representation of mouse, you need to pay attention to other words of a certain window size around it: [ the , mouse ,
$$[the,mouse,ate,the,cheese]\stackrel{\varphi }{\Rightarrow }\left[\left(\begin{array}{c}1\\ 0.1\end{array}\right),\left(\begin{array}{l}0\\ 1\end{array}\right),\left(\begin{array}{l}1\\ 1\end{array}\right),\left(\begin{array}{c}1\\ -0.1\end{array}\right),\left(\begin{array}{c}0\\ -1\end{array}\right)\right].$$

• Symbolic representation: $\varphi :V^L\to R^{d\times L}$ is defined as the embedding function (similar to the feature map of the sequence, mapped to the corresponding vector representation).
• For the token sequence $x1:L=[x_1,\dots ,x_L]$ ,$\varphi$ generates context vector representation$\varphi (x_{1:L})$。

（1）only encoder

• Models: bert, RoBERTa, etc., generate context vectors instead of directly generating text, often used for classification tasks
• Advantages: for each $xi$ , the context vector representation can depend on the left context bidirectionally$(x_{1:i-1})$ and the right context$(x_{i+1:L})$。

（2）only decoder

• Model: gpt is an autoregressive model, given a hint $x_{1:i}$, they can generate context vector representations and identify the next token $x_{i+1}$(And recursively, the entire completes $x_{i+1:L}$) generates a probability distribution. $x_{1:i}\Rightarrow \varphi (x_{1:i}),p(x_{i+1}\mid x_{1:i})$。
• Disadvantages: for each $xi$ , the context vector representation can only depend unidirectionally on the left context ($x_{1:i-1}$)。

（3）encoder-decoder

• Models: original transformer, BART, T5, etc.
• Advantages: They can be usedBidirectional context vector representationto process the input $x_{1:L}$, and can generate the output $y_{1:L}$. It can be formulated as:
  $$x1:L\Rightarrow \varphi (x1:L),p(y1:L|\varphi (x1:L)).$$
  Taking the table-to-text generation task as an example, the input and output can be expressed as:
  $$[name:,plants,|,type:,flowers,Shop]\Rightarrow [Flowers,Yes,one,one,store].$$
• Disadvantages: Requires more specific training goals

3. Infrastructure

(1) Return to transformer architecture

Single-head attention matrix form:
def $Attention(x_{1:L}:R^{d\times L},y:R^d)\to R^d$：

• By combining it with each $x_i$Compare to handle $y$。
• 返回$W_{value}{x}_{1:L}\mathrm{softmax}\left(x_{1:L}^{\top }{W}_{key}^{\top }{W}_{query}y/\sqrt{d}\right)$

Multi-head attention mechanism:
def $MultiHeadedAttention(x_{1:L}:R^{d\times L},y:R^d)\to R^d$:

• Process y by comparing it to each xi with nheads aspects.
• 返回$W_{output}[\underset{n_{heads}times}{\underbrace{\left[\mathrm{Attention}\left(x_{1:L},y\right),\dots ,\mathrm{Attention}\left(x_{1:L},y\right)\right]}}$

For the self-attention layer , we will use $x_i$Replace $y$ is generated as a query parameter, which is essentially its own$x_i$Attention to other contextual content of the sentence$Attention$ operation:

def $SelfAttention(x_{1:L}:R_{d\times L})\to R_{d\times L})$：

• Compare each element xi with other elements.
• 返回$[Attention(x_{1:L},x_1),\dots ,Attention(x_{1:L},x_L)]$。

Self-attention allows all markers to "communicate" with each other, while feed-forward layers provide further connections:

def $FeedForward(x_{1:L}:R^{d\times L})\to R^{d\times L}$：

• Each tag is processed independently.
• For $i=1,\dots ,L$：
  • 计算$y_i=W_{2}max(W_1{x}_i+b_1,0)+b_2$。
• 返回$[y_1,\dots ,y_L]$。
  

(2) Optimize network training

(1) Residual link: add residual (jump) connection to prevent $When\; f$ gradient disappears, you can also use$x_{1:L}$Calculate
(2) layer normalization: def $LayerNorm(x_{1:L}:R^{d\times L})\to R^{d\times L}$ , specifically define the function to accept a sequence model$f$ and make it "robust":

def $AddNorm(f:(R{d}^{\times L}\to R^{d\times L}),x_{1:L}:R_{d\times L})\to R^{d\times L}$：

• Apply f to $x_{1:L}$。
• Definition $LayerNorm(x_{1:L}+f(x_{1:L}))$。

Finally define the Transformer block as follows:

def $TransformerBlock(x_{1:L}:R^{d\times L})\to R^{d\times L}$：

• Process each element xi x_{i} in the context$x_i$。
• Definition $AddNorm(FeedForward,AddNorm(SelfAttention,x_{1:L}))$。

（3）位置嵌入
def $EmbedTokenWithPosition(x_{1:L}:R^{d\times L})$：

• Add location information.
• Define location embedding:
  • Even dimensions: $P_{i,2}=sin(i/10000^{2j/dmodel})$
  • Odd dimensions: $P_{i,2j+}=cos(i/10000^{2j/dmodel})$
• Return $[x_1+P_1,\dots ,x_L+P_L]$。
• Marking above: $i$ represents the position of the marker in the sentence,$j$ represents the vector of this marker indicating the dimensional position.

Note: Large LLM models now also use ROPE rotation position encoding, ALiBi position encoding, etc. to expand the window size.

4. Large model architecture

(1) GPT3 architecture:

• Stack transformer blocks 96 times
• Dimension of hidden state: dmodel=12288
• Dimension of the intermediate feedforward layer: dff=4dmodel
• Note the number of heads: nheads=96
• Context length: L=2048

(2) llama model architecture

• transform architecture (Vaswani et al., 2017),
• Use the RMSNorm (Root Mean Square Layer Normalization) method to perform a reduction (norm) operation on the input of each layer of the transformer, instead of norming the output before the transformer: apply pre-normalization using RMSNorm (Zhang and Sennrich, 2019) ,
• SwiGLU activation function: use the SwiGLU activation function (Shazeer, 2020),
• Rotary positional encoding: rotary positional embeddings (RoPE, Su et al. 2022).
• Context length and grouped-query attention (GQA): The primary architectural differences from Llama 1 include increased context length and grouped-query attention (GQA).

7. Model training

1. decoder-only model

Conditional distribution p ( xi ∣ x 1 : i − 1 ) p(x_i \mid x_{1:i-1}) of the autoregressive language model$p(x_i\mid x_{1:i-1})$：

• General $x_{1:i-1}$Maps to context embedding $\varphi (x_{1:i-1})$。
• Apply embedding matrix $E\in {\mathbb{R}}^{V\times d}$ to obtain the score of each token$E\varphi (x_{1:i-1}{)}_{i-1}$。
• Exponentialize and normalize it to get the prediction $x_i$Distribution.
• In conclusion:
  $$p(x_{i+1}\mid x_{1:i})=softmax(E\varphi (x_{1:i}{)}_i).$$

Maximum likelihood: Let $\theta$ are all parameters of the large language model. Let$D$ is the training data consisting of a set of sequences. Then we can follow the maximum likelihood principle and define the following negative log-likelihood objective function:
$$O\left(i\right)=\sum _{x\in D}-\log p_{}(x)=\sum _{x\in D}\sum _{i=1}^L-\log p_{}(x_i\mid x_{1:i-1}).$$

2. Optimization algorithm

Known: The objective function of the autoregressive language model: $$O\left(i\right)=\sum _{x\in D}-\log p_{}(x).$$

（1）Adam (adaptive moment estimation)

The Adam algorithm has the following two innovations:

1. Introduce momentum (continue moving in the same direction).
2. Parameter $i_0$Each dimension of has an adaptive (different) step size (inspired by second-order methods).

Its steps are as follows:

• Initialization parameter $i_0$

• Initialize momentum $m_0,v_0\leftarrow 0$

• Repeat the following steps:

  • Sample mini-batch $B_t\subset D$

  • Follow the steps below to update parameters:

    • Calculate gradient

    $$g_t\leftarrow \frac{1}{|B_t|}\sum _{x\in B_t}{\nabla }_{}(-\log p_{}(x)).$$

    • Update first-order and second-order momentum

    $$\begin{gathered} m_t\leftarrow b_1{m}_{t-1}+(1-b_1)g_t \\ {v}_t\leftarrow b_2{v}_{t-1}+(1-b_2)g_t^2 \end{gathered}$$

    • Correct the deviation

    $$\begin{gathered} {\hat{m}}_t\leftarrow m_t/(1-b_1^t) \\ {\hat{v}}_t\leftarrow v_t/(1-b_2^t) \end{gathered}$$

    • Update parameters

    $$i_t\leftarrow i_{t-1}-the{\hat{m}}_t/(\sqrt{{\hat{v}}_t}+ϵ).$$

Storage usage analysis:

Adam will store the model parameters from 2 times ( $i_t,g_t$) increased to 4 times ( $i_t,g_t,m_t,v_t$）。

（2） AdaFactor

AdaFactor is an optimization algorithm to reduce storage usage. Features:

• It does not store $m_t,v_t$Such $O(m\times n)$ matrix, instead it stores the sum of rows and columns$O(m+n)$ and reconstruct the matrix
• Remove momentum
• It is used to train T5
• AdaFactor can make training difficult (see Twitter thread and blog post )

(3) Model parameter initialization

• Given matrix $W\in {\mathbb{R}}^{m\times n}$ , the standard initialization (i.e., xavier initialization) is$W_{ij}\sim N(0,1/n)$。
• GPT-2 and GPT-3 pass an additional $1/\sqrt{N}$Scaling weights, where $N$ is the number of residual layers.
• T5 increases the attention matrix by $1/\sqrt{d}($ code).

Taking GPT-3 as an example, the parameters used are as follows:

• Adams: $b_1=0.9,b_2=0.95,ϵ=10^{-8}$
• Small batch size: 3.2 million tokens (about 1,500 sequences)
• Use gradient clipping ( $g_t\leftarrow g_t/\min (1,\Vert g{\Vert }_2)$）
• Linear learning rate warm-up (first 375 million tokens)
• Cosine learning rate decays to 10%
• Gradually increase batch size
• Weight decay is set to 0.1

8. Distributed training

[Background] Suppose that a certain layer in the neural network is doing matrix multiplication, and the input $The\; shape\; of\; x$ is$4\times 5$ , model parameter$The\; shape\; of\; w$ is$5\times 8$ , then, the matrix multiplication output shape is$4\times 8$ . as follows:

• Data parallelism: split data $x$ to different devices. During the backpropagation process, AllReduce needs to be performed on the gradients on each device to ensure that the model on each device is always consistent. Suitable for situations where the data is large and the model is small, such as resnet50

  • allReduce: First reduce the gradients of the models on all devices, and then broadcast the results to all devices.
    

• Model parallelism: AllReduce gradients between multiple devices are omitted; however, since each device requires complete data input, the data will be broadcast among multiple devices, resulting in communication costs. For example, the final result in the picture above is out $(4\times 8)$ , if it is used as the input of the next layer network, then it needs to be broadcast to two devices. Such as bert
  

• Pipeline parallelism: The model network is too large. In addition to using model parallelism, pipeline parallelism can also be used (the following network has 4 layers, T1-T4)
  

• Hybrid parallelism: Combine the above multiple strategies for training together, such as GPT3:

  • First, it is divided into 64 stages for pipeline parallelization. Each stage runs on 6 DGX-A100 hosts.
  • Data parallel training is performed between 6 hosts;
  • Each host has 8 GPU graphics cards, and the 8 GPU graphics cards on the same machine perform model parallel training.

reference:

• stanford course：https://stanford-cs324.github.io/winter2022/lectures/parallelism/
• Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism. M. Shoeybi, M. Patwary, Raul Puri, P. LeGresley, J. Casper, Bryan Catanzaro. 2019.
• GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism . Yanping Huang, Yonglong Cheng, Dehao Chen, HyoukJoong Lee, Jiquan Ngiam, Quoc V. Le, Z. Chen. NeurIPS 2018.
• Efficient large-scale language model training on GPU clusters using Megatron-LM . D. Narayanan, M. Shoeybi, J. Casper, P. LeGresley, M. Patwary, V. Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, J. Bernauer, Bryan Catanzaro, Amar Phanishayee, M. Zaharia. SC 2021.
• TeraPipe: Token-Level Pipeline Parallelism for Training Large-Scale Language Models . Zhuohan Li, Siyuan Zhuang, Shiyuan Guo, Danyang Zhuo, Hao Zhang, D. Song, I. Stoica. ICML 2021.

9. New model architecture

• The hybrid expert model is a bit like the previous MMOE. The retrieval-based model can be combined with the knowledge base question and answer. The specific model will be studied and updated later. Thank you for your reminder.

1. Mixed expert model

References:

• Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. Noam M. Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, J. Dean. ICLR 2017. Trains 137 billion parameter model; mixture of experts (1000 experts) applied convolutionally between LSTM layers.
• GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding. Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, M. Krikun, Noam M. Shazeer, Z. Chen. ICLR 2020. Trains Transformer for neural machine translation (100 languages) with 600 billion parameters. Use top-2 experts.
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. W. Fedus, Barret Zoph, Noam M. Shazeer. 2021. Trains language model, 4x speedup over T5-XXL (13 billion parameters). Use top-1 expert.
• GLaM: Efficient Scaling of Language Models with Mixture-of-Experts . Nan Du, Yanping Huang, Andrew M. Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, M. Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, Barret Zoph , Liam Fedus, Maarten Bosma, Zongwei Zhou, Tao Wang, Yu Emma Wang, Kellie Webster, Marie Pellat, Kevin Robinson, K. Meier-Hellstern, Toju Duke, Lucas Dixon, Kun Zhang, Quoc V. Le, Yonghui Wu, Zhifeng Chen, Claire Cui . 2021. Trains 1.2 trillion parameter model, 64 experts. Use top-2 experts. Also creates new dataset.
• BASE Layers: Simplifying Training of Large, Sparse Models. M. Lewis, Shruti Bhosale, Tim Dettmers, Naman Goyal, Luke Zettlemoyer. ICML 2021. Solve optimization problem for token-to-expert allocation to balance allocation. Trains 110 billion parameter model.
• Efficient Large Scale Language Modeling with Mixtures of Experts . Mikel Artetxe, Shruti Bhosale, Naman Goyal, Todor Mihaylov, Myle Ott, Sam Shleifer, Xi Victoria Lin, Jingfei Du, Srinivasan Iyer, Ramakanth Pasunuru, Giridhar Anantharaman, Xian Li, Shuohui Chen, H. Akin, Mandeep Baines, Louis Martin, Xing Zhou, Punit Singh Koura, Brian O'Horo, Jeff Wang, Luke Zettlemoyer, Mona Diab, Zornitsa Kozareva, Ves Stoyanov . 2021. Trains 1.1 trillion parameter models. Use top-2 experts (512 experts).
• Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Experts. Max Ryabinin, Anton I. Gusev. NeurIPS 2020.
• Distributed Deep Learning in Open Collaborations. Michael Diskin, Alexey Bukhtiyarov, Max Ryabinin, Lucile Saulnier, Quentin Lhoest, A. Sinitsin, Dmitry Popov, Dmitry Pyrkin, M. Kashirin, Alexander Borzunov, Albert Villanova del Moral, Denis Mazur, Ilia Kobelev, Yacine Jernite, Thomas Wolf, Gennady Pekhimenko. 2021.
• Dense-to-Sparse Gate for Mixture-of-Experts . Xiaonan Nie, Shijie Cao, Xupeng Miao, Lingxiao Ma, Jilong Xue, Youshan Miao, Zichao Yang, Zhi Yang, Bin Cui. 2021.

2. Retrieval-based model

References:

• REALM: Retrieval-Augmented Language Model Pre-Training. Kelvin Guu, Kenton Lee, Z. Tung, Panupong Pasupat, Ming-Wei Chang. 2020. Introduces REALM.
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks . Patrick Lewis, Ethan Perez, Alexander Pictus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, M. Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, Douwe Kiela . NeurIPS 2020. Introduces RAG .
• Improving language models by retrieving from trillions of tokens. Sebastian Borgeaud, A. Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, G. V. D. Driessche, J. Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, T. Hennigan, Saffron Huang, Lorenzo Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, K. Simonyan, Jack W. Rae, Erich Elsen, L. Sifre. 2021. Introduces RETRO.

10. Adaptation of large models

1. Common adaptation configuration

1. Pre-trained language model (Pre-trained LM) :
   Use parameters $\theta LM$ represents

2. Downstream Task Dataset :
   Downstream task distribution $P_{task}$sample data. It can be a specific instance of tasks such as text classification and sentiment analysis. Each sample consists of input x and target output y, such as: $\left(x^{(1)},y^{(1)}\right),\dots ,\left(x^{(n)},y^{(n)}\right)$。

3. Adaptation Parameters :
   In order to make the pre-trained LM suitable for a specific downstream task, it is now necessary to find a set of parameters $c$，This set of parameters can come from a subset of existing parameters or introduce new parameters, $C$. These parameters will be used to tune the model so that it performs better on a specific task.

4. Task Loss Function :
   Define a loss functionℓ ${\ell }_{\text{task }}$To measure the performance of the model on downstream tasks. For example, cross-entropy loss is a common choice to measure the difference between the probability distribution predicted by the model and the true distribution.

5. Optimization Problem :
   The goal is to find a set of adaptation parameters $c_{\text{adapt }}$, minimizing the task loss on the entire downstream data set. Mathematically, this can be expressed by the following optimization problem:

$$c_{\text{adapt }}={\mathrm{argmin}}_{\gamma \in \Gamma }\frac{1}{n}\sum _{i=1}^n{\ell }_{\text{task }}\left(c,i_{\mathrm{LM}},x_i,y_i\right).Finally$$
obtain a set of adaptation parameters$c_{\text{adapt }}$, used for the parameterized adapted model $p_{adapt}$。

2.Adaptation method

（1）Probing

（2）Fine-tuning

Fine-tuning for human alignment

InstructGPT three steps to fine-tune the GPT-3 model:

1. Collect demonstration behavior of human writing : This step involves collecting examples that match human expectations and performing supervised fine-tuning on these examples.

2. Instruction-based sampling with human preferences : For each instruction, k outputs are sampled from the LM of step 1. Human feedback is then collected on which sampled output takes priority. These data are cheaper compared to step 1.

3. Fine-tune the LM with a reinforcement learning objective : Fine-tune the LM in step 1 with a reinforcement learning objective to maximize the human-preferred reward.

After such fine-tuning, the 1.3B InstructGPT model was prioritized over the 175B GPT-3 85% of the time, and 71% when using few-shot hints. In terms of closed-domain Q&A/summaries, InstructGPT generates fictitious information 21% of the time, which is an improvement compared to GPT-3’s 41%. When prompted to respect, InstructGPT produces 25% less toxic output than GPT-3.

（3）Lightweight Fine-tuning

• You can use peftthe library (Parameter-Efficient Fine-Tuning) for fine-tuning, which supports the following tuning:
  • Adapter Tuning (fix the parameters of the original pre-trained model and only fine-tune the new adapter)
  • Prefix Tuning (Construct a task-related virtual tokens as a prefix before inputting the token. Only the parameters of the Prefix part are updated during training, while other parameters of the Transformer are fixed regardless of the parameters. It is similar to constructing a prompt, except that the prompt is artificially constructed and cannot be used in the model. Update parameters during training, and Prefix can learn the <implicit> prompt)
  • Prompt Tuning (a simplified version of Prefix Tuning, which only adds prompt tokens to the input layer and does not need to add MLP)
  • P-tuning (convert prompts into learnable embedding layers, v2 adds prompts tokens as input)
  • LoRA (Low-Rank Adaption, in order to solve the problem that the adapter increases the model depth and increases the model inference time. The prompts in the above tuning are difficult to train and reduce the available sequence length of the model)
    • This method can directly add the two trained AB matrices to the parameters of the original pre-training model during inference, and the addition result replaces the parameters of the original pre-training model.
    • Equivalent to using LoRA to simulate the full-tunetune process

Lightweight Fine-tuning saves resources while maintaining the same effect as full parameter fine-tuning:

1. Prompt Tuning : Optimize the performance of the model by fine-tuning the model's input prompts. Prompt tuning can be viewed as a more flexible method of fine-tuning, allowing users to guide the model's output by adjusting input prompts rather than directly modifying model parameters.
2. Prefix Tuning : Similar to prompt tuning, prefix tuning is also focused on the input part. It tailors the model's behavior to specific tasks by adding specific prefixes to adjust the model's behavior.
3. Adapter Tuning : Adapter Tuning is a method of fine-tuning a model by inserting trainable "adapter" modules between the hidden layers of the model. These adapter modules allow models to be fine-tuned without changing the original pre-trained parameters, thereby reducing storage and computation requirements.

11. Environmental impact

Patterson et al., 2021
简单形式：
$$\text{emissions}=R_{\text{power}\to \text{emit}}(\text{energy-train}+\text{queries}\cdot \text{energy-inference})$$

• NVIDIA: 80% of ML workloads are inference, not training

For training:

$$\text{emissions}=\text{hours-to-train}\cdot \text{num-processors}\cdot \text{power-per-processor}\cdot \text{PUE}\cdot R_{\text{power}\to \text{emit}}$$

Estimates from different models:

• T5：86 MWh，47t CO2eq
• GShard (MOE model for machine translation): 24 MWh, 4.3t CO2eq
• Switch Transformer：179 MWh，59t CO2eq
• GPT3：1287 MWh，552t CO2eq

Reference

[1] 斯坦福大学CS324课程：https://stanford-cs324.github.io/winter2022/lectures/introduction/#a-brief-history
[2] CS224N lecture notes on language models
[3] Language Models are Few-Shot Learners. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, J. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, T. Henighan, R. Child, A. Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, Dario Amodei. NeurIPS 2020.
[4] Challenges in Detoxifying Language Models. Johannes Welbl, Amelia Glaese, Jonathan Uesato, Sumanth Dathathri, John F. J. Mellor, Lisa Anne Hendricks, Kirsty Anderson, P. Kohli, Ben Coppin, Po-Sen Huang. EMNLP 2021.
[5] Scaling Language Models: Methods, Analysis&Insights from Training Gopher
[5] CommonCrawl
[6] OpenWebText Similar to WebText, used to train GPT-2.
[7] An Empirical Exploration in Quality Filtering of Text Data. Leo Gao. 2021.
[8] Deduplicating Training Data Makes Language Models Better. Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, D. Eck, Chris Callison-Burch, Nicholas Carlini. 2021.
[9] Foundation models report (legality section)
[10] A Survey of Large Language Models：http://arxiv.org/abs/2303.18223
[11] Attention is All you Need. Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, Illia Polosukhin. NIPS 2017.
[12] CS224N slides on Transformers
[13] Rethinking Attention with Performers. K. Choromanski, Valerii Likhosherstov, David Dohan, Xingyou Song, Andreea Gane, Tamás Sarlós, Peter Hawkins, Jared Davis, Afroz Mohiuddin, Lukasz Kaiser, David Belanger, Lucy J. Colwell, Adrian Weller. ICLR 2020. Introduces Performers.
[14] Efficient Transformers: A Survey. Yi Tay, M. Dehghani, Dara Bahri, Donald Metzler. 2020.
[15] BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova. NAACL 2019. Introduces BERT from Google.
[16] RoBERTa: A Robustly Optimized BERT Pretraining Approach. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, M. Lewis, Luke Zettlemoyer, Veselin Stoyanov. 2019. Introduces RoBERTa from Facebook.
[17] BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension. M. Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, Luke Zettlemoyer. ACL 2019. Introduces BART from Facebook.
[18] Language Models are Few-Shot Learners. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, J. Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, T. Henighan, R. Child, A. Ramesh, Daniel M. Ziegler, Jeff Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, Dario Amodei . NeurIPS 2020. Introduces GPT-3 from OpenAI.
[19] Fixing Weight Decay Regularization in Adam . I. Loshchilov, F. Hutter. 2017. Introducing AdamW
[20] Mixed Precision Training
[21] Easy to understand explanation Model: Tokenizer
[22] huggingface official document explanation: Summary of the tokenizers
[23] Microsoft guidance: https://github.com/guidance-ai/guidance#token-healing-notebook
[24] Understanding Byte-Pair Encoding (BPE) Word Piece And Unigram
[25] [Teacher Zhang Yue of West Lake University | Natural Language Processing Online Course Chapter 16 - Section 4] BPE (Byte-Pair Encoding) Encoding
[26] [LLM Series of Tokenizer] How to scientifically train an LLM word segmenter
[27] Large vocabulary language model in the continuation task A problem and countermeasures. Su Jianlin (For example, after LLM uses a very large vocabulary, "Baiyun", "Baiyun Mountain", and "Baiyun Airport" are all independent tokens. After the user enters "Baiyun in Guangzhou", the following It is almost impossible to continue writing "Baiyun Airport in Guangzhou" and "Baiyun Mountain in Guangzhou")
[28] A brief exploration of random word segmentation: from Viterbi Decoding to Viterbi Sampling. Su Jianlin
[29] Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates: https://arxiv.org/abs/1804.10959
[30] Common distributed parallel strategies.oneflow

Attached: time schedule

Mission information	deadline	Precautions
task1: Introduction	9.11 Monday	Finish
task2: Capabilities of large models	Tuesday 9.12	Finish
task3: The harmfulness of large models	Wednesday 9.13	Finish
task4: Data of large model	Thursday 9.14	Finish
task5: Legal chapter of large model	Friday 9.15	Finish
task6: Model architecture	9.16 Saturday	Finish
task7: Model training	9.17 Sunday	Finish
task8: Distributed training	Monday 9.18
task9: New model architecture	Tuesday 9.19
task10: Adaptation of large model	Wednesday 9.20
task11: Environmental impact of large models	Thursday 9.21	Finish