Lion:Adversarial Distillation of Closed-Source Large Language Model

Introduction

The author shows that ChatGPT and GPT4 have achieved good results in various industries, but their models and data are closed sources. The current mainstream solution is to distill knowledge to the student model through a teacher model.

But in our actual teaching, when teachers teach knowledge to students, students will also give feedback to the teacher which questions they can't, that is, hard samples.

The authors propose an adversarial framework that includes the following steps:

1. Model stage: align the student's reply to the teacher's reply.
2. Discriminative phase: The discriminative phase identifies difficult samples.
3. Generation phase: Generate new instructions about model generation difficulties.

Finally, through this framework, the author generated 70k data based on 175 Alpaca seed data, fine-tuned on LLaMA 7B, and reached the level of 95% ChatGPT, and the model was named Lion.

Methodology

Define the teacher model as: $T(x,i^T)$，student model 为 $S(x;i^S)$。

In the previous method, the common practice is to fine-tune the student model with the data generated by the teacher model. After the training is completed, the students will not reach the level of the teacher model 100%. There is a gap between them, and the author believes that hard sample dominates This gap, so the optimization direction is to focus on these hard samples to achieve high efficiency.

Therefore, it is necessary to continuously generate hard samples for the student model for student model training, and finally the student model will convert the hard samples into simple samples.

As shown in the figure above, first initialize two Dataset Pools, first use the Teacher model to generate data in a method similar to Self-Instruct, the template is as follows:

Then put the generated data into the Train Pool and Cache Pool respectively, and then fine-tune the Student Model from the data in the Train Pool. After the fine-tuning is completed, the Teacher model and the Student model respectively use the data in the Cache Pool for reasoning. After getting their reasoning results, they compare the quality of the results in a way similar to Vicuna, that is, find the hard sample. The template is as follows: According to the score

, Distinguish between hard sample and easy sample, and continue to use ChatGPT to generate new data in the form of few shots. In order to ensure the diversity of data and the number of hard samples, the ratio is 1:1. The template is as follows:

Then the new data replaces the existing data in the Train Pool and inserts it into the Cache Pool.

Continue to reciprocate the previously depicted operations.

experiment