Author: Zen and the Art of Computer Programming
1 Introduction
Model Distillation is a process of compressing a complex large model into a small model. Traditional model compression methods will lose some characteristics or details of the model, resulting in poor results for the final small model. Model distillation can preserve these details, thereby improving the performance of the final model. Distillation methods can be divided into three types: soft model distillation, hard model distillation and joint distillation.
Soft model distillation: through the optimization of the loss function, the sub-model can be fitted to the output of the main model, that is, the sub-model is required to be as close as possible to the target function in terms of loss function, and to reduce the loss on the target function to some extent. In practice, KL divergence is usually used as the objective function, and the smaller the distance between the two, the more accurate the knowledge learned by the sub-model. However, there is a correlation between the loss functions of different layers, so it is necessary to consider how to accumulate the loss functions of different layers for optimization.
Hard model distillation: Strengthen the ability of the main model by changing the network structure, rather than relying solely on the loss function. For example, use a narrower neural network model to replace the current network structure. In order to ensure that the network accuracy required for soft model distillation remains unchanged, it is also possible to use the distilled network structure as input for the main model and compress it into a smaller model.
Joint distillation: Both soft model distillation and hard model distillation belong to separate distillation tasks. But when two models of a task need to learn together, joint distillation is needed. The basic idea of joint distillation is to train two sub-models, one for capturing the global information of the large model and the other for capturing the local information of the large model. The latter will better describe the distribution characteristics of the training data. To achieve this, constraints can be introduced between the two models, such as using a Laplace distribution.
Overall, model distillation is an effective transfer learning method that can balance performance and efficiency. With the distillation method, we can compress the model into a small volume model adapted to the specific task requirements, while maintaining the overall performance of the original model. At the same time, model distillation can also solve many limitations caused by the lack of sufficient training data. In addition, distillation-based pre-training models can help improve generalization and enhance model