[Part 20] Knowledge Distillation: Distilling the Knowledge in a Neural Network

Summary

A very simple way to improve the performance of almost any machine learning algorithm is to train many different models on the same data and then average their predictions [3]. Unfortunately, making predictions using the entire ensemble of models is cumbersome and likely too computationally expensive to deploy to a large number of users