[Efficient Alchemy - Foundation Building Period] When deep learning meets probability distribution, you must know the magical measurement method: KL divergence!

What is KL divergence

KL divergence (Kullback-Leibler divergence), also known as relative entropy, is a measure of the difference between two probability distributions. It can describe the distance between two distributions and is used to measure the difference between the model prediction results and the true distribution. It is one of the commonly used measurement methods in deep learning.

Application scenarios of KL divergence

KL divergence is widely used in deep learning. It is often used to measure the difference between the model's predicted distribution and the true distribution to guide model optimization. It can also be used for tasks such as cluster analysis to help us understand the differences between different data. In practical applications, KL divergence is often used together with cross-entropy loss to measure the difference between the prediction results of a classification model and the true class.

Calculation formula of KL divergence

The calculation formula of KL divergence is as follows:

$$D_{KL}(P||Q)=\sum _{i}P(i)\log \frac{P(i)}{Q(i)}$$

Among them, $P$ and$Q$ represents two probability distributions respectively,