Amount of Information, Entropy Entropy, Cross Entropy Cross Entropy, KL Divergence KL Divergence, Cross Entropy Loss Function Cross Entropy Loss

1. Amount of Information

• Information volume: A measure of how difficult it is for an event to occur
  • Small probability events, it is more difficult to happen, so there is a larger amount of information
  • High probability event, it is less difficult to happen, so there is a smaller amount of information

Information formula: $I(x):=lo{g}_2(\frac{1}{p_{(x)}})=-lo{g}_2(p_{\left(x\right)})$

Properties: For independent events A and B: $p_{(AB)}=p_{(A)}{p}_{(B)}$, the amount of information of two events occurring at the same time is equal to the addition of the amount of information of the two events: $I(AB)=I(A)+I(B)$

$\to I\left(AB\right)=lo{g}_2(\frac{1}{p_{(AB)}})=lo{g}_2\left(\frac{1}{p_{(A)}{p}_{(B)}}\right)=lo{g}_2\left(\frac{1}{p_{\left(A\right)}}\right)+lo{g}_2\left(\frac{1}{p_{\left(B\right)}}\right)=I\left(A\right)+I\left(B\right)$

$0\le p_{(x)}\le 1$

Example 1: Flip a coin, heads probability $p_{(A)}=0.5$ , tails probability$p_{(B)}=0.5$

$\to I(A)=-lo{g}_2(0.5)=1$ , $I(B)=-lo{g}_2\left(0.5\right)=1$

Example 2: Flip a coin, heads probability $p_{\left(A\right)}=0.2$ , tails probability$p_{\left(B\right)}=0.8$

$\to I(A)=-lo{g}_2(0.2)=2.32$ , $I(B)=-lo{g}_2(0.8)=0.32$

Conclusion: Small probability events have a large amount of information, and high probability events have a small amount of information

---

2. Entropy Entropy

Definition: The expected information content of a probability distribution: $H(p):=E\left(I\right(x))$ , (It can also be understood as: the amount of information of the system as a whole. Among them, the system as a whole is composed of all possible events. For example, flipping a coin, the front and back constitute a whole system)

Function: used to evaluate the degree of uncertainty of the probability model

• The greater the uncertainty, the greater the entropy
• The smaller the uncertainty, the smaller the entropy

Form : $H(p)=\sum p_i{I}_i^p=-\sum p_{i}lo{g}_2(p_i)$

Example 1: Toss a coin, the probability of heads $p_{(A)}=0.5$ , tails probability$p_{(B)}=0.5$

$\begin{array}{rl}H(p)& =\sum p_i{I}_i^p\\ & =p_{(A)}\cdot lo{g}_2(1/p_{\left(A\right)})+p_{(B)}\cdot lo{g}_2(1/p_{\left(B\right)})\\ & =0.5\cdot lo{g}_2(1/0.5)+0.5\cdot lo{g}_2(1/0.5)\\ & =0.5\cdot 1+0.5\cdot 1\\ & =1\end{array}$

Example 2: Flip a coin, heads probability $p_{(A)}=0.2$ , tails probability$p_{(B)}=0.8$

$\begin{array}{rl}H(p)& =\sum p_i{I}_i^p\\ & =p_{\left(A\right)}\cdot lo{g}_2(1/p_{(A)})+p_{(B)}\cdot lo{g}_2(1/p_{\left(B\right)})\\ & =0.2\cdot lo{g}_2(1/0.2)+0.8\cdot lo{g}_2(1/0.8)\\ & =0.2\cdot 2.32+0.8\cdot 0.32\\ & =0.72\end{array}$

Conclusion:
If the probability density is uniform, the uncertainty of the generated random variable is higher, and the value of entropy is larger.
If the probability density is gathered, the certainty of the generated random variable is higher, and the value of entropy is smaller

---

3. Cross Entropy

Suppose the true probability distribution is $p$ , predicted probability distribution (estimated probability distribution) is$q$
definition: predictive probability distribution$q$ versus the true probability distribution$The\; estimation\; of\; the\; average\; amount\; of\; information\; of\; p$ is called cross entropy

公式 ： $H(p,q)=\sum p_i{I}_i^q=-\sum p_{i}lo{g}_2(q_i)$

Example 1: Toss a coin, the true probability of heads $p(A)=0.5$ , the real probability of tails$p(B)=0.5$ ; positive estimated probability$q(A)=0.2$ , negative estimated probability$q(B)=0.8$

$\begin{array}{rl}H(p,q)& =-\sum p_{i}lo{g}_2(q_i)\\ & =p_{(A)}\cdot lo{g}_2(1/q_{\left(A\right)})+p_{(B)}\cdot lo{g}_2(1/q_{\left(B\right)})\\ & =0.5\cdot lo{g}_2(1/0.2)+0.5\cdot lo{g}_2(1/0.8)\\ & =0.5\cdot 2.32+0.5\cdot 0.32\\ & =1.32\end{array}$

Example 2: Toss a coin, the true probability of heads $p(A)=0.5$ , the real probability of tails$p(B)=0.5$ ; Positive estimated probability$q(A)=0.4$ , negative estimated probability$q(B)=0.6$

$\begin{array}{rl}H(p,q)& =-\sum p_{i}lo{g}_2(q_i)\\ & =p_{(A)}\cdot lo{g}_2(1/q_{\left(A\right)})+p_{(B)}\cdot lo{g}_2(1/q_{\left(B\right)})\\ & =0.5\cdot lo{g}_2(1/0.4)+0.5\cdot lo{g}_2(1/0.6)\\ & =0.5\cdot 1.32+0.5\cdot 0.74\\ & =1.03\end{array}$

Conclusion:
(1) The closer the estimated probability distribution is to the real probability distribution, the smaller the cross entropy.
(2) The value of cross entropy is always greater than the value of entropy (according to Gibbs inequality)

---

4. Relative entropy (KL divergence, KL Divergence)

KL divergence is named after Kullback and Leibler, also known as relative entropy

Role: Used to measure the difference between two probability distributions

Official:

$\begin{array}{rl}D(p||q)& =H(p,q)-H(p)\\ & =\sum p_{i}lo{g}_2(1/q_i)-\sum p_{i}lo{g}_2(1/p_i)\\ & =\sum p_i[lo{g}_2(1/q_i)-lo{g}_2(1/p_i)]\\ & =\sum p_i[I_q-I_p]\#I_q-I_{p}difference\; in\; amount\; of\; information\\ & =\sum p_{i}lo{g}_2(p_i/q_i)\end{array}$

Important properties:
(1) According to Gibbs inequality: $D(p||q)\ge 0$ ; when the distribution q is exactly the same as the distribution p,$D(p||q)=0$

（2）$D(p||q)$ and$D(q||p)$ is different, that is,$D(p||q)\ne D(q||p)$

• $D(p||q)$ represents the estimated probability distribution qqbased on p (the real probability distribution)$q$ and the true probability distributionppthe gap between$p$
• $D(q||p)$ represents the estimated probability distribution ppbased on q (the real probability distribution)$p$ and the true probability distributionqqthe gap between$q$

---

5. Cross Entropy Loss Function Cross Entropy Loss

It can be seen from the above that the KL divergence $D(p||q)$ represents the gap between the predicted distribution q and the real distribution p, so we can directly define the loss function as the KL divergence:$Loss=D(p|\; |q)$
and we hope that the predicted distribution q of the model is exactly the same as the real distribution p, namely: Loss function$Loss=D(p||q)=0$

$$Loss\; function:Loss=D(p||q)=H(p,q)-H(p)=\sum p_{i}lo{g}_2(1/q_i)-\sum p_{i}lo{g}_2(1/p_i)\text{\hspace{1em}(1)}$$

For classification problems, the real distribution is a single-point distribution, the probability of the real category is 1, and the probability of other categories is 0, similar to the following:

category	class1	class 2	class 3	class 4
probability	0	0	1	0

$p_{class1}=p_{class2}=p_{class4}=0,lo{g}_2(1/p_{class3})=0$

所以， $H(p)=\sum p_{i}lo{g}_2(1/p_i)=0$

The loss function (1) can be further simplified as: $$Loss=D(p||q)=H(p,q)-H\left(p\right)=H(p,q)\text{\hspace{1em}(2)}$$

$H(p,q)$ is cross entropy, so the loss function is also called cross entropy loss function:
$$Cross\_Entropy\_Loss=H(p,q)=-\sum p_{i}lo{g}_2(q_i)\text{\hspace{1em}(3)}$$

And because the real distribution is a single-point distribution, the probability of the real class $p_{class}=1$ , the probability of other classes$p_{\overline{class}}=0$

$$Cross\_Entropy\_Loss=H(p,q)=-lo{g}_2(q_{class})$$