Evaluation indicators (Metric) (3)

Perplexity , translated as perplexity in Chinese , is a concept in information theory. It can be used to measure the uncertainty of a random variable, and it can also be used to measure the quality of model training. Usually, the higher the Perplexity value of a random variable, the higher its uncertainty; the higher the Perplexity value of a model during inference, the worse the model's performance, and vice versa.

Perplexity of probability distribution of random variables

For discrete random variables $X$ ​, assuming that the probability distribution can be expressed as$p\left(x\right)$ ​​​, then the corresponding perplexity is:$$2^{H(p)}=2^{-{\sum }_{x\in X}p(x)lo{g}_{2}p\left(x\right)}$$ where,$H\left(p\right)$ is the probability distribution$The\; entropy\; of\; p$ . It can be seen that the greater the entropy of a random variable, the greater its corresponding confusion, and the greater the uncertainty of the random variable.

Perplexity of model distribution

Perplexity can also be used to measure how well a model is trained, that is, it measures the difference between the model distribution and the sample distribution. Generally speaking, during the training process of the model, the closer the model distribution is to the sample distribution, the better the model training will be.

Suppose there is a batch of data $x_1,x_2,x_3,...,x_n$, its corresponding empirical distribution is $p_r\left(x\right)$ . Now we have successfully trained a model$p_{}\left(x\right)$ , then the model distribution$p_{}The\; quality\; of\; (x)$​​​​ can be defined by the degree of confusion:$$2^{H(p_r,p_{})}=2^{-{\sum }_i^n{p}_r(x_i)lo{g}_2{p}_{}\left(x_i\right)}$$ in which,$H(p_r,p_{})represents$ the empirical distribution of the sample${\tilde{p}}_r$and model distribution $p_{}$The cross entropy $between$Assume that each sample$The\; generation\; probability\; of\; xi$ is equal, that is,$p_r\left(x_i\right)=\frac{1}{n}$, then the perplexity of the model distribution can be simplified as: $$2^{H(p_r,p_{})}=2^{-\frac{1}{n}{\sum }_i^{n}lo{g}_2{p}_{}(x_i)}$$

Perplexity in NLP

In the field of NLP, language models can be used to calculate the probability of a sentence. Suppose there is a sentence like $s=w_1,w_2,w_3,...,w_n,$ We can calculate the generation probability of this sentence like this:$$\begin{array}{rl}p(x)& =p(w_1,w_2,...,w_n)\\ & =\prod _{i=1}^{n}p(w_i|w_1,w_2,...,w_{i-1})\end{array}$$After the language model training is completed, how to judge the quality of the language model? This is where confusion comes into play. Generally speaking, the test sets used to evaluate language models are reasonable and high-quality corpus. As long as the language model has a higher perplexity on the test set, it means the language model is better trained, and vice versa.

After understanding the calculation of statement probability, for the statement $s=w_1,w_2,w_3,...,w_n$​​，其困惑度可以这样来定义：$$\begin{array}{rl}perplexity& =p(s{)}^{-\frac{1}{n}}\\ & =p(w_1,w_2,...,w_n{)}^{-\frac{1}{n}}\\ & =\sqrt[n]{\frac{1}{p(w_1,w_2,...,w_n)}}\\ & =\sqrt[n]{\prod _{i=1}^n\frac{1}{p(w_i|w_1,w_2,...,w_{i-1})}}\end{array}$$Obviously, the greater the probability of a sentence in the test set, the smaller the confusion.