Evaluation indicators (Metric) (4)

BLUE

BLEU (BiLingual Evaluation Understudy) was first used in machine translation tasks to evaluate the rationality of machine-translated statements. Specifically, BLEU is calculated by measuring the overlap between the generated sequence and the reference sequence. Below we will use machine translation as an example to discuss this indicator.

Suppose there is currently a source text $s,$ and the corresponding translation reference sequence$r_1,r_2,...,r_n$​​​​​​. The machine translation model is based on the source text $sgenerated$ a generated sequence$x$ ​​​​​​​​​​，And$Wis$ based on the candidate sequence$The\; N-ary\; word\; combinations\; generated\; by\; x$ , the accuracy of these N-ary combinations is:$$P_N(x)=\frac{{\sum }_{w\in W}min(c_w(x),ma{x}_{k=1}^n{c}_w(r_k))}{{\sum }_{w\in W}{c}_w\left(x\right)}$$Among them, $c_w\left(x\right)$ ​is an N-ary compound word$w$​ is generating the sequencexxThe number of occurrences in$x$$c_w(r_k)$ ​is an N-ary compound word$w$​ in the reference sequence$r_k$The number of times it appears $in$Accuracy of N-element combination$P_N\left(x\right)$ is the proportion of N-gram combinations in the generated sequence that appear in the reference sequence.

As can be seen from the above formula, $P_{N}The\; core\; idea\; of\; (x)$ is to measure the generated sequence$Whether\; the\; N-gram\; compound\; words\; in\; xappear$ in the reference sequence, the calculation result prefers the short generated sequence, that is, the generated sequencexxThe shorter$x$$P_N\left(x\right)$ ​​​​will be higher. In this case, a length penalty factor can be introduced. If the sequence$x$​ is compared to the reference sequence$r_k$short, the sequence xx will be generated$x$ 进行惩罚。$$b(x)=\{\begin{array}{ll}1& \text{if }{l}_x>l_r\\ \exp (1-l_s/l_r)& \text{if }{l}_s\le l_r\end{array}$$Among them, $l_{x}Represents$ the generated sequence$The\; length\; of\; x$ ,$l_r$​Represents the reference sequence $l_r$The minimum length of .

A concept has been mentioned repeatedly before - N-ary compound words, we can generate the sequence $x$ ​​Construct N-ary compound words of different lengths, so that the accuracy of compound words of different lengths can be obtained, such as$P_1(x)，P_2\left(x\right)，P_3\left(x\right)$ ​And so on. The BLEU algorithm calculates the accuracy PN (x) P_N(x)of N-element combinations of different lengths$P_N(x)$，$N=1,2,\mathrm{3...}$ , and obtained by geometrically weighted average, as shown below. $$BLUE-N(x)=b(x)\times \exp (\sum _{N=1}^{N^{\prime }}{a}_N\log P_N)$$ where,$N\prime$​ is$a_{N}is$ the weight of different N-gram combinations, generally set to$\frac{1}{N\prime },$ The value range of the BLEU algorithm is [0,1]​. The larger the value, the better the quality of the generation.

The BLEU algorithm can better calculate whether the words in the generated sequence x have appeared in the reference sequence, but it does not pay attention to whether the words in the reference sequence have appeared in the generated sequence. That is, BLEU only cares about the accuracy of the generated sequence, not its recall.

ROUGE

Since the BLEU algorithm only cares about whether the words in the generated sequence appear in the reference sequence, and does not care about whether the words in the reference sequence appear in the generated sequence, this may have some impact in the actual index evaluation process, so it cannot be compared. Good for assessing the quality of generated sequences.

The ROUGE (Recall-Oriented Understudy for Gisting Evaluation) algorithm is a solution. It can measure whether the words in the reference sequence have appeared in the generated sequence, that is, it can measure the recall rate of the generated sequence. Let's take machine translation as an example to discuss the calculation of ROUGE.

Assume that there is currently a source text $s$ , and the corresponding translation reference sequence$r_1,r_2,...,r_n$. The machine translation model is based on the source text $s$ generates a generated sequence$x$ , and$W$ is based on the candidate sequencexxFor the N-word combination $generated$ by$$ROUGE-N(x)=\frac{{\sum }_{k=1}^n{\sum }_{w\in W}min(c_w(x),c_w(r_k))}{{\sum }_{k=1}^n{\sum }_{w\in W}{c}_w(r_k)}$$Among them, $c_w\left(x\right)$ is the N-ary compound word$w$ is generating sequencexxThe number of occurrences in$x$$c_w\left(r_k\right)$ is the N-ary compound word$w$ is in the reference sequence$r_k$the number of times it appears.

As can be seen from the formula, the ROUGE algorithm can better calculate whether the words in the reference sequence have appeared in the generated sequence, but it does not pay attention to whether the words in the generated sequence have appeared in the reference sequence. That is, the ROUGE algorithm only cares about the generated sequence. Recall, not precision.