Evaluation language model Perplexity

Evaluation language model Perplexity

More: https://github.com/fansking/NlpWithMe
how to assess the quality of a language model it? A good language model generates a probability of error of normal sentences and sentence should be different. For example: mice love to eat rice and rice rat to eat. The two sentences were put into language model generates a probability that the first sentence should be larger.

Perplexity (perplexity) The basic idea is: to sentence test set gives a higher probability value of the language model is better, when the language model training completion, the test set of sentences are normal sentences, then the model is trained in the probability of the test set as high as possible

$PP(W)=P(w_{1}w_{2}...w_{N})^{-\frac{1}{N}}=\sqrt[N]{\frac{1}{P(w_{1}w_{2}...w_{N})}}$

According to the n-gram model, $P(w_{1}w_{2}...w_{N})$ Can be counted out (maximum likelihood probability)

Note that $w_{1}w_{2}...w_{N}$ Word sentence is obtained word sequence, N is the total number of words, since the addition of a negative power, so the greater the probability of a sentence, the better the language model perplexity perplexity i.e. less.