[Deep Learning] Sequence Generation Model (3): N-ary Statistical Model

N-ary statistics model

  N-Gram Model is a commonly used sequence modeling method, especially when dealing with data sparse problems. This model is based on the Markov hypothesis, which assumes that the generation of the current word only depends on its previous $N-1$ word.
  The core idea of ​​the N-ary model is to use the previous$N-The\; historical\; information\; of\; a$ word is used to estimate the conditional probability of the current word. For an N-ary model, the conditional probability can be expressed as:
$$p(x_t|{\mathbf{x}}_{1:(t-1)})\approx p(x_t|x_{t-(N-1):t-1})$$ in which$x_{t-(N-1):t-1}$means from $x_{t-(N-1)}$to $x_{t-1}$N $N-A\; sequence\; of\; 1$ word.

• When $N=When\; 1$ , it is called a unigram model.
  • The generation of each word is only related to itself and has nothing to do with any previous words.
• When $N=2$ , it is called a binary (Bigram) model.
  • The generation of each word depends only on its preceding word.
• When $N=3$ , it is called the Trigram model.
  • The generation of each word depends on its two previous words.
• By analogy, when $As\; N$ increases, the historical information considered by the model also increases.

  This assumption of Markov properties simplifies the complexity of probabilistic modeling, but it also brings about the problem of data sparsity. Especially when N is large, the specific combinations contained in the training data may be very limited. In order to solve the problem of data sparseness, smoothing technology is often used to assign a certain prior probability to combinations that do not appear in the training data. Additive smoothing is a common smoothing technique that adds a small constant δ \delta when calculating conditional probabilities.$\delta$ , to avoid the denominator being zero.

1. Univariate model

1.1 Overview

• Definition: A unigram model is a special case of an N-gram statistical model, in which the generation probability of each word is independent of other words and has nothing to do with context.

• Build a model :
    In the unary model, the occurrence of each word in the sequence is independent. For a given sequence $x_{1:T}$For example:
  $$p(x_{1:T};i)=\prod _{t=1}^{T}p\left(x_t\right)=\prod _{k=1}^{|V|}{i}_k^{m_k}$$where, $p\left(x_t\right)$ is the word$x_t$Probability of occurrence in the entire word list.

1.2 Generation probability

• Multinomial distribution assumption: The probability of word generation conforms to a multinomial distribution, and the parameter is the probability of each word.
    In the unary model, the probability of each word depends only on the frequency of that word in the entire vocabulary. This can be modeled by a multinomial distribution, where $i=[i_1,i_2,\dots ,i_{|V|}]$ is a parameter of the multinomial distribution, indicating the selection probability of each word in the sequence.

1.3 Maximum likelihood estimation

• Maximum likelihood estimation of a univariate model can be transformed into a constrained optimization problem

  • Goal: Determine the multinomial distribution parameters through maximum likelihood estimation to maximize the likelihood of the entire training set.

  • Constrained optimization: introduce Lagrange multipliers to obtain frequency estimates.

• Specific process

  • Log-likelihood function :

    For a set of training sets { x 1 : T n ( n ) } n = 1 N ′ \{\mathbf{x}^{(n)}_{1:T_n}\}_{n=1}^{N '}{ x1:Tn​(n)​}n=1N′​,In other words:
    $$\log \left(\prod _{n=1}^{N^{\prime }}p({\mathbf{x}}_{1:T_n}^{\left(n\right)};i)\right)=\log \prod _{k=1}^{|V|}{i}_k^{m_k}=\sum _{k=1}^{|V|}{m}_k\log i_k$$

    Among them, $m_k$It’s the $The\; number\; of\; times\; k$ words appear in the entire training set.

  • Maximum likelihood estimation problem :
      This is a maximum likelihood estimation problem. We need to optimize the parameter $\theta$ to maximize the log-likelihood function.

  • Introducing Lagrange multipliers :

    Introducing the Lagrange multiplier $\lambda$ , define the Lagrangian function$L(i,\lambda )$ is:

  $$L(i,l)=\sum _{k=1}^{|V|}{m}_k\log i_k+l\left(\sum _{k=1}^{|V|}{i}_k-1\right)$$

  • Find the partial derivative :

    For parameters $i_k$and $\lambda$ can be infinite, and one of the following is:$$\frac{\partial \Lambda (\theta ,l}{\partial {\theta }_k}=\frac{m_k}{i_k}+l=0,k=1,2,\dots ,|V|$$ $$\frac{\partial \Lambda (\theta ,l}{\partial \lambda }=\sum _{k=1}^{|V|}{i}_k-1=0$$ further solve to get$l=-{\sum }_{k=1}^{|V|}{m}_k$θ $i_k=\frac{m_k}{\bar{m}}$, where $\bar{m}={\sum }_{k^{\prime }=1}^{|V|}{m}_{k^{\prime }}$is the length of the document collection.

  • Final result :
      Finally, the maximum likelihood estimate of the unary model is equivalent to the frequency estimate, parameter $i_k$The estimated value is $\frac{m_k}{\bar{m}}$。

2. N-ary model

  In the N-element model, the conditional probability $p(x_t|x_{t-N+1:t-1})$ means N − 1 N-1in front of the given$N-In\; the\; case\; of\; 1$ word, the$The\; probability\; of\; occurrence\; of\; t$ words. This probability can be obtained by maximum likelihood estimation:
$$p(x_t|x_{t-N+1:t-1})=\frac{m(x_{t-N+1:t})}{m(x_{t-N+1:t-1})}$$

其中 $m\left(x_{t-N+1:t}\right)$ represents the sequence$x_{t-N+1:t}$The number of occurrences, and $m\left(x_{t-N+1:t-1}\right)$ represents the sequence$x_{t-N+1:t-1}$The number of occurrences.

3. Smoothing technology

3.1 Data sparsity problem

• Challenge: N-ary models face the problem of data sparseness, especially for unseen N-ary combinations. Data sparseness causes the model to have zero probability for unseen N-element combinations.

3.2 Smoothing technology

  N-ary models face the problem of data sparsity, especially when the training data set is relatively small. The data sparse problem refers to the fact that due to insufficient training samples, the probability estimate of the model for some N-gram combinations that may appear but is not observed in the training set is zero, which will affect the generalization ability of the model. In natural language processing, this problem is particularly significant because most words in natural languages ​​obey Zipf's law, that is, the words with the highest frequency appear much more than other words .
  Smoothing technology is a method to solve the problem of data sparseness. Its basic idea is to alleviate the excessive penalty of unseen events by the model by assigning some probability mass to unseen events. Here are some common smoothing techniques:

1. Add-One Smoothing :
     Add-One Smoothing is a simple and intuitive smoothing technique that works by adding a small constant $\delta$ to avoid the zero probability problem. For the conditional probability in the N-ary model, the calculation formula of additive smoothing is:
   $$p(x_t|x_{(t-N+1):(t-1)})=\frac{m\left(x_{(t-N+1):t}\right)+}{m\left(x_{(t-N+1):(t-1)}\right)+\delta |V|}$$

   where the constant $\delta$ takes a value between 0 and 1, taking$d=$When$1$ is called plus 1 smoothing.

2. Good-Turing Smoothing :
     Good-Turing smoothing is a more complex but more effective smoothing technique that adjusts for observed frequencies and expected frequencies of unobserved events. It weights low-frequency events and reduces the estimate of high-frequency events. This method requires frequency distribution statistics of the training data.

3. Kneser-Ney smoothing :
     Kneser-Ney smoothing is an advanced smoothing technique, especially suitable for N-ary models. It considers the frequency of N-1-ary prefixes and N-ary combinations, improving the performance of the model by recursively considering shorter prefixes.

4. Application

• Widely used: N-ary models are widely used in the field of natural language processing, including speech recognition, machine translation, Pinyin input method, etc.