Speech and Language Processing之Part-of-Speech Tagging

Tagging
is a disambiguation task, where words are ambiguous and have more than one possible part of speech, and the goal is to find the correct label for the situation. For example, book can be a verb (book that flight) or a noun (hand me that book). This can be a determiner (Does that flight serve dinner) or a complement (I thought that your flight was earlier). The goal of pos tagging is to resolve these ambiguities, choosing an appropriate tag for the context. How common is label ambiguity?

1. HMM Algorithm
In this section, we will introduce the use of hidden Markov models for part-of-speech tagging. HMM is a sequence model. A sequence model or sequence classifier is a model whose job is to assign a label or class to each unit in the sequence, thereby mapping a sequence of observations to a sequence of labels. HMM is a probabilistic sequence model: given a sequence of units (words, letters, morphemes, sentences, etc.), it computes the probability distribution of possible label sequences and chooses the best one.

  HMM是基于马尔可夫链的增广。马尔可夫链是一个模型，它告诉我们随机变量，状态序列的概率，每一个都可以取某个集合的值。这些集合可以是单词、标签或表示任何东西的符号，例如天气。马尔可夫链做了一个非常强的假设，如果我们想要预测序列的未来，所有重要的是当前状态。所有在当前状态之前的状态，除了通过当前状态之外，对未来没有任何影响。这就好像为了预测明天的天气，你可以查看今天的天气，但你不能查看昨天的天气。

More formally, consider a series of state variables q1,q2,...,qi. Markov models embody this Markovian assumption of sequential probability: when predicting the future, the past does not matter, only the present.

   当我们需要计算一系列可观察事件的概率时，马尔可夫链是有用的。然而，在许多情况下，我们感兴趣的事件是隐藏的:我们没有直接观察到它们。例如，我们通常不会在文本中观察到词性标签。相反，我们看到单词，并且必须从单词序列中推断出标签。我们称这些标签为隐藏的，因为它们不被观察到。

Hidden Markov Model (HMM) is a commonly used sequence modeling tool, which is often used in natural language processing, speech recognition, bioinformatics and other fields. The HMM model is a statistical model used to describe a Markov process with implicit unknown parameters.

An HMM model usually consists of three parts: state sequence, observation sequence and model parameters.

State sequence $S = \{s_1, s_2, ..., s_T\}$ indicates a hidden state.
View order $O = \{o_1, o_2, ..., o_T\}$ denote observations generated from a sequence of states.
model reference number $\lambda = (\pi, A, B)$ includes:
- Initial state probability vector $\pi = (\pi_1, \pi_2, ..., \pi_N)$ , where $\pi_i$ Indicates that it is initially in state $probability of i$ .
- State transition matrix $[a_{ij}]_{N \times N}$ , among which $a_{ij}$ Indicates from state $i$ transitions to state $The probability of j$ .
- Emission matrix $[b_j(o_t)]_{N \times M}$ , in which $b_j(o_t)$ means that in state $o_t$ is generated under $j$ $o_{t}$ The probability.

In the HMM model, there are two basic problems to be solved:

Given a model and an observation sequence $O$ , how to calculate the observation sequence $The probability P$ corresponding to $P(O|\lambda)$ 。
Given a model and an observation sequence $O$ , how to find the most probable state sequence $S$ 。

Forward-Backward Algorithm for HMM Models

The forward-backward algorithm can be used to solve the first problem: Given a model and an observation sequence $O$ , how to calculate the observation sequence $The probability P$ corresponding to $P(O|\lambda)$ 。

设 $\alpha_t(i)$ represents time $t$ state is $i$ , the observation sequence is $o_1, o_2, ..., o_{t}$ The sum of the probabilities of all possible paths of , namely:

$\alpha_t(i) = P(o_1, o_2, ..., o_t, q_t=s_i|\lambda)$

Using the dynamic programming method, it is possible to recursively calculate $\alpha_t(i)$ ：

$\alpha_t(i) = \sum_{j=1}^N \alpha_{t-1}(j) a_{ji} b_i(o_t)$

where $\alpha_1(i) = \pi_i b_i(o_1)$ 。

The probability of observing the sequence $P(O|\lambda)$ is $\alpha_T(i)$ all states $The sum of i$ :

$P(O|\lambda) = \sum_{i=1}^N \alpha_T(i)$

The backward algorithm can be used to solve the second problem: given the model and observation sequence $O$ , how to find the most probable state sequence $S$ 。

设 $\beta_t(i)$ represents time $t$ state is $i$ , the observation sequence is $o_{t+1}, o_{t+2}, ..., o_T$ The sum of the probabilities of all possible paths of , namely:

$\beta_t(i) = P(o_{t+1}, o_{t+2}, ..., o_T|q_t=s_i, \lambda)$

$\beta_t(i)$ can be calculated recursively $b_{t} (i)$ ：

$\beta_t(i) = \sum_{j=1}^N a_{ij} b_j(o_{t+1}) \beta_{t+1}(j)$

Among them, $\beta_T(i) = 1$ 。

Given an observation sequence $O$ , you can use the forward-backward algorithm to calculate each time $t$ state isThe probability of $i$ $P(q_t=s_i|O,\lambda)$ , that is, in the observation sequence $O$ next, moment $t$ is in state $s_i$ The probability. The probability can be calculated by the results of the forward algorithm and the backward algorithm:

$P(q_t=s_i|O,\lambda) = \frac{\alpha_t(i)\beta_t(i)}{P(O|\lambda)}$

Next, the most probable state sequence $S$ 。

Viterbi algorithm for HMM models

The Viterbi algorithm is a dynamic programming algorithm for solving the most probable state sequence $S.$ _ Specifically, let $\delta_t(i)$ represents time $t$ state is $i$ , the observation sequence is $o_1, o_2, ..., o_t$ The probability of the path with the highest probability among all possible paths of , namely:

$\delta_t(i) = \max_{s_1,s_2,...,s_{t-1}} P(s_1,s_2,...,s_{t-1},q_t=s_i,o_1,o_2,...,o_t|\lambda)$

Using the dynamic programming method, it is possible to recursively calculate $\delta_t(i)$ ：

$\delta_t(i) = \max_{1 \leq j \leq N} \{\delta_{t-1}(j) a_{ji}\} b_i(o_t)$

where $\delta_1(i) = \pi_i b_i(o_1)$ 。

In order to find the most probable state sequence $S$ $\psi_t(i)$ in the recursion process $p_{t} (i)$ array, representing time $t$ state isThe value of the previous state on the path with the greatest probability of $i .$ specifically:

$\psi_t(i) = \arg\max_{1 \leq j \leq N} \{\delta_{t-1}(j) a_{ji}\}$

$\delta_t(i)$ calculated according to recursion $d_{t} (i)$ 和 $\psi_t(i)$ array, the most likely state sequence $S$ ：

at time $T$ , choose such that $\delta_T(i)$ Maximum state $i$ as the final state.
For $t = T - 1, T - 2, ..., 1$ , select in turn to make $\delta_t(i)\times a_{ij}$ max state $j$ as timestate $s_t$ $of t$ $s_{t}$ 。
The final state sequence $S = \{s_1, s_2, ..., s_T\}$ is the most likely state sequence.

In conclusion, the forward-backward algorithm and Viterbi algorithm of HMM model solve two basic problems of HMM model respectively. The former is used to calculate the probability of the observation sequence, and the latter is used to solve the most likely state sequence.