Introduction to Deep Learning (66) Recurrent Neural Networks - Beam Search

foreword

The core content comes from blog link 1 blog link 2 I hope you can support the author a lot
This article is used for records to prevent forgetting

Recurrent Neural Networks - Beam Search

courseware

greedy search

In seq2seq we use greedy search to predict the sequence

• Output the word with the highest prediction probability at the current moment

But being greedy is probably not optimal:

exhaustive search

Optimal algorithm: For all possible sequences, calculate its probability, and then choose the best one.
If the output dictionary size is n, and the longest sequence is T, then we need to examine nT sequences

• n = 10000，T= 10: $n^T=10^{40}$
• computationally infeasible

beam search

Save the best k candidates
At each moment, add a new item (n possibilities) to each candidate, and select the best k among the kn options

Summarize

beam search saves the k best candidates at each search

• When k = 1, it is a greedy search
• Exhaustive search when k = n

Textbook

In the previous section, we predicted the output sequence one by one until the specific sequence-ending token “ ” appeared in the predicted sequence <eos>. This section will first introduce 贪心搜索（greedy search）the strategy and explore its problems, and then compare other alternative strategies: 穷举搜索（exhaustive search）and 束搜索（beam search）.

Before formally introducing greedy search, we define the search problem using the same mathematical notation as in the previous section. At any time step $t^{\prime }$ , the decoder outputs$y_{t^{\prime }}$The probability of depends on the time step $t^{\prime }$ The previous output subsequence$y_1,\dots ,y_{t^{\prime }-1}$And the context variable c obtained by encoding the information of the input sequence $c$ . To quantify the computational cost, use$Y$ represents the output vocabulary, which contains "<eos>", so the cardinality of this vocabulary set$|Y|$ is the size of the vocabulary. We also specify the maximum number of tokens for the output sequence as$T^{\text{'}}$ . Therefore, our goal is to start from all$\mathcal{O}({\left|\mathcal{Y}\right|}^{T^{\text{'}}})$to find the ideal output among possible output sequences. Of course, for all output sequences,<eos>the part after " " (not this sentence) will be discarded in the actual output.

1 Greedy search

First, let's look at a simple strategy: greedy search, which was used for sequence prediction in the previous section. For each time step of the output sequence, we will find the token with the highest conditional probability based on greedy search, namely:
$$y_{t^{\prime }}=\underset{y\in \mathcal{Y}}{\mathrm{argmax}}P(and\mid y_1,\dots ,y_{t^{\prime }-1},\mathbf{c})$$

Once the output sequence contains " <eos>" or reaches its maximum length $T^{’}$ , the output is complete.

As shown in the figure, suppose there are four tokens "A", "B", "C" and "<eos>" in the output. The four numbers under each time step represent the<eos>conditional probabilities of generating "A", "B", "C" and " " at that time step respectively. At each time step, greedy search selects the token with the highest conditional probability. Therefore, the output sequence "A" "B" "C" and " " will be predicted in the graph<eos>. The conditional probability of this output sequence is$0.5\times 0.4\times 0.4\times 0.6=0.048$ .
So what is the problem with greedy search? In reality,最优序列（optimal sequence）it should be maximized${\prod }_{t^{\prime }=1}^{T^{\prime }}P({and}_{t^{\prime }}\mid y_1,\dots ,y_{t^{\prime }-1},c)$ The output sequence of values, which is the conditional probability of generating the output sequence based on the input sequence. However, greedy search cannot guarantee the optimal sequence.

Another example in the image above illustrates the problem. Unlike the previous figure, at the time step, we choose the token "C" in Figure 2, which has the second highest conditional probability. Since the output subsequence at the time step and at which the time step is based has changed from "A" and "B" in Figure 1 to "A" and "C" in Figure 9.8.2, each time step at The conditional probabilities of tokens are also changed in Fig. 2. Assuming that we select the word "B" at the time step, then the current time step is based on the output subsequences "A", "C" and "B" of the first three time steps as conditions, which is the same as "A"" in Figure 1 B" and "C" are different. Therefore, the conditional probability of generating each token at the time step in Figure 2 is also different from that in Figure 1. As a result, the<eos>conditional probability of the output sequence "A", "C", "B" and " " in Figure 2 is$0.5\times 0.3\times 0.6\times 0.6=0.054$ , which is greater than the conditional probability of greedy search in Figure 1. This example shows that the output sequence "A", "B", "C" and "<eos>" obtained by greedy search is not necessarily the best sequence.

2 exhaustive search

If the goal is to obtain the optimal sequence, we can consider using 穷举搜索（exhaustive search）: Exhaustively enumerate all possible output sequences and their conditional probabilities, and then calculate the one with the highest output conditional probability.

Although we can use exhaustive search to obtain the optimal sequence, but the calculation amount $\mathcal{O}({\left|\mathcal{Y}\right|}^{T^{\prime }})$can be surprisingly high. For example, when$|\mathcal{Y}|=10000$ and$T^{\prime }=At\; 10$ , we need to evaluate$10000^{10}=10^{40}$ sequence, this is a huge number, and it is almost impossible for existing computers to calculate it. However, the calculation amount of greedy search$\mathcal{O}(\left|\mathcal{Y}\right|T^{\prime })$, which is significantly smaller than the exhaustive search. For example, when$|Y|=10000$ and$T^{\prime }=10$ , we only need to evaluate$10000\times 10=10^5$ sequences.

3 beam search

So which sequence search strategy should you choose? If precision is most important, then obviously exhaustive search. Obviously greedy search if computational cost matters most. The practical application of beam search lies between these two extremes.

束搜索（beam search）It is an improved version of greedy search. It has a hyperparameter called 束宽（beam size）k. At time step 1, we select the k tokens with the highest conditional probabilities. These k tokens will be the first tokens of the k candidate output sequences respectively. At each subsequent time step, based on the k candidate output sequences from the previous time step, we will proceed from $k|Y|$ picks the k candidate output sequences with the highest conditional probabilities out of the possible choices.

The figure above demonstrates the process of beam search. Suppose the output vocabulary contains only five elements: Y = { A , B , C , D , E } \mathcal{Y} = \{A, B, C, D, E\}Y={ A,B,C,D,E } , one of which is "<eos>". Set the beam width to 2 and the maximum length of the output sequence to 3. At time step 1, the hypothesis has the highest conditional probability$P({and}_1\mid$The lemmas of$c$$)\; are\; A\; and\; C.$At time step 2, we compute all$y_2\in Y1$ :
$$\begin{array}{r}\begin{array}{r}P(A,y_2\mid \mathbf{c})=P(A|\mathbf{c})P(y_2|A,\mathbf{c}),\\ P(C,y_2|c)=P(C|c)P(y_2|C,\mathbf{c}),\end{array}\end{array}$$

Choose the largest two from these ten values, such as $P(A,B|c)$和$P(C,E\mid c)$ . Then at time step 3, we compute all$y_3\in Y$ is:

$$\begin{array}{r}\begin{array}{r}P(A,B,y_3|\mathbf{c})=P(A,B|\mathbf{c})P(y_3|A,B,\mathbf{c}),\\ P(C,E,y_3\mid \mathbf{c})=P(C,E|\mathbf{c})P(y_3|C,E,\mathbf{c}),\end{array}\end{array}$$

Choose the largest two from these ten values, namely $P(A,B,D|c)$和$P(C,E,D|c)$ , we will get six candidate output sequences: (1) A; (2) C; (3) A, B; (4) C, E; (5) A, B, D; (6) C, E, D.

Finally, based on these six sequences (for example, discarding <eos>the part including " " and after), we obtain the final set of candidate output sequences. We then select the sequence in which the product of conditional probabilities is the highest as the output sequence:

$$\frac{1}{L^a}\log P({and}_1,\dots ,y_L\mid \mathbf{c})=\frac{1}{L^a}\sum _{t^{\prime }=1}^L\log P({and}_{t^{\prime }}|y_1,\dots ,y_{t^{\prime }-1},\mathbf{c}),$$

where $L$ is the length of the final candidate sequence,$\alpha$ is usually set to$0.75$ . Because a longer sequence will have more logarithmic terms in the summation of the above formula, so$L^{\alpha }$ is used to penalize long sequences.

The computational cost of beam search is $O\left(k\left|\mathcal{Y}\right|T^{\prime }\right)$, the result is between greedy search and exhaustive search. In fact, greedy search can be seen as a special type of beam search with a beam width of 1. By flexibly choosing the beam width, beam search can provide a trade-off between accuracy and computational cost.

4 Summary

• Sequence search strategies include greedy search, exhaustive search and beam search.

• The sequence selected by greedy search has the least amount of computation, but the accuracy is relatively low.

• The sequence selected by exhaustive search has the highest precision, but the largest amount of calculation.

• Beam search makes a trade-off between accuracy and computational cost by flexibly choosing the beam width.