Simple understanding of Attention (attention mechanism)

Attention (attention mechanism)

Straightforward understanding : attention mechanism, for data, we have important data and unimportant data . In the process of data processing by the model, if we only focus on the more important parts of the data and ignore the unimportant parts, then the speed of training and the accuracy of the model will become better.

It can be seen from the figure : as human beings, we often focus on more important content.

Therefore, the main purpose of Attention is to imitate human beings, learn to focus on important content parts, find the target, and calculate the similarity of the searched target.

calculation process

We set up three parameters, Q, V, K. The process of the whole attention mechanism is as follows:

1.Q is the most suitable for finding the target

​ 2.K is the most suitable for receiving search

3.V is the content

​ 4.Q (search target), $K=k_1,k_2.....,k_n$, generally use points to multiply Q, K to get the similar value of Q and each K $Q\ast k_n=s_n$ 。

5. Make a layer of $softmax(s_1,s_2,\cdots ,s_n)=a_n$Get the probability of each query object.

​ 6. calculated $a_n\ast V=V^{\text{'}}$ Importance of things, similarity calculation, find Q most similar object.

The summarized formula is: $Attention(Q,K,V)=softmax(\frac{Q\ast K_i}{{\sqrt{d}}_k})\ast V_i$

PS: Why does softmax need to be divided by a ${\sqrt{d}}_k$

softmax () is used as a normalization process. When the obtained probability gap is large, such as (0.05, 0.95), the gap of V obtained by the final point multiplication will be too large.

Generally in the attention mechanism, we often use $\frac{512}{8}$as processing.