Attention principle + vector inner product + Scaled Dot-Product Attention in Transformer

1. Attention principle

  将$The\; constituent\; elements\; in\; Source$ are imagined as a series of$<Key,Value>$ Data pair composition, givenT arget TargetAn element$in$$T$$a$$r$$get$$Query$$Query$ , by computing$Query$ and eachKey KeyThe similarity or correlation of$Key$$Key$对应 V a l u e Value The weight coefficient of$V$$l$$u$$e$$Value$ is weighted and summed to get the final$Attention$$Attention$ value. Soessentially$Attention$ mechanism is forSource SourceValue Valueof elements in$S$$o$$u$$rce$$Vlue$ values ​​are weighted and summed, and$Query$ and$Key$ is used to calculate the corresponding$Value\text{'}s$ weight$coefficient$ .$\_$That is, its essential idea can be rewritten as the following formula:

$$Attention(Query,Source)=\sum _{i=1}^{L_x}Similarity(Query,Ke{y}_i)\ast Valu{e}_i$$

2. Vector inner product

  Vector inner product is also called vector dot product, the formula is as follows:

$$\vec{a}\cdot \vec{c}=\parallel \vec{a}\parallel \times \Vert \vec{c}\Vert \times cos\theta$$

  The derivative formula of vector inner product is as follows:

$$\frac{\partial (\bar{x}\cdot \bar{w})}{\partial \bar{w}}={\bar{x}}^T$$

3. Scaled Dot-Product Attention in Transformer

  The formula is as follows:

$$Attention(Q,K,V)=softmax(\frac{Q{K}^T}{\sqrt{d_k}})V$$

  For a set of key-value pairs and n queries, two matrix multiplications can be used to calculate each element in parallel.