MIA-Net: Multimodal Interactive Attention Network for Multimodal Sentiment Analysis

MIA-Net: Multimodal Interactive Attention Network for Multimodal Sentiment Analysis

Summary: During multi-modal fusion, multiple modalities are first divided into main modalities and auxiliary modalities. By constructing an interactive attention module, information helpful to the main modality is extracted from the auxiliary modalities for fusion.

(It belongs to the feature-level attention-based fusion method)

Article information

Author: Shuzhen Li, Tong Zhang

Unit: South China University of Technology

Journal: IEEE Transactions on Affective Computing

题目：MIA-Net: Multi-Modal Interactive Attention Network for Multi-Modal Affective Analysis

Year: 2023

Research purposes

Explore how to generalize the bimodal model to trimodal or multimodal tasks under limited calculations and parameter conditions? And how to solve the different contributions of different modalities to sentiment analysis?

research content

A fusion model MIA-Net (Multi-modal Interactive Attention Network) that supports three modalities or multi-modalities is proposed to fuse the main modalities and auxiliary modalities and be used for sentiment analysis.

Research methods

This framework is composed of a single-modal feature extractor (Feature Extractor), N-1 MIA modules, regressors and classifiers.

Feature Extractor

For text data: First, a GPT-2 (a tokenizer) is used to mark each text description as a series of symbols. Then use the RoEBRTa model to extract text features.

For audio data: first use the VQ-Wav2Vec model to convert the variable-length audio representation of the audio into a discrete audio representation, and then still use the RoBERTa model to extract acoustic features from the discrete acoustic representation.

For video data: First, the RetinaFace model is used to detect and segment the facial area in each video frame. Video frames are then extracted from each video based on the number of keyframes. Finally, the Fabnet model is used to extract visual features from the extracted video frames.

Text feature dimension: 1024, acoustic feature dimension: 768, visual feature dimension: 256

MIA module

Symbol representation:

$M_j^n$means $M^n$ 's$j$ main mode eigenvectors,${\mathbf{A}}_i^n$means $A^{nth}$ ii_$i$ auxiliary modal feature vectors.

$\mathcal{K}({\mathbf{A}}_i^n,M_j^n)$ represents the general General Linear Kernel, used to calculate$S_{M^A{\_}^n}$(represents affinity matrix).

$\mathbf{P}({\mathbf{S}}_{M^A{\_}^n})$ represents the interactive attention weight, which is used to improve the main modal representation.

${\mathbf{P}}_{ji}({\mathbf{S}}_{M^A{\_}^n})$ represents the importance of the i-th auxiliary mode eigenvector to the j-th main mode eigenvector.

${\hat{M}}_{m\leftarrow a^n}^n$Represents updated primary mode features enhanced by auxiliary modes.

---

Each MIA module is composed of three sub-modules: extraction, conversion, and fusion. The function of the MIA module is to select important features from the auxiliary mode to improve the main mode features.

Extraction

First, a general linear kernel is used to capture the similarity between modal features , and then an affinity matrix $S_{M^A{\_}^n}$。

$$\begin{array}{r}k(x,y)=x^{T}W\end{array}$$
To satisfy this formula, W must be a positive semidefinite matrix, which is actually a real symmetric matrix.

Every real symmetric matrix can be decomposed into an orthogonal matrix. (According to the theorem $Q=P^T\Lambda P$) So the final calculation formula of the general linear kernel is:
$$\begin{array}{rl}\mathcal{K}({\mathbf{A}}_i^n,{\mathbf{M}}_j^n)& ={\left({\mathbf{A}}_i^n\right)}^{\mathrm{T}}{\mathbf{W}}^n{\mathbf{M}}_j^n\\ & ={\left(A_i^n\right)}^T({\mathbf{P}}^n{)}^{\mathrm{T}}\mathbf{\Lambda }{\mathbf{P}}^n{M}_j^n\\ & ={\left(P^A{\_}_i^n\right)}^{T}L(P^n{M}_j^n).\end{array}$$
After getting the affinity matrix (the similarity between the two modalities), the affinity matrix is ​​encoded into the interactive attention weight $\mathbf{P}({\mathbf{S}}_{M^A{\_}^n})$ , through$P\left(S_{M^A{\_}^n}\right)$ to improve the characteristics of the main mode.
$$P\left(S_{M^A{\_}^n}\right)=\left[P_1(S_{M^A{\_}^n}),\cdots ,P_j\left(S_{M^A{\_}^n}\right),\cdots ,P_{dm}(S_{M^A{\_}^n})\right]$$

$$P_j\left(S_{M^A{\_}^n}\right)=\left[\begin{array}{c}{P}_j\left(S_{M^A{\_}^n}\right)\\ ⋮\\ P_{ji}\left(S_{M^A{\_}^n}\right)\\ ⋮\\ P_{j{d}_{u^n}}(S_{M^A{\_}^A{\_}^n})\end{array}\right]=\left[\begin{array}{c}\frac{exp\left(K\right(A_1^n,M_j^n))}{{\sum }_k^{d_{a}n}exp(\mathcal{K}({\mathbf{A}}_k^n,{\mathbf{M}}_j^n))}\\ ⋮\\ \frac{exp\left(K\right(A_i^n,M_j^n))}{{\sum }_k^{d_{a}n}exp\left(K\right(A_k^n,M_j^n))}\\ ⋮\\ \frac{exp\left(K\right(A_{d_a^n}^n,M_j^n))}{{\sum }_k^{d_{a}n}exp\left(K\right(A_i^n,M_j^n))}\end{array}\right]$$

Transform

The interactive attention weight $\mathbf{P}({\mathbf{S}}_{{\mathbf{M}}^A{\_}^n})$ , and then in the Transform module, the interactive attention weight and the auxiliary modal feature are matrix multiplied to obtain the new main modal feature${\hat{M}}_{m\leftarrow a^n}^n$

Then this new main mode feature is improved through the self-gating mechanism GA (extracting more discriminative features and suppressing possible noise features) to obtain the improved main mode feature $M_{gated}^n$。
$$\begin{gathered} M_{gated}^n={\hat{M}}_{m\leftarrow a^n}^n\cdot \mathbf{G} \\ G=s(K_1\odot {\hat{M}}_{m\leftarrow a^n}^n+{\mathbf{b}}_1) \end{gathered}$$
$\odot$ represents convolution,$\sigma$ represents the logistic sigmoid activation function, and G represents the gating coefficient. $K_1$and $b_1$Represent convolution kernel and bias respectively.

Fusion

Finally, the improved main mode feature $M_{gated}^n$, with the initial main mode characteristics $M^n$ to fuse. Get the final fused modal features.
$$M^{n+1}=f_N\left(p\right(K_2\odot [{\mathbf{M}}_{gated}^n:M^n]+b_2))$$
$f_N(\cdot )$ represents the batch normalization function,$\left[:\right]$ indicates connection operation

---

By stacking multiple MIA modules, the main mode features enhanced by N auxiliary modes are finally obtained. Then through the two residual connections , the final main mode feature M ˇ \check{\boldsymbol{M}} after MIA-Net is obtained$\check{M}$.

residual connection

The first residual connection:
$$\begin{array}{rl}{M}_{res}& =M^1+M_{MIA}\\ & =M^1+f_{MIA}({\mathbf{M}}^1,{\mathbf{A}}^1,\cdots ,A^N)\end{array}$$
$M_{M.I.A.S\_\_}$Represents M 1 M^1 improved by multiple MIA modules$M^1$ main mode characteristics.

$M_{res}$Represents the residual main mode characteristics.

Second residual connection:
$$\check{M}=M_{res}+f_{}(M_{res})$$

Main mode feature M after residual connection $\stackrel{ˇSend}{M}$ it to Regressor or Classifier for final emotion recognition or emotion classification.

Regression

The regression model consists of a fully connected layer and a joint loss.

Determine:
$$\begin{gathered} \mathcal{L}=(1-c)L_{MAE}+\gamma {\mathcal{L}}_{KR} \\ \begin{array}{c}{L}_{MAE}=-\frac{1}{N}\sum _{n=1}^N|y^{(n)}-z^{(n)}|\\ z=i^{\mathrm{T}}\check{M}+{\mathbf{b}}_3\end{array} \end{gathered}$$
$L_{MAE}$represents the mean absolute error loss. $L_{KR}$Represents the kernel regularization loss ( $L_{KR}$The purpose is to find the affinity matrix $S_{M^A{\_}^n}$When, $W$ is a positive semidefinite matrix)

Classifition

The classification model consists of a fully connected layer, a Softmax function and a joint loss.

Determine:
$$\begin{array}{c}L=(1-c)L_{CE}+\gamma L_{KR}\\ \begin{array}{r}{L}_{CE}=-\frac{1}{N}\sum _{n=1}^N{y}_{n}log(P({\hat{y}}_n))\end{array}\\ \begin{array}{r}P(\hat{y})=softmax({\mathbf{\theta }}^T\check{M}+b_3).\end{array}\end{array}$$
$L_{CE}$stands for standard cross-entropy loss.

conclusion and discussion

1. A comparative experiment was conducted between the MIA-Net model and some SOTA models on the CMU-MOSI and CMU-MOSEI data sets, and MIA-NET achieved the best results.
2. MIA-Net is effective when processing data from three or more modalities. (Ablation Research)
3. MIA-Net can be generalized to new data sets, tasks and modalities. (general research)
4. The multi-modal interactive attention module can effectively achieve auxiliary fusion. (Verification of the proposed modal)
5. Study of hyperparameters.
6. Joint losses are valid. (Study of loss function)
7. Comparison with different fusion methods. (Prove the practicality of this study)

Code and datasets

The code is not public

Data set: CMU-MOSI(1.5G), CMU-MOSEI(25G), MELD(10G)

GPU configuration: not mentioned