[Gait recognition] MvGGAN based on multi-view gait generation confrontation network algorithm learning "Multi-View Gait Image Generation for Cross-View Gait Recognition"

1. Papers & Code Sources

"Multi-View Gait Image Generation for Cross-View Gait Recognition"
paper address: https://ieeexplore.ieee.org/document/9349211/Code
download address: The author did not provide the source code?

2. Highlights of the paper

In reality, it is difficult to capture gait data in various perspectives. Cross-view gait recognition is to identify gait data in unknown views based on known gait data.
There are two main methods for cross-view gait recognition: view transition model (VTM) and view invariant feature extraction.
The basis of View Transformation Model is Singular Value Decomposition (SVD). For related content, please refer to the Knowledge Supplement.

1. Gait images from different viewpoints are automatically generated by a single generator. Different subjects and different data sets can be used to jointly extract gait features, and a single generator can learn richer gait information from different subjects and data sets, increasing the diversity of fake samples.
2. The domain alignment based on projected maximum mean difference (MMD) is performed on the real and fake gait samples to reduce the impact of domain transfer generated by the fake sample generation process.
3. Add fake samples generated from the same or different datasets to extend the dataset and improve the generalization ability of the gait classification model.
   

3. Multi-view Generative Adversarial Network

3.1 Network generation process

When the generator accepts an image as input, the generator can transfer the input image into the domain defined by the target image, realizing the domain transfer task. Given paired samples, the domain transfer task can be implemented by supervised learning (such as the pix2pix method).
(Question 1) But the gait image is discrete, and it is difficult to achieve unbiased alignment under different viewing angles, so unsupervised GANs based on non-pairing are more suitable.
At a certain viewing angle, $\mathbf{U}\to V$ needs to build$G_A,D_A$；$\mathbf{V}\to U$ need to build$G_B,D_B$, assuming $k$ views, you need to train$k(k-1)$ times.
It is not clear here
(Question 2)When the amount of data is small, it is difficult to train multiple GANs, which is prone to overfitting, so the author proposes to train a conditional GAN ​​conditioned on the target view label to control the generation of gait images The single generator G of the view of , realizes multi-view mapping, and the specific working principle of MvGGAN is as follows:
(leads to this method)
{ Given an input image sequence X = { x 1 , x 2 , . . . , x N } , the target view label c Obtain the output image sequence Y = { y 1 , y 2 , . . . , y N } from G ( X , v ) → Y. The discriminator predicts the true or false of the output image perspective and identity \begin{cases} given the input image Sequence X=\{x_1, x_2, ..., x_N\}, target view angle label c\\ The output image sequence Y=\{y_1, y_2, ..., y_N is obtained from G(X, v)\to Y \}\\ The discriminator predicts the true and false of the output image perspective and identity\\ \end{cases}⎩⎪⎨⎪⎧​Given an input image sequence X={ x1​,x2​,...,xN​} , target viewpoint label cby G ( X ,v)→Y gets the output image sequence Y={ y1​,y2​,...,yN​}The discriminator predicts whether the output image perspective and identity are true or false
$MvGGANOT$ S$StarGAN$ , the difference is:

StarGAN	MvGGAN
Convert images from one domain to another without preserving identity information	To generate more multi-domain and diverse fake samples, identity information needs to be retained as training data

3.2 Network structure

The angles in the above diagram are for illustration only. The generator has 2 initial inputs: the original gait image ( $0°$ ), target gait angle ($90°$ ), these two input quantities will output a set angle of$90°$ fake image (Fake image); there are 2 process inputs: the original gait angle ($0°$ ), fake image ($90°$ ), these two input quantities will output a set angle of$0°$ Reconstructed image(false in false).
All angles are$The\; real\; and\; fake\; images\; at\; 90°$ are input into the discriminator (Discriminator) and the identification discriminator (Identification discriminator), and the discrimination loss, perspective classification loss and discrimination loss are calculated respectively$.$

3.3 Loss function

3.3.1 Discriminator Loss (Discriminator Loss)

We need to judge the true and false probability of the image output by the generator, the discriminator tries to distinguish the fake image, and the generator tries to "deceive" the discriminator, so the adversarial loss function on the discriminator is as follows: L adv = EX [
$$L_{adv}={\mathbb{E}}_X[\log D(X)]+E_{X,v_t}[\log (1-D(G(X,v_t)))]\left(1\right)$$
Among them,$G(X,v_t)$ represents the input image sequence$X$ is based on the viewing angle$v_t$Build in the generator.

3.3.2 View Classification Loss

In order to generate a specific viewing angle $v_t$For the gait image below, the generated gait samples need to be classified into the corresponding viewing angle $v_t$middle. Suppose there is $k$ views, in the discriminatorDDAdd one withkk$on\; D$A classifier with$k\; output\; nodes\; realizes\; the\; classification\; of\; views.$The view classification network is represented as$D_{view}$, first of all need to $D_{view}$For optimization, the real gait image is used as input, and the corresponding real viewing angle label is used as output. The loss function of the optimization process is as follows:
$$L_{view}^{real}={\mathbb{E}}_{X,v_t}[\log D_{view}(v_t|X)]$$
其中，$D_{view}(v_t|X)$ represents the input image$X$ belongs to view angle$v_t$Likelihood (probability value) of . Is this loss negative?
This network is also used to transfer $The\; fake\; image\; generated\; by\; G$ is correctly classified to the target view$v_t$in, so $G$需要最小化以下损失函数：
$$\begin{gathered} L_{view}^{fake}={\mathbb{E}}_{X,v_t}[-\log D_{view}(v_t|Y)] \\ =E_{X,v_t}[-\log D_{view}(v_t|G(X,v_t))](2) \end{gathered}$$

3.3.3 Cycle Consistency Loss

The purpose of the measurement of the first two losses is to generate a gait image that looks more realistic and belongs to the target perspective, but in addition to changes in perspective, there may also be other changes (such as: clothing, carrying objects, etc.), which will make Build results become unmanageable. In order to preserve other gait information of the input gait samples while only changing a single element of the viewing angle , a cycle consistency loss function is introduced here to preserve the gait information beyond the viewing angle change as much as possible .
$$L_{rec}={\mathbb{E}}_{X,v,v_t}[||X-G(G(X,v_t),v)|{|}_1](3)$$
Among them, the generator$G$ is used twice, once as$G(X,v_t)$ to generate a viewing angle$v_t$The following (fake) gait image, another time $G(G(X,v_t),v)$ reconstruction viewvvThe original input sequence under$v\; .$

3.3.4 Identification Loss

The reconstruction loss attempts to preserve the structural information and perspective information of each gait image in the input sequence, but the construction of this loss processes each gait image separately, and does not consider the relationship between different image frames. The reality is that gait is dynamic information composed of a series of frames, and calculating the loss frame by frame may lead to subject recognition errors. Therefore, it is necessary to design a recognition discriminator $D_{id}$. The recognition discriminator takes the original gait data sequence and the gait image sequence output by the generator as a training data pair, and generates the probability of whether the data pair comes from the same person, if it comes from the same subject, the output is 1 $1$ , otherwise output$0$ , the loss function is as follows:
$$L_{id}={\mathbb{E}}_X[\log D_{id}(X)]+v{\mathbb{E}}_{X,v_t}[\log (1-D_{id}(G(X,v_t)))]\left(4\right)$$
In order to make full use of the dynamic and static gait information, the article designsa LBThe recognition discriminator structure of the$LB$$network\; .$

3.3.5 Full Objective Loss Function (Full Objective)

Combining the above four loss functions, the final target loss function can be obtained:
$$L(G,D)=L_{adv}+l_{view}{L}_{view}^{real}+l_{view}{L}_{view}^{fake}+l_{rec}{L}_{rec}+l_{id}{L}_{id}\left(5\right)$$
Among them,$G$ tries to minimize this objective loss, while$D$ tries to maximize it,$\lambda$ is a hyperparameter (the framework parameter in the machine learning model), which is used to control the importance (weight ratio) of different loss functions in the optimization process.

3.4 Analysis of other details

3.4.1 Multi-view generation across datasets

By controlling the viewing angle label vector $v_t$To control the target angle of view of the generated gait image, if $v_t$by including $k_1$Dataset $A$ composition, then it can be in$k_1$Generate gait samples from different viewpoints. Additionally, $k_2$BB $B$ and$v_t$Combined, then $MvGGAN$ can be in$k_1+k_2$Generate gait samples from different viewpoints. In this way, dataset $A$ and dataset$B$ will include$k_1+k_2$From this perspective, both datasets are extended to include more samples.
In order to achieve the goal described in the above text, the author based on $The\; StarGAN$ method defines a unified label vector$\tilde{v}$：
$$\tilde{v}=[v_1,...,v_n,m]\left(6\right)$$
Among them,$v_i$Represents the second $i$ vector labels,$m$ is a$n-$ dimensional one-hot vector, if$m$ 's$i$ elements are$1$ , indicating that the target view angle of the generated gait image is the$i$ .

3.4.2 Multi-dataset training

In gait recognition, in addition to the difference in viewing angle, clothing and carrying objects are also two very important influencing factors, so the author assumes that the gait samples of the same clothing condition or the same carrying object are the same "domain", as mentioned above $k$ means that there are kkin the data set$K$ view labels, the whole is a$A\; k-$ dimensional one-hot vector. Now you need to combine the tags representing the attributes of clothing or carrying items with$k$ is concatenated to form a unified label vector$u$，有$u=[k,d]$ where$d$ represents a one-hot vector of clothing or carrying conditions.
In real life, the changes of clothing and carrying objects are diverse, which should correspond to different domains, but it is difficult to achieve in the experimental process, so the author only focuses on the two typical clothing (summer clothes and coats) involved in the CASIA-B data set. ), and two typical carrying items (backpack and shoulder bag) for experiments.
During the experiment, consider the "nm", "bg" and "cl" types as three domains.

3.4.3 Dataset adaptability between real and fake gait images

Since the author uses a single generator to generate fake gait images under different walking conditions (even different data sets), in theory, there should be a distribution difference between the real and fake gait images. Therefore, the author uses the t-SNE method to The distribution of true and false gait images is visualized, as shown in the figure below:

labels 0-4 are false gait images, and labels 5-9 are true gait images. The fake gait images in the left image cover 22 different viewpoints from CASIA-B and OU-MVLP datasets, and the true gait images only cover 11 different viewpoints from CASIA-B; the right image is only shown at 90 It is not difficult to see that there are obvious differences in the distribution of the true and false gait images under the viewing angle of °.
In order to solve the above problems, the author introduces a domain alignment method and learns a feature mapping function $F(\cdot )$ to reduce the distribution difference between real and fake gait images. The true gait feature set isD r = { xri , yri } i = 1 n \mathcal D_r = \{\mathbf x_{r_i},y_{r_i}\}_{i=1}^nDr​={ xri​​,yri​​}i=1n​, the false gait feature set is $D_f=\{x_{f_j},y_{f_j}{\}}_{d=n+1}^{n+m}$, and the label space ${\mathcal{Y}}_r=Y_f$. The distribution difference between different data sets is dominated by the marginal distribution (P), and the sample distribution difference in the same data set is dominated by the conditional distribution (Q). Considering these two factors, the author proposes: DF ( D r , D f ) =
$$D_F({\mathcal{D}}_r,D_f)=(1-h)D_F(P_r,P_f)+the\sum _{c=1}^C{D}_F^{(c)}(Q_r,Q_f)\left(7\right)$$
Among them,$\eta$ is an adaptive factor for balancing marginal distribution and conditional distribution;$c\in 1,...,C$ represents the ID of the subject of the gait image,$D_F(P_r,P_f)$ is the marginal distribution arrangement,$D_F^{(c)}(Q_r,Q_f)$ is ID isccConditional distribution permutation of$c\; .$
The distribution difference between real and fake gait images canbe calculated byMaximum Mean Discrepancy (MMD)
$$\begin{gathered} D_F({\mathcal{D}}_r,D_f)=(1-h)||E\left[F\right(x_r)]-\mathbb{E}[F({\mathbf{x}}_f)]|{|}_{{\mathcal{H}}_K}^2 \\ +the\sum _{c=1}^C||\mathbb{E}[F({\mathbf{x}}_r^{(c)})]-\mathbb{E}[F({\mathbf{x}}_f^{\left(c\right)})]|{|}_{H_K}^2\left(8\right) \end{gathered}$$
Among them,$E[\cdot ]$ is the mean value of embedded features;$H_K$Represents the Reproducing Kernel Hilbert Space (RKHS) , based on the Representer Theorem , the formula $\left(8\right)$ can be expressed as
$$D_F({\mathcal{D}}_r,D_f)=\text{tr}({\mathbf{F}}^T\mathbf{M}\mathbf{F})(9)$$
其中，$\mathcal{F}\in {\mathbb{R}}^{(n+m)\times d}$ is the feature matrix formed by connecting the feature vectors of each sample, and each row in the feature matrix represents a true and false samplex \mathbf xThe eigenvector F of$x$$F(\mathbf{x})$，$d$ is the eigenvector$Dimensions\; of\; F\left(x\right)$ .
MMD matrix$\mathbf{M}=(1-h)M_0+the{\sum }_{c=1}^C{M}_c\left(10\right)$ is used to calculate the matrixF \mathbf FDistribution divergence of gait features in$F$$Let\; (10)$ be the inverse function:
$$({\mathbf{M}}_0{)}_{ij}=\{\begin{array}{ll}\frac{1}{n^2},& {\mathbf{x}}_i,x_j\in {\mathcal{D}}_r\\ \frac{1}{m^2},& x_i,x_j\in D_j\\ -\frac{1}{mn},& \text{others}\end{array}(11)$$
$$({\mathbf{M}}_c{)}_{ij}=\{\begin{array}{ll}\frac{1}{n_c^2},& {\mathbf{x}}_i,{\mathbf{x}}_j\in {\mathcal{D}}_r^{(c)}\\ \frac{1}{m_c^2},& x_i,x_j\in D_j^{\left(c\right)}\\ -\frac{1}{m_c{n}_c},& x_i\in D_r^{\left(c\right)},x_j\in D_f^{\left(c\right)}or{\mathbf{x}}_i\in D_f^{\left(c\right)},x_j\in D_r^{\left(c\right)}\\ 0,& \text{others}\end{array}(12)$$
where$n_c$，$m_c$Respectively represent the subjects ccThe number of true and false samples of$c\; .$In the experiment, for the fake gait samples generated by the same dataset, set$the=0.8$ , fake gait samples generated across datasets, set$the=0.2$。

In order to reduce the distribution difference between true and false samples, the mapping function $F$ needs to minimize formula (7)during trainingThe objective function in$($$7$$)\; .$In this paper, the mapping function$F$ is built as amulti-layer fully connected network-the input gait features can be extracted from any layer of the gait classification network. The author sets the number of nodes in the middle layer to 1024. The reason is that the purpose of the domain alignment network is to reduce the distribution difference between the real and fake gait images, that is to say, we only require the overall distribution of the real and fake images to be consistent. The merged features of the image are not constrained, and the experiment found that the number of nodes in the middle layer is 1024 enough to achieve distribution alignment. A higher number of nodes increases computational cost and risk of overfitting, and does not lead to improved performance. The aligned gait features are input to the next layer of the gait classification network for training or testing.

3.5 Experiment implementation

3.5.1 Network structure

The generator refers to the StarGAN model, the discriminator uses the PatchGAN model, and the discriminant loss uses the LB network in the article [A comprehensive study on cross-view gait based human identification with deep CNN] .

3.5.2 Training process

In the loss function ${\mathcal{L}}_{adv}$The Wasserstein GAN (WGAN) is used to stabilize the adversarial training process.
Hyperparameter setting $l_{view}=1$ ,$l_{rec}=10$ ,$l_{id}=5$ , use Adam to optimize the generator and discriminator. The batch size is$16$ , the discriminator is updated every five times, and the generator is updated once. The initial value of the learning rate is$0.0001$ , the epoch value is$50$ , the learning rate will decay linearly to$0$。

4. Experimental results

5. Summary

In this paper, based on the multi-view gait generation confrontation network (MvGGAN) , the author generates fake gait samples from different views under different gait datasets. MvGGAN consists of a generator, which generates fake gait samples under several walking conditions, and a discriminator, which implements adversarial training and preserves the person's identity information. By adding generated fake gait samples to the original gait dataset, and performing domain alignment operations between real samples and fake samples, the performance of deep learning-based gait classification networks can be significantly improved.

This paper demonstrates that it is feasible to improve cross-view gait recognition performance by adding fake samples to the original gait dataset, and also demonstrates that it is possible to generate fake samples for another dataset under different viewpoints or other walking conditions. Samples are also available. Gait image generation across datasets is important, as it enables gait classification networks to learn gait information in as many walking conditions as possible, and improves gait recognition in real-world scenarios (with a large number of unregistered gait samples) performance.

0. Knowledge Supplement

0.1 Singular value decomposition (SVD)

Singular Value Decomposition (Singular Value Decomposition) is an algorithm widely used in the field of machine learning. It can be used not only for feature decomposition in dimensionality reduction algorithms, but also for recommendation systems and natural language processing. It is the realization of many machine learning algorithms. Foundation.

Eigenvalues ​​and eigenvectors
$$Aw=\lambda w$$
in which,$A$ is$n\times n-$ dimensional matrix,$w$ is a$n-$ dimensional vector (eigenvector),$\lambda$ is the eigenvectorwwThe corresponding eigenvalues ​​of$w\; .$
After the eigenvalues ​​and eigenvectors are obtained, the matrix$A$ performs eigendecomposition:
$$A=W\Sigma W^{-1}$$
out of$W$ by$n$ eigenvectors$w_1,w_2,...,w_n$composed of $n\times n-$ dimensional matrix,$\Sigma$ is defined by the feature vector$w_i$The corresponding eigenvalue $l_i$Diagonal matrix formed on the main diagonal.
If $A$ is not$n\times n-$ dimensional square matrix, but$m\times n-$ dimensional matrix, then a singular value decomposition is required.

SVD
$$A=U\Sigma V^T$$
其中，$U\in {\mathbf{R}}^{m\times m}$，$S\in {\mathbf{R}}^{m\times n}$，$V\in R^{n\times n}$，$\Sigma$ is 0except for the elements on the main diagonal$0$ , the elements on the main diagonal are the singular values,$UV$ and$V$ is a positive definite matrix (unitary matrix), satisfying$U^{T}U=I,V^{T}V=I.$ _
Transpose$A^T$ and Matrix$A$ does multiplication to get a$n\times For\; an\; n-$ dimensional square matrix, perform eigendecomposition on this square matrix to obtain eigenvalues ​​and eigenvectors that satisfy the following formula:
$$(A^{T}A)v_i=l{v}_i$$
Among them, by the feature vector $v_i$The matrix VV composed of$V$ is the VVin the singular value decomposition formula$V$；同理，
$$(A{A}^T)u_i=\lambda u_i$$
Generally the matrix $U$ becomes a left singular vector.
Next solve for$\Sigma$矩阵。
$$\begin{gathered} A=U\Sigma V^T\to AV=U\Sigma V^{T}V=TheS \\ \to A{v}_i=u_i{p}_i \\ \to p_i=\frac{A{v}_i}{u_i} \end{gathered}$$
After the above derivation, each singular value σ i \sigma_i can be obtained$p_i$, so as to obtain $\Sigma$ matrix.
In addition, there is a solution to$The\; method\; of\; \Sigma$ matrix is ​​from the previous definition$(A^{The\; characteristic\; matrix\; of\; T}A)$is$V$，$(A{A}^T)$feature matrix is$U$ , that's the point to start with. to$V$矩阵为例：
$$\begin{gathered} A=U\Sigma V^T \\ \to A^T=(U\Sigma V^T{)}^T=V\Sigma U^T \\ \to A^{T}A=(V\Sigma U^T)(U\Sigma V^T)=V{S}^{2V}{\_}^{} \end{gathered}$$
The derivation process above$${}^T$$$A^{}$The matrix composed of the eigenvectors of${}^T$$A$$V$ , and the eigenvalue matrix is ​​the square of the singular value matrix, that is,
$$p_i={\sqrt{l}}_i$$

0.2 Generative Adversarial Network (GAN)

0.2.1 Basic concepts

Generate Adversarial Networks (Generate Adversarial Networks) is a generative model, mainly composed of two parts : the discriminator and the generator .
Discriminator (discriminator): mathematically expressed as $y=f\left(x\right)$ or conditional probability distribution$p(y|x)$ . When the input training set picture is$When\; x$ , the discriminator outputs the classification label$y$ , learn the input image$x$ with output label$The\; mapping\; relationship\; of\; y$ .
Generator:mathematically expressed as a probability distribution$p\left(x\right)$ . A generator without constraints is an unsupervised model that, given a simple prior distribution$\pi \left(z\right)$ (usually a Gaussian distribution) is mapped to the pixel probability distribution$p\left(x\right)$ , that is, output a sheet that obeys$Images\; from\; the\; p\left(x\right)$ distribution with features from the training set.

0.2.2 Other generative networks

1. AutoRegressive model (AR)
2. Variational AutoEncoder (VAE)
3. Flow-based generator model

0.2.3 Loss function

$$L=\min \max V(D,G)={\mathbf{E}}_{x\sim p_{data}(x)}[logD(x)]+{\mathbf{E}}_{Z\sim p_z(x)}[log(1-D\left(z\right)\left)\right]$$
Look at
$V(D,G)$ is the difference between generated (fake) samples and real samples;
$\max V(D,G)$ The generator is fixed, the cross-entropy loss is maximized, and the discriminator is updated;
$\min \max V(D,G)$ The generator minimizes this loss while maximizing the cross-entropy loss.

0.2.4 Training process

First train the discriminator,
put the training set data (training set) on the real label, and the fake image generated by the generator (fake image) on the fake label, and the two form a batch and send it to the discriminator for training. When calculating the loss, the discriminator's judgment on the training set tends to be true, and the fake image tends to be false. This process only updates the discriminator parameters.
Retrain the generator
to apply Gaussian distribution noise $z$ (random noise) is sent to the generator, and the fake image generated by the generator is tagged with a true label and sent to the discriminator. When calculating the loss, the discriminator will make the fake image discrimination approach to true. This process only updates the parameters of the generator.

0.3 t-SNE

The main use of t-SNE is to visualize and explore high-dimensional data. It was developed and published by Laurens van der Maatens and Geoffrey Hinton in JMLR Volume IX (2008). The main goal of t-SNE is to transform a multi-dimensional dataset into a low-dimensional one. Compared to other dimensionality reduction algorithms, t-SNE works best for data visualization. If we apply t-SNE to n-dimensional data, it will intelligently map n-dimensional data to 3d or even 2d data, and the relative similarity of the original data is very good. Like PCA, t-SNE is not a linear dimensionality reduction technique, it follows nonlinearity, which is the main reason why it can capture the complex manifold structure of high-dimensional data.

Reference blog:

A simple understanding of the loss function loss of the generated confrontation network GAN

Comparative Study of GAN's Loss (1) - Understanding of Traditional GAN's Loss 1

singular value decomposition

The principle of singular value decomposition (SVD) and its application in dimensionality reduction

What are the similarities and differences between singular values ​​and eigenvalues ​​of a matrix

t-SNE: the best dimensionality reduction algorithm for visualization