论文笔记：Evolving Losses for Unsupervised Video Representation Learning

Evolving Losses for Unsupervised Video Representation Learning 论文笔记

Distillation

Knowledge Distillation from: zhihu

Distillate Knowledge from Teacher model Net-T to Student model Net-S.

目的：为了精简模型方便部署。

$L=\alpha L_{s o f t}+\beta L_{h a r d}$

$L_{s o f t}=-\sum_{j}^{N} p_{j}^{T} \log \left(q_{j}^{T}\right), \text { where } p_{l}^{T}=\frac{\exp \left(v_{i} / T\right)}{\sum_{k}^{N} \exp \left(v_{k} / T\right)}, q_{i}^{T}=\frac{\exp \left(z_{i} / T\right)}{\sum_{k}^{N} \exp \left(z_{k} / T\right)}$

$L_{h a r d}=-\sum_{j}^{N} c_{j} \log \left(q_{j}^{1}\right), \text { where } q_{i}^{1}=\frac{\exp \left(z_{i}\right)}{\sum_{j}^{N} \exp \left(z_{j}\right)}$

第一部分是从Teacher 模型中学习，第二部分是从ground truth 中学习

温度的高低改变的是Net-S训练过程中对负标签的关注程度: 温度较低时，对负标签的关注，尤其是那些显著低于平均值的负标签的关注较少；而温度较高时，负标签相关的值会相对增大，Net-S会相对多地关注到负标签。

Main idea: Multiple modalities to multiple tasks

Loss Function

$\mathcal{L}=\sum_{m} \sum_{t} \lambda_{m, t} \mathcal{L}_{m, t}+\sum_{d} \lambda_{d} \mathcal{L}_{d}$

where

$\lambda$ is weight

$\mathcal{L}_{m,t}$ is loss function of modality $m$ to task $t$

$\mathcal{L}_{d}$ is $L_2$ distance of a layer in the main network $M_i$ to another network $L_i$
$\mathcal{L}_{d}\left(L_{i}, M_{i}\right)=\left\|L_{i}-M_{i}\right\|_{2}$

Evolution Algorithm

Using GA to determine the $\lambda$

Each $λ_{m,t}$ or ${λ_d}$ is constrained to be in $[0,1]$

Unsupervised loss function

Zipf Distribution matching (ELo)

cluster centroids $\left\{c_{1}, c_{2}, \ldots c_{k}\right\} \text { where } c_{i} \in \mathcal{R}^{D}$

Naively assuming all clusters have the same variance, and let $2\sigma^2 = 1$

we can compute the probability of a feature vector $x ∈ R^D$ belonging to a cluster $c_i$ as
$p\left(x \mid c_{i}\right)=\frac{1}{\sqrt{2 \sigma^{2} \pi}} \exp \left(-\frac{\left(x-c_{i}\right)^{2}}{2 \sigma^{2}}\right)$
Bayes rules:
$\begin{aligned} p\left(c_{i} \mid x\right) &=\frac{p\left(c_{i}\right) p\left(x \mid c_{i}\right)}{\sum_{j}^{k} p\left(c_{j}\right) p\left(x \mid c_{j}\right)}=\frac{\exp -\frac{\left(x-c_{i}\right)^{2}}{2 \sigma^{2}}}{\sum_{j=1}^{k} \exp -\frac{\left(x-c_{j}\right)^{2}}{2 \sigma^{2}}} \\ &=\frac{\exp -\left(x-c_{i}\right)^{2}}{\sum_{j=1}^{k} \exp -\left(x-c_{j}\right)^{2}} \end{aligned}$
which is standard softmax function

given the above probability of each video belonging to each cluster, and the Zipf distribution, we compute the prior probability of each class as $q\left(c_{i}\right)=\frac{1 / i^{s}}{H_{k, s}}$ where $H$ is $k_{th}$ harmonic number and $s$ is real constant.

$p\left(c_{i}\right)=\frac{1}{N} \sum_{x \in V} p\left(c_{i} \mid x\right)$, the average over all videos in the set.

KL divergence :
$K L(p \| q)=\sum_{i=1}^{k} p\left(c_{i}\right) \log \left(\frac{p\left(c_{i}\right)}{q\left(c_{i}\right)}\right)$
This will be our fitness function.

it poses a prior constraint over the distribution of (learned) video representations in clusters to follow the Zipf distribution.

Loss Evolution

tournament selection and CMA-ES.