In the era of fragmented reading full of eyeballs, fewer and fewer people will pay attention to the exploration and thinking behind each paper.
In this column, you will quickly get the highlights and pain points of each selected paper, and keep up with the cutting-edge AI achievements.
Click " Read the original text " at the bottom of this article to join the community immediately to view more recent paper recommendations.
This is PaperDaily's 60th articleThe paper notes recommended in this issue are from PaperWeekly community user @jindongwang . This paper is the work of the University of Wollongong published in CVPR 2018. The author team applies sample weights to adversarial Partial Transfer.
If you are interested in the work of this paper, click on the original text at the bottom to view the original paper.
About the author: Wang Jindong is a doctoral student at the Institute of Computing Technology, Chinese Academy of Sciences. His research interests are transfer learning and machine learning.
■ 论文 | Importance Weighted Adversarial Nets for Partial Domain Adaptation
■ Link | https://www.paperweekly.site/papers/1805
■ Author | Jing Zhang / Zewei Ding / Wanqing Li / Philip Ogunbona
background
Most of the transfer learning problem scenarios we have encountered so far are: the feature space of the source domain and the target domain is consistent with the category space , but the distribution of data is inconsistent, how to transfer. That is to say, if the source domain and the target domain are several categories, they are all categories.
But this situation is obviously very restrictive: in real applications, we often do not know the category of the target domain, let alone whether it is exactly the same as the category of the source domain. This greatly limits its application.
The goal of transfer learning is to use a large amount of annotated source domain data to model the target domain data. Does the problem seem more general if we assume that the existing source domain data is rich, which itself contains the categories of the target domain?
Formally speaking, we use Y s and Yt to represent the category space of the source and target domains, respectively, then the partial transfer learning is Y ⊃ Yt . This situation is more challenging than transfer learning in the usual sense.
Thesis motivation
The author's motivation is very intuitive: if there are more categories in the source domain than in the target domain, it is impossible to transfer directly, then we first select the samples in the source domain that are closest to the category of the target domain , give them high weights, and then Migrate again.
Based on this idea, the authors design a two-stage deep adversarial network . We all know that the adversarial network is mainly composed of a feature extractor and a domain classifier. The feature extractor is used to extract common features of the samples, and the domain classifier is used to determine whether two domains are similar.
Existing work usually employs a common feature extractor for source and target domains. However, in partial transfer learning, the categories of the source domain and target domain are often different, so the author proposes to use different feature extractors for them respectively. Another benefit of doing this is that different feature extractors can learn their own representational features for their respective domains.
During learning, the feature extractor of the fixed source domain is unchanged, and only the features of the target domain are learned, which is more in line with the theory of the original GAN.
method
作者提出了一个深度加权对抗网络（Importance Weighted Adversarial Nets）。这个网络的示意图如下图所示。
网络的主要部分为：分别作用于源域和目标域的两个特征提取器（分别叫做 Fs 和 Ft)，以及两个领域分类器（分别叫做 D 和 D0）。
第一个领域分类器用来筛选出源域中与目标域相似的那部分样本（或者源域中与目标领域共享的那部分类别），第二个领域分类器进行正常的 Domain Adaptation。
相应地，方法主要分为两个部分：1）筛选出源域中与目标域属于相同类别的样本，2）综合学习。
样本筛选
此部分是该论文的核心创新点。主要思路是，由 Fs 和 Ft 产生的源域和目标域特征 Zs 和 Zt，通过一个领域分类器 D，判别样本是来自于源域还是目标域。
这个 D 是一个二类分类器，如果 D=1，表示样本来自源域；否则，样本则来自目标域。那么，如何才能知道样本是否属于源域和目标域的公共类别，而不是特异于源域的类别（作者叫它们 outlier）？
如果 D(z)≈1，那么就表示 z 这个样本是来自于源域。更进一步分析，为什么这部分样本只是来自源域？那是因为这部分样本是源域独有的。否则的话，它就同样来自目标域了。
从另一个角度说，如果 D(z)≈0，则表示这部分样本来自目标域。同时，也表示它可能来自源域中与目标领域共享的类别。因为两部分的类别是共享的。
这个简单而直观的道理指导着我们设计不同的权重。我们的目标是，对于筛选出的那部分与目标域属于相同类别的源域样本，给它们赋予大权重；另一部分源域特有的样本，权重调小。该权重可以被表示为：
这里的 D⋆(z) 表示的是领域分类器 D 的最优值，它可以通过求导得出：
从这个式子可以看出，如果 D⋆(z) 值较大，则表示样本更可能是 outlier，那么权重值就会变小；反之，如果 D⋆(z) 值较小，则表示样本更可能是源域和目标域共同部分，那么权重值就会变大。这个方法很好地对源域和目标域中的共同类别的样本完成了筛选工作。
作者还对源域部分的权重进行了归一化，以更加明确样本的从属关系。加入权重以后，优化目标变成了：
下一步工作是处理领域分类器 D0。D0 也可以以同样的求导方式得到。
综合学习
综合学习之前，作者还加了一个熵最小化项用于对目标域的样本属性进行约束，这也是比较常见的做法。熵最小化可以被表示为：
其中的 H(⋅) 就是熵的表达形式。C(⋅) 是分类器。现在，总的学习目标就是：
实验
部分迁移学习实验主要在传统的 Office-Caltech 以及 Office-31 上做。不过实验任务与之前的迁移学习有所不同：源域的类别比目标域多。
作者和一些最新的迁移学习方法进行了对比，表明了所提方法的优势。
作者还特别做了一个实验：当目标域的类别个数逐渐变多时，精度如何变化？结论是，随着目标域类别个数的减少，精度逐渐增加。这表明知识在进行迁移时，源域知识越多，通过筛选，对目标越有效。
具体实验设定、结果、以及其他的可视化结果可以参照原文。
总结
本文核心创新点是，从任务出发，直观地构造出两阶段式对抗网络，对源域中与目标域共享的类别样本进行有效筛选。
另一个与已有工作不同的地方是，作者分别对源域和目标域采用了不同的特征提取器。其实这一点与作者发表在 CVPR-17 的 JGSA [1] 方法很像。可以参考学习。
参考文献
[1] Zhang J, Li W, Ogunbona P. Joint geometrical and statistical alignment for visual domain adaptation[J]. arXiv preprint arXiv:1705.05498, 2017.
[2] 与本文相关的另一篇部分迁移学习工作：
https://zhuanlan.zhihu.com/p/32161502
This article is selected and recommended by PaperWeekly, an AI academic community. The community currently covers research directions such as natural language processing, computer vision, artificial intelligence, machine learning, data mining and information retrieval. Click " Read the original text " to join the community immediately!
Click on the title to view more paper interpretations:
Brain-opening machine vision multi-domain learning model structure
Novel RNN: Independence of neurons in layers to improve long-term memory
Confidence-based knowledge graph representation learning framework
#author recruitment #
Let your text be seen by many, many people, if you like us, it is better to join us
I am an Easter egg
Unlock new features: Popular Job Recommendations!
The PaperWeekly applet has been upgraded
arXiv today √ Guess what you like √ Popular jobs √
Finding a full-time internship is no problem
Unlock method
1. Identify the QR code below to open the applet
2. Log in with the PaperWeekly community account
3. All functions can be unlocked after logging in
job posting
Please add a small assistant WeChat ( pwbot02 ) for consultation
Long press to identify the QR code and use the applet
*Click to read the original text to register
About PaperWeekly
PaperWeekly is an academic platform for recommending, interpreting, discussing, and reporting AI cutting-edge papers. If you are researching or engaged in the field of AI, you are welcome to click "Communication Group" in the background of the official account, and the little assistant will bring you into the PaperWeekly communication group.
▽ Click | Read the original article | View the original paper