——The original text was published on my WeChat public account "Big Data and Artificial Intelligence Lab" (BigdataAILab). Welcome to pay attention.
What is transfer learning?
Transfer Learning is a machine learning method that transfers knowledge from one domain (ie, the source domain) to another domain (ie, the target domain), so that the target domain can achieve better learning effects.
Usually, the amount of data in the source domain is sufficient, and the amount of data in the target domain is small, this scenario is very suitable for transfer learning. For example, we want to classify a task, but there is not enough data in this task (target domain), but There is also a large amount of relevant training data (source domain), but this training data is different from the test data feature distribution in the required classification task (for example, in speech emotion recognition, there is sufficient speech data in one language, but it needs to be classified The emotional data of the task is extremely lacking), in this case, if a suitable transfer learning method can be used, the classification and recognition results of tasks with insufficient samples can be greatly improved.
Why is transfer learning needed now?
Former Baidu chief scientist Andrew Ng, a professor at Stanford, once said: "Transfer learning will be the next driving force for the commercial success of machine learning after supervised learning."
At this 2016 NIPS conference, Wu Enda gave a technical development map of the future AI direction, which is still very objective:
Undoubtedly, supervised learning is currently the most mature, and it can be said that it has been successfully commercialized, and the next commercial technology will be Transfer Learning, which is also the AI technology that Andrew predicts is most likely to become commercial in the next five years.
In an interview, Andrew Ng also mentioned that transfer learning will be a very dynamic field. The reason why we are excited about transfer learning is that the great value of modern deep learning is for the problem that we have huge amounts of data. However, there are also many problem areas where we don't have enough data. such as speech recognition. In some languages, such as Mandarin, we have a lot of data, but those languages that are only spoken by a few people, our data is not large enough. So, in order to do speech recognition for the dialects spoken by a small minority of people in China, is it possible to transfer what is learned from learning Mandarin? Our technology can indeed do this, and we are doing it, but advances in this area give AI the ability to solve a much wider range of problems.
How is traditional machine learning different from transfer learning?
In the classic supervised learning scenario of machine learning, if we were to train a model for some task and domain A, we would assume that we were provided with labeled data for the same domain and task. As shown in the figure below, where our model A has the same domain and task in both training data and test data.
Even in multi-task learning, which is similar to transfer learning, multi-task learning is to jointly learn the target domain and the source domain, while transfer learning is mainly to solve the recognition task of the target domain by learning the source domain. The following figure shows the difference between traditional machine learning methods and transfer learning:
What is suitable for migration?
In some learning tasks, some features are unique to individuals, and these features cannot be transferred. While some features are contributed in all individuals, these can be transferred.
Negative transfer can sometimes result if the transfer is inappropriate, for example, when the tasks in the source and target domains are irrelevant.
Classification of Transfer Learning
According to Sinno Jialin Pan and Qiang Yang's article on TKDE 2010, transfer learning algorithms can be divided into four categories according to the knowledge representation to be transferred (ie "what to transfer"):
-
Instance-based transfer learning: A certain part of the data in the source domain can be reused by reweighting for learning in the target domain.
-
Feature-representation transfer learning: learn a good feature representation through the source domain, encode the knowledge in the form of features, and transfer it from the suorce domain to the target domain to improve the target domain task Effect.
-
Parameter-transfer learning: tasks in the target domain and source domian share the same model parameters or obey the same prior distribution.
-
Relational-knowledge transfer learning: Knowledge transfer between related domains, assuming that in the source domain and the target domain, the relationship between the data (data) is the same.
The first three types of transfer learning methods all require the assumption that the data is independent and identically distributed. At the same time, the four types of transfer learning methods all require the selected sorc doma to be related to the target domain.
The following table gives the transfer learning classification of transfer content:
Applications of Transfer Learning
For sentiment classification, image classification, named entity recognition, WiFi signal localization, automation design, Chinese to English translation, etc.
The value of transfer learning
-
Reuse existing knowledge domain data, and a lot of existing work will not be completely discarded;
-
There is no need to spend a lot of money to re-collect and calibrate huge new data sets, and it is possible that the data cannot be obtained at all;
-
For rapidly emerging new fields, it can be quickly migrated and applied, reflecting the advantage of timeliness.
Summarize
In conclusion, transfer learning will be the next exciting research direction, especially many applications require models that can transfer knowledge to new tasks and domains, which will become another important booster of artificial intelligence.
Welcome to follow my WeChat public account "Big Data and Artificial Intelligence Lab" (BigdataAILab) for more information
