AI Experience Hall is online! Integrate the industry-leading NLP scene deep transfer learning framework EasyTransfer

Introduction: In October 2020, Alibaba Cloud officially open-sourced the deep transfer learning framework EasyTransfer, which is the industry's first deep transfer learning framework for NLP scenarios. At present, the AI ​​experience hall that integrates this ability has been officially launched, and it is free to experience: https://workbench.data.aliyun.com/experience.htm#/paiAbilityVenue/

In October 2020, Alibaba Cloud officially open-sourced the deep transfer learning framework EasyTransfer, which is the industry's first deep transfer learning framework for NLP scenarios. Open source link: https://github.com/alibaba/EasyTransfer
Currently, the AI experience hall that integrates this ability has been officially launched and is free to experience: https://workbench.data.aliyun.com/experience.htm#/paiAbilityVenue/

image.png

Users can easily click, and the free experience includes NLP (article classification, content review), image classification, voice recognition, video classification, and automatic generation of video highlights!
Today, I will take you to the back of the AI ​​Experience Hall and unveil the mystery of NLP's leading technology.

The EasyTransfer framework is developed by the Alibaba Cloud Machine Learning PAI team, which makes the model pre-training and migration learning development and deployment of natural language processing scenarios easier and more efficient.
Deep transfer learning for natural language processing scenarios has a huge demand in real scenarios. Because a large number of new fields are constantly emerging, traditional machine learning needs to accumulate a large amount of training data for each field, which will consume a lot of manpower and annotation. Material resources. The deep transfer learning technology can transfer the knowledge learned in the source domain to the task of the new domain, thereby greatly reducing the resources of annotation.

Although there are many requirements for deep transfer learning for natural language scenarios, the open source community does not yet have a complete framework, and it is a huge challenge to build a simple, easy-to-use and high-performance framework.

First of all, pre-training model plus knowledge transfer is now the mainstream NLP application mode. Generally, the larger the size of the pre-training model, the more effective the knowledge representation learned. However, the super-large model brings great challenges to the distributed architecture of the framework. How to provide a high-performance distributed architecture to effectively support super-large-scale model training.

Secondly, the diversity of user application scenarios is very high, and a single migration learning algorithm cannot be applied. How to provide a complete migration learning tool to improve the effect of downstream scenarios.

Third, it usually takes a long link from algorithm development to business landing. How to provide a simple and easy-to-use one-stop service from model training to deployment.
Facing these three challenges, the PAI team launched EasyTransfer, a simple, easy-to-use and high-performance transfer learning framework. The framework supports mainstream migration learning algorithms, supports automatic mixed precision, compilation optimization, and efficient distributed data/model parallel strategies, and is suitable for industrial-grade distributed application scenarios.

It is worth mentioning that, with mixed precision, compilation optimization, and distributed strategy, the ALBERT model supported by EasyTransfer is more than 4 times faster than the community version of ALBERT in the calculation speed of distributed training.

At the same time, after more than 10 BUs and more than 20 business scenarios within Ali, it provides a variety of conveniences for NLP and migration learning users, including the industry-leading high-performance pre-training tool chain and pre-training ModelZoo, and the rich and easy-to-use AppZoo , Efficient migration learning algorithms, and full compatibility with Alibaba’s PAI ecological products, providing users with a one-stop service from model training to deployment.

Lin Wei, head of the Alibaba Cloud machine learning PAI team, said: This open source EasyTransfer code hopes to empower more users with Ali’s capabilities, lower the threshold of NLP pre-training and knowledge transfer, and also work in-depth with more partners. Create a simple, easy-to-use, high-performance NLP and migration learning tool.

image.png

Overview of the framework of the EasyTransfer tool The
overall framework of EasyTransfer is shown in the figure below, which simplifies the algorithm development difficulty of deep transfer learning as much as possible in design. The framework abstracts commonly used IO, layers, losses, optimizers, models. Users can develop models based on these interfaces, or they can directly access the pre-trained model library ModelZoo for rapid modeling. The framework supports five transfer learning (TL) paradigms, model finetuning, feature-based TL, instance-based TL, model-based TL and meta learning. At the same time, the framework integrates AppZoo, supports mainstream NLP applications, and facilitates users to build commonly used NLP algorithm applications. Finally, the framework is seamlessly compatible with PAI ecological products, bringing users a one-stop experience from training to deployment.

image.png

The industry-leading high-performance pre-training tool chain and pre-training ModelZoo
EasyTransfer framework support industrial-grade distributed application scenarios, improve the distributed optimizer, and cooperate with automatic mixing precision, compilation optimization, and efficient distributed data/model parallel strategies. It is 4 times faster than the community version of multi-machine multi-card distributed training in computing speed. Based on this high-performance distributed base, the framework launches a complete pre-training tool chain to facilitate users to pre-train language models such as BERT and ALBERT. It is worth mentioning that the model based on the pre-training tool has achieved good results on multiple public lists, such as the first place in the multi-round dialogue list QuAC (October 2019) and the Chinese CLUE list. First place (December 2019), and second place on the English SuperGLUE list. At the same time, EasyTransfer integrates the pre-training model ModelZoo, Continual Pretrain and Finetune that support mainstream models such as BERT, ALBERT, XLNet, and also integrates the high-quality pre-training model trained on the PAI platform and the self-developed e-commerce scene multi-modal model FashionBERT. .

The rich and easy-to-use AppZoo & Knowledge Distillation
EasyTransfer encapsulates the highly easy-to-use, flexible and low-cost learning AppZoo, which supports users to run “leading-edge” open source and self-developed algorithms on a “large scale” with only a few lines of commands, namely It can quickly access NLP applications under different scenarios and business data, including text vectorization, matching, classification, reading comprehension, and sequence labeling. And it integrates a rich knowledge distillation algorithm, which enables users to distill small online models with few parameters and high inference performance from large models with large parameters and slow inference speed. For example, EasyTransfer integrates the task-adaptive distillation model AdaBERT. From the new perspective of Neural Architecture Search (NAS), it searches for the small model architecture most suitable for the target task. On the 6 classic NLP tasks, the BERT model is compressed to The original 1/17~1/10, the reasoning speed up to 12~29 times the original. At the same time, the corresponding paper of the model has been accepted by the top AI conference IJCAI 2020.

The efficient transfer learning algorithm
EasyTransfer framework supports all mainstream transfer learning paradigms, including Model Fine-tuning, Feature-based TL, Instance-based TL, Model-based TL and Meta Learning. Based on these migration learning paradigms, more than 10 algorithms have been developed, and good results have been achieved in Ali's business practices. All subsequent algorithms will be open sourced to the EasyTransfer code base. In specific applications, users can choose a transfer learning paradigm to test the effect according to the following figure.

image.png

Integrating self-developed meta-learning algorithm for multi-tasking
EasyTransfer framework integrates multi-tasking learning algorithm based on meta-learning, which supports users to use data sets of other tasks for learning enhancement when training models for specific tasks. EasyTransfer integrates the self-developed Meta Fine-tuning algorithm, draws on the idea of ​​meta-learning, and aims to learn the cross-domain Meta-leaner of the pre-trained language model, so that the learned Meta-leaner can be quickly transferred to a specific domain On the task. The corresponding paper of this algorithm has been accepted by the top NLP conference EMNLP 2020. Since the above model still has too many parameters and slow inference speed, the EasyTransfer team further self-developed the meta-knowledge distillation algorithm. In the distillation stage, the Meta-leaner is additionally selectively distilled, so that the small model obtained by distillation is in the corresponding field. The effect is significantly improved, which is close to the effect of the original model. Related codes and papers will be released in the near future.

Fully compatible with Alibaba's PAI ecological product
EasyTransfer framework is fully compatible with PAI-Tensorflow, users can use PAI self-developed and efficient distributed training, compilation and optimization features through simple code or configuration file modification; at the same time, the framework is perfectly compatible with PAI ecological products , It can be used directly on PAI Web component (PAI Studio), development platform (PAI DSW), cloud native training platform (PAI DLC), and PAI Serving platform (PAI EAS).
Application landing and innovative algorithm solutions.

The EasyTransfer framework has been implemented in dozens of NLP scenarios within the Alibaba Group, including intelligent customer service, search recommendation, security risk control, large entertainment, etc., which has brought significant business effects. At present, there are hundreds of millions of calls to EasyTransfer's daily services, and the average monthly training call volume exceeds 50,000. The EasyTransfer team has accumulated a lot of innovative algorithm solutions while landing the business, including meta-learning, multi-modal pre-training, enhanced transfer learning, feature transfer learning and other directions. A total of dozens of top conference articles have been published. , Here are some representative works. Some of these algorithms have been open sourced, and other parts will be open sourced in the EasyTransfer framework for users to use.

[EMNLP 2020]. Meta Fine-Tuning Neural Language Models for Multi-Domain Text Mining. 2020.
[SIGIR2020] FashionBERT: Text and Image Matching for Fashion Domain with Adaptive Loss. 2020.
[IJCAI 2020] AdaBERT: Task-Adaptive BERT Compression with Differentiable Neural Architecture Search. 2020.
[KDD 2019]. A Minimax Game for Instance based Selective Transfer Learning. 2019.
[CIKM 2019]. Cross-domain Attention Network with Wasserstein Regularizers for E-commerce Search, 2019.
[WWW 2019]. Multi-Domain Gated CNN for Review Helpfulness Prediction, 2019.
[WSDM 2019]. Learning to Selectively Transfer: Reinforced Transfer Learning for Deep Text Matching. 2019.
[WSDM 2018]. Modeling Domain Relationships for Transfer Learning on Retrieval-based Question Answering Systems in E-commerce. 2018.
[ACL 2018]. Transfer Learning for Context-Aware Question Matching in Information-seeking Conversations in E-commerce. 2018.
[ICDM 2017]. A Short-Term Rainfall Prediction Model using Multi-Task Convolutional Neural Networks. 2017.

Author: Cen Ming / grape

Original link: https://developer.aliyun.com/article/782127?

Copyright statement: The content of this article is voluntarily contributed by Alibaba Cloud real-name registered users. The copyright belongs to the original author. The Alibaba Cloud Developer Community does not own its copyright and does not assume corresponding legal responsibilities. For specific rules, please refer to the "Alibaba Cloud Developer Community User Service Agreement" and the "Alibaba Cloud Developer Community Intellectual Property Protection Guidelines". If you find suspected plagiarism in this community, fill in the infringement complaint form to report it. Once verified, the community will immediately delete the suspected infringing content.

Guess you like

Origin blog.csdn.net/alitech2017/article/details/114538339