Recommended practical introductory books for transfer learning technology: "Natural Language Processing Transfer Learning Practice"
[Ghana] Paul Azunre (Paul Azunre), Li Xiang, Zhu Zhongshu, Zhang Shiwu translation
A book will take you to understand the technology behind ChatGPT, natural language processing transfer learning, unlock a new realm of machine learning, from shallow to deep, master the mysteries of NLP transfer learning, and make your model stand out!
Recommended reason:
It is suitable for reading by developers related to machine learning and data science with NLP foundation, and it is also suitable as a reference book for students of computer and related majors in colleges and universities.
- The content is basic and practical. This book is a practical guide to transfer learning technology in the field of NLP, which can help readers quickly understand related machine learning concepts and apply them to real-world problems.
- The content technology is new. This book introduces in detail how to use transfer learning technology to solve problems in new scenarios, new tasks and new environments, making machine learning systems more reliable and robust.
- It is practical. This book provides practical guidance on how to use transfer learning to improve NLP models. Readers can learn how to start with a pre-trained model and tune it to meet exact needs to deliver advanced results.
- Covers a wide range of topics. This book covers a variety of NLP applications, including email spam classifiers, IMDb movie review sentiment analyzers, automated fact checkers, question answering systems, and translation systems, etc., enabling readers to gain a comprehensive understanding of the application of transfer learning in the field of NLP.
- The reference value is high. This book is not only suitable for reading by developers of machine learning and data science with NLP foundation, but also suitable for students of computer and related majors in colleges and universities and students of software colleges. It has high reference value.
- Clear and easy-to-understand text expressions: This book uses concise and clear text expressions to enable readers to easily understand the content of the book and lower the threshold for reading.
- The explanations in this book are simple and easy to understand. It not only has complete mathematical arguments, but also helps readers establish perceptual cognition through vivid language. People can't help but applaud the author's ingenuity! At the same time, this book takes a number of common problems in the NLP field as the main line, leads readers to practice themselves through sample codes, and appreciates the real business effect improvement brought by NLP transfer learning.
brief introduction:
As an important method in the field of machine learning and artificial intelligence, transfer learning has been widely used in computer vision, natural language processing (NLP), speech recognition and other fields. This book is a practical introduction to transfer learning technology, which can lead readers to practice natural language processing models in depth. First, this book reviews the key concepts in machine learning, and introduces the development history of machine learning, as well as the progress of NLP transfer learning; secondly, it discusses in depth some important NLP transfer learning methods—NLP shallow transfer learning and NLP deep transfer learning; finally, it covers an important subfield in the field of NLP transfer learning—deep transfer learning technology with Transformer as a key function. Readers can get hands-on with applying existing state-of-the-art models to real-world applications, including email spam classifiers, IMDb movie review sentiment classifiers, automated fact checkers, question answering systems, and translation systems, among others.
This book is concise in text, incisive in discussion, and clear in hierarchy. It is not only suitable for reading by developers related to machine learning and data science with NLP foundation, but also suitable as a reference book for students of computer and related majors in colleges and universities.
Professional Comments:
This book is an excellent exposition of transfer learning in the context of NLP. The content is easy to understand and the cases are rich, which is worth reading in depth. Transfer learning is essentially the reuse of knowledge and computing power. In areas such as target detection, pattern recognition, and NLP, transfer learning is promising.
——Xu Guoqiang, Chief Information Officer of Sany Heavy Industry SaaS
Transfer learning is one of the most important research directions in the field of NLP in recent years. This book gives a detailed introduction to the basic concepts, business applications and development directions of NLP transfer learning in the form of examples and codes. Many advanced models and algorithms introduced in this book are widely used in business practice. This is a good reference book for researchers who want to understand NLP transfer learning and implement it in practical work.
——Liang Lei, senior technical expert of Ant Group
Migration learning technology has made great progress in perceptual machine learning scenarios, especially the pre-training models of the BERT series have raised the baseline in the NLP field to a new level. This book systematically, comprehensively and practically introduces this fast-growing topic, and is worthy of in-depth reading and exploration by engineers in the NLP field.
——Zhu Liang, Senior Algorithm Engineer at Meta (formerly Facebook)
Transfer learning is a revolutionary technological breakthrough in the field of machine learning, especially in the field of NLP, which has made exciting achievements. At the same time, we believe that the profound ideas of transfer learning will also be applied to other fields and achieve good results. It is recommended that engineers in machine learning related fields read this book and maintain a continuous focus on transfer learning.
——Liu Bingyang, Google Senior Algorithm Engineer
This book gives a comprehensive and detailed introduction to the theory of transfer learning, which can help readers establish a clear understanding. What's even more rare is that this book is driven by practical business issues, leading readers to read and learn. Recommended to related engineers in the NLP field.
——Zhao Hai, technical expert of Meituan
full catalog
Introduction to the first part
Chapter 1 Introduction to Transfer Learning 3
1.1 Overview of typical tasks in the NLP field 5
1.2 Understanding NLP technology in the context of artificial intelligence 6
1.2.1 Artificial Intelligence 7
1.2.2 Machine Learning 7
1.2.3 Natural Language Processing 11
1.3 A Brief History of NLP Development 12
1.3.1 Introduction to NLP 12
1.3.2 Progress in transfer learning 14
1.4 Transfer Learning in Computer Vision 16
1.4.1 Overview 16
1.4.2 ImageNet pre-training model 17
1.4.3 Fine-tuning of ImageNet pre-trained model 18
1.5 Reasons why transfer learning in NLP is an exciting research topic 19
Summary 20
Chapter 2 From Scratch: Data Preprocessing 21
2.1 Preprocessing of sample data in spam email classification task 23
2.1.1 Loading and viewing the Enron email corpus 24
2.1.2 Loading and viewing the fraudulent email dataset 26
2.1.3 Convert email text to numeric value 30
2.2 Preprocessing of Example Data in Movie Review Sentiment Classification Task32
2.3 Generalized linear models 35
2.3.1 Logistic regression 36
2.3.2 Support Vector Machines 38
Summary 38
Chapter 3 From Scratch: Benchmarking and Optimization 41
3.1 Decision tree-based models 41
3.1.1 Random Forest 42
3.1.2 Gradient Boosting Machine 42
3.2 Neural Network Model 47
3.2.1 Language model embedding 47
3.2.2 Introduction to BERT model 52
3.3 Effect optimization 56
3.3.1 Manual hyperparameter tuning 56
3.3.2 Systematic hyperparameter tuning 57
Summary 58
The second part is shallow transfer learning and deep transfer learning based on recurrent neural network
Chapter 4 Shallow Transfer Learning for NLP 63
4.1 Semi-supervised learning based on pre-trained word embeddings 65
4.2 Semi-supervised learning with advanced representations 69
4.3 Multi-task learning 71
4.3.1 Problem formulation and shallow neural network single-task baseline 72
4.3.2 Dual-task experiment 74
4.4 Domain Adaptation 75
Summary 78
Chapter 5 Data Preprocessing for Deep Transfer Learning Experiments Based on Recurrent Neural Networks 79
5.1 Preprocessing of tabular categorical data 81
5.1.1 Obtaining and viewing table data 82
5.1.2 Preprocessing tabular data 85
5.1.3 Digitally encoding preprocessed data 87
5.2 Fact checking of preprocessed example data 87
5.2.1 Special problem considerations 88
5.2.2 Loading and viewing fact-checking data 88
Summary 89
Chapter 6 Deep Transfer Learning for NLP Based on Recurrent Neural Networks 91
6.1 SIMOn 91
6.1.1 Overview of general neural network architectures 92
6.1.2 Modeling tabular data 93
6.1.3 Application of SIMOn in Tabular Data Classification 93
6.2 ELMo 100
6.2.1 ELMo Bidirectional Language Modeling 101
6.2.2 Application of ELMo to Fake News Detection Task 102
6.3 ULMFiT 104
6.3.1 Fine-tuning with language model as target task 104
6.3.2 Fine-tuning with classification as the target task 105
Summary 105
The third part is Transformer-based deep migration learning and adaptation strategy
Chapter 7 Transformer-based Deep Transfer Learning and GPT 109
7.1 Transformer 110
7.1.1 Introduction to transformers library and attention visualization 113
7.1.2 Self-attention 115
7.1.3 Residual connections, encoder-decoder attention, and positional encoding 117
7.1.4 Application of Pretrained Encoder-Decoder to Machine Translation Tasks 120
7.2 GPT 121
7.2.1 Overview of the model structure 122
7.2.2 Transformer pipeline and its application in text generation tasks 124
7.2.3 Application in chatbot tasks 126
Summary 128
Chapter 8 Deep Transfer Learning for NLP Based on BERT and mBERT 129
8.1 BERT 130
8.1.1 BERT model structure 131
8.1.2 Application in automatic question answering tasks 134
8.1.3 Application in blank filling and follow-up sentence prediction tasks 136
8.2 Cross-lingual learning with mBERT 138
8.2.1 Overview of JW300 dataset 139
8.2.2 Transfer mBERT to Twi monolingual data with pre-trained tokenizer 139
8.2.3 Training mBERT model and tokenizer from scratch based on Twi monolingual data 142
Summary 143
Chapter 9 Adaptation Strategy of ULMFiT and Knowledge Distillation 145
9.1 Gradual unfreezing and differential fine-tuning 146
9.1.1 Pre-trained language model fine-tuning 147
9.1.2 Fine-tuning with classification as the target task 149
9.2 Knowledge Distillation 151
Summary 156
Chapter 10 ALBERT, Adapters, and Multitasking Adaptation Strategies 157
10.1 Embedding factorization and cross-layer parameter sharing 158
10.2 Multitasking fine-tuning 162
10.2.1 GLUE dataset 163
10.2.2 GLUE single-task fine-tuning 164
10.2.3 Serialization adaptation 167
10.3 Adapters 169
Summary 171
Chapter 11 Summary 173
11.1 Overview of key concepts 174
11.2 Other Emerging Research Trends 179
11.2.1 RoBERTa 180
11.2.2 GPT-3 180
11.2.3 XLNet 182
11.2.4 BigBird 182
11.2.5 Longformer 182
11.2.6 Reformer 183
11.2.7 T5 183
11.2.8 BART 184
11.2.9 XLM 185
11.2.10 TAPAS 185
11.3 The Development Direction of NLP Transfer Learning 186
11.4 Ethical and environmental considerations 187
11.5 Recent Developments 189
11.5.1 Kaggle and Zindi Competition 189
11.5.2 arXiv 190
11.5.3 News and social media 190
11.6 Written at the end 191
Appendix A Introduction to Kaggle 193
A.1 Free use of GPU through Kaggle Kernel 193
A.2 Contests, Discussion Boards, and Blogs 198
Appendix B Introduction to Basic Tools for Deep Learning 201
B.1 Stochastic Gradient Descent 202
B.2 TensorFlow 203
B.3 PyTorch 206
B.4 Transformers library for Keras, fast.ai library and Hugging Face 207