With the development of NLP technology, information flow and computing capabilities have been continuously enhanced. We now only need to enter a few characters in the search bar to retrieve the exact information needed to complete the task. The first few auto-completion options provided by the search are usually so suitable that we feel like someone is helping us with the search.
What is driving the development of NLP?
- Is it a new understanding of the ever-expanding unstructured web data?
- Is the increase in processing power keeping up with the researchers' thinking?
- Has the efficiency of using human language to interact with machines improved?
In fact, all of the above are, in fact, there are more. You can enter the question "Why is natural language processing so important now?" in any search engine , and you can find articles on Wikipedia that give various good reasons.
There are some deeper reasons, one of which is the accelerated pursuit of general artificial intelligence (AGI) or deep artificial intelligence (Deep AI). Human wisdom may only be reflected in our ability to organize our thoughts into discrete concepts, store them (memory) and share them effectively. This allows us to expand our intelligence across time and space and connect our brains to form collective intelligence.
Steven Pinker put forward a point in " The Stuff of Thought " ( The Stuff of Thought ): We actually think in natural language. Calling it "inner dialogue" is not without reason. Facebook, Google, and Elon Musk are betting on the fact that text will become the default communication protocol for thinking. They all invested in some projects, trying to convert thoughts, brain waves and electrical signals into words. In addition, Wolfe hypothesis believes that language affects our way of thinking. Natural language is undoubtedly the medium of culture and collective consciousness.
Therefore, if we want to imitate or simulate human thinking on machines, natural language processing may be crucial. In addition, you will learn important clues about intelligence that may be hidden in the data structure of words and nesting relationships in "Natural Language Processing in Action". You will use these structures, and neural networks enable inanimate systems to digest, store, retrieve, and generate natural language in ways that look like humans.
There is a more important reason, why do you want to learn how to write a system that uses natural language? This is because you may be able to save the world! I hope everyone has paid attention to the discussion between the bigwigs on the issue of artificial intelligence control and the challenge of developing "friendly artificial intelligence". Nick Bostrom, Calum Chace, Elon Musk, and many others believe that the future of mankind depends on our ability to develop friendly machines. In the foreseeable future, natural language will become an important link between humans and machines.
Even if we can "think" directly through machines, these ideas are likely to be shaped by natural words and language in our brains. The boundary between natural language and machine language will become blurred, just as the boundary between human and machine will disappear. In fact, this line began to blur in 1984, when the publication of the "Cyborg Manifesto" made George Orwell's dystopian prophecy more possible and acceptable.
I hope that the phrase "help save the world" doesn't make everyone confused. As this book progresses, we will show readers how to build and connect the "brain" of a chat robot. In this process, readers will find that in the social feedback loop between humans and machines, even small disturbances may have a profound impact on machines and humans. Just like a butterfly flapping its wings somewhere, a small adjustment to the "selfishness" of a chatbot may bring a chaotic storm of conflicting behaviors between rival chatbots. You will also notice that some kind and selfless systems will quickly gather a group of loyal supporters to help quell the chaos caused by those short-sighted robots. Due to the network effect of pro-social behavior, pro-social collaborative chatbots can have a huge impact on the world.
This is why the authors of "Natural Language Processing in Action" gathered together. Through open, honest, and pro-social communication on the Internet using our natural language, a supportive community has been formed. We are using collective intelligence to help build and support other semi-intelligent participants (machines). We hope that our words will leave a deep impression in everyone's minds, and spread widely in the chatbot world like meme, infecting others with the enthusiasm of building a pro-social NLP system. We hope that when superintelligence finally emerges, this pro-social spirit can slightly promote it.
Natural language processing practical use Python to understand, analyze and generate text
[America] Hobson Lane, Cole Howard, Hannas Max Hapke, Shi Liang, Lu Xiao, Tang Kexin, Wang Bin Translated
brief introduction
This book is a practical book that introduces natural language processing (NLP) and deep learning. NLP has become the core application field of deep learning, and deep learning is a necessary tool in NLP research and application. This book is divided into 3 parts: The first part introduces the basics of NLP, including word segmentation, TF-IDF vectorization, and the conversion from word frequency vectors to semantic vectors; the second part describes deep learning, including neural networks, word vectors, and convolutional neural networks ( Basic deep learning models and methods such as CNN), Recurrent Neural Network (RNN), Long Short-Term Memory (LSTM) network, sequence-to-sequence modeling and attention mechanism; the third part introduces the actual content, including information extraction, question and answer Model construction, performance challenges and coping methods of real-world systems such as systems and human-machine dialogue.
This book is aimed at middle and senior Python developers, and has both basic theory and actual programming. It is a practical reference book for practitioners in the modern NLP field.
Editor's Choice
- Essential Natural Language Processing for Python Developers
- A practical reference guide for practitioners in the field of modern natural language processing
- Translation provided by the NLP team of Xiaomi AI Lab
1. This book is a practical guide for building machines that can read and interpret human language;
2. Readers can use existing Python packages to capture the meaning of text and respond accordingly;
3. This book extends traditional natural language Processing methods, including neural networks, modern deep learning algorithms and generation techniques, are used to solve real-world problems, such as extracting dates and names, synthesizing text and answering questions with no fixed format;
4. Provide source code.
Advances in the field of deep learning have enabled various applications to understand text and speech with extreme accuracy. As a result, chat bots can simulate humans to screen out resumes that highly match the position, perform excellent predictive searches, and automatically generate document summaries-all of which can be achieved at a very low cost. The development of new technologies, together with the emergence of easy-to-use tools such as Keras and TensorFlow, has made professional-quality natural language processing (NLP) easier than ever.
The main content of this book
● Use of tools such as Keras, TensorFlow, gensim and scikit-learn.
● Rule-based natural language processing and data-based natural language processing.
● Scalable natural language processing pipeline.
To read this book, readers need to have a basic understanding of deep learning knowledge and have medium-level Python programming skills.
Features of "Natural Language Processing Actual Combat":
About the author
table of Contents
The first part of the text processing machine
Chapter 1 NLP overview 3
1.1 Natural language and programming language 3
1.2 Magical magic 4
1.2.1 Machine that can talk 5
1.2.2 Mathematics in NLP 5
1. 3 Practical application 7
1.4 Language in the computer "eye" 8
1.4.1 Lock language (regular expression) 9
1.4.2 Regular expression 9
1.4.3 A simple chat robot 11
1 .4.4 Another method 14
1.5 A brief description of hyperspace 17
1.6 Word order and grammar 19
1.7 Natural language pipeline of chatbots 20
1.8 Deep processing 22
1.9 Natural language IQ 24
1.10 Summary 26
Chapter 2 Build Your Own Vocabulary-Word Segmentation 27
2.1 Challenge (Preview of Stemming) 28
2.2 Use Word Segmenter to Build Vocabulary 29
2.2.1 Dot Product 37
2.2.2 Measure Words Coincidence between bags 37
2.2.3 Punctuation processing 38
2.2.4 Extending the vocabulary to n-gram 43
2.2.5 Normalizing the vocabulary 48
2.3 Emotion 55
2.3 ．1 VADER: a rule-based sentiment analyzer 56
2.3.2 Naive Bayes 58
2.4 Summary 61
Chapter 3 Mathematics in
Words 62 3.1 Bag of Words 63
3.2 Vectorization 67
3.3 Zipf's Law 74
3.4 Topic Modeling 76
3.4.1 Back to Zipf's law 79
3.4.2 Relevance ranking 80
3.4.3 Tools 82
3.4.4 Other tools 83
3.4.5 Okapi BM25 85
3.4.6 The future Outlook 85
3.5 Summary 85
Chapter 4 The semantics behind word frequency 87
4.1 From word frequency to topic score 88
4.1.1 TF-IDF vector and morphological merge 88
4.1.2 Topic vector 89
4.1. 3 Thought experiment 90
4.1.4 A topic scoring algorithm 94
4.1.5 An LDA classifier 95
4.2 Latent semantic analysis 99
4.3 Singular value decomposition 103
4.3.1 Left singular vector U 105
4. 3.2 Singular value vector S 106
4.3.3 Right singular vector VT 107
4.3.4 Direction of SVD matrix 107
4.3.5 Theme reduction 108
4.4 Principal component analysis 109
4.4.1 PCA on three-dimensional vectors 111
4.4.2 Regression NLP 112
4.4.3 Short message semantic analysis based on PCA 114
4.4.4 Short message semantic analysis based on truncated SVD 116
4.4 .5 The effect of short message classification based on LSA 117
4.5 Potential Dirichlet distribution (LDiA) 119
4.5.1 The idea of LDiA 120
4.5.2 The semantic analysis of short messages based on LDiA topic model 121
4.5 .3 LDiA+LDA=Spam filter 124
4.5.4 Fairer comparison: 32 LdiA topics 125
4.6 Distance and similarity 127
4.7 Feedback and improvement 129
4.8 The power of topic vectors 132
4 .8.1 Semantic Search 133
4.8.2 Improvement 135
4.9 Summary 135
Part Two Deep Learning (Neural Network)
Chapter 5 Preliminary Neural Network (Perceptron and Back Propagation) 139
5.1 Composition of Neural Network 140
5.1.1 Perceptron 140
5.1.2 Digital Perceptron 141
5.1.3 Understanding Bias 142
5.1.4 Error Surface 153
5.1.5 Different Types of Error Surface 154
5.1. 6 Multiple gradient descent algorithms 155
5.1.7 Keras: implement neural network 155 with Python
5.1.8 Outlook 158
5.1.9 normalization: formatted input 159
5.2 Summary 159
Chapter 6 word vector reasoning (Word2vec) 160
6. 1 Semantic query and analogy 160
6.2 Word vector 162
6.2.1 Vector-oriented reasoning 165
6.2.2 How to calculate Word2vec representation 167
6.2.3 How to use gensim. word2vec module 175
6.2.4 Generate customized word vector representation 177
6.2.5 Word2vec and GloVe 179
6.2.6 fastText 180
6.2.7 Word2vec and LSA 180
6.2.8 Visualization of word relationships 181
6 .2.9 Unnatural words 187
6. 2.10 Calculating document similarity with Doc2vec 188
6.3 Summary 190
Chapter 7 Convolutional Neural Network (CNN) 191
7.1 Semantic Understanding 192
7.2 Toolkit 193
7. 3 Convolutional neural network 194
7.3.1 Building block 195
7.3.2 Step size 196
7.3.3 The composition of the convolution kernel 196
7.3.4 Filling 198
7.3.5 Learning 199
7.4 Narrow window 199
7.4.1 Keras implementation: preparing data 201
7.4.2 Convolutional neural network architecture 206
7.4.3 Pooling 206
7.4.4 dropout 208
7.4.5 Output layer 209
7.4.6 Start learning (training) 211
7.4.7 Use model in pipeline 212
7.4.8 Prospects 213
7.5 Summary 214
Chapter 8 Recurrent Neural Network (RNN) 215
8.1 Recurrent network memory function 217
8.1.1 Backpropagation algorithm over time 221
8.1.2 Weight update at different moments 223
8.1.3 Brief review 225
8.1.4 Difficulties 225
8.1.5 Using Keras to realize cyclic neural network 226
8.2 Integrating parts 230
8.3 Self-learning 231
8.4 Hyperparameters 232
8.5 Prediction 235
8.5.1 Stateful 236
8.5. 2 Two-way RNN 236
8.5.3 Encoding vector 238
8.6 Summary 238
Chapter 9 Improving memory: Long short-term memory network (LSTM) 239
9.1 Long short-term memory (LSTM) 240
9.1.1 Backpropagation over time 247
9.1.2 Use of models 250
9.1.3 Dirty data 251
9.1.4 Processing of "unknown" entries 254
9.1.5 Character-level modeling 255
9.1.6 Generating chat text 260
9.1.7 Further generating text 262
9.1.8 The problem of text generation: uncontrolled content 269
9.1.9 Other memory mechanisms 269
9.1.1 Deeper Network 270
9.2 Summary 271
Chapter 10 Sequence-to-Sequence Modeling and Attention Mechanism 272
10.1 Encoding-Decoding Architecture 272
10.1.1 Decoding Ideas 273
10.1.2 Deja vu? 275
10.1.3 Sequence-to-sequence dialogue 276
10.1.4 Review LSTM 277
10.2 Assemble a sequence-to-sequence pipeline 278
10.2.1 Prepare a data set for sequence-to-sequence training 278
10.2.2 Keras Sequence-to-sequence model in 279
10.2.3 Sequence encoder 280
10.2.4 Idea decoder 281
10.2.5 Assemble a sequence to sequence network 282
10.3 Training sequence to sequence network 282
10.4 Use Sequence-to-sequence network to build a chatbot284
10.4.1 Prepare corpus for training 285
10.4.2 Build character dictionary 286
10.4.3 Generate one-hot coding training set 286
10.4.4 Train sequence to sequence chat robot 287
10.4.5 Assemble sequence Generating model 288
10.4.6 Predicting output sequence 288
10.4.7 Generating replies 289
10.4.8 Talking with chatbots 290
10.5 Enhancement 290
10.5.1 Using bucketing to reduce training complexity 290
10 .5.2 Attention mechanism 291
10.6 Practical applications 292
10.7 Summary 294
Part III Enter the real world (NLP challenges in reality)
Chapter 11 Information extraction (named entity recognition and question answering system) 297
11.1 Naming Entities and Relations 297
11.1.1 Knowledge Base 298
11.1.2 Information Extraction 300
11.2 Regular Pattern 300
11.2.1 Regular Expression 301
11.2.2 Regarding Information Extraction as a Feature in Machine Learning Extraction task 302
11.3 Information worth extracting 303
11.3.1 Extracting GPS location 303
11.3.2 Extracting date 304
11.4 Extracting person relations (things) 309
11.4.1 Part of speech tagging 309
11.4.2 Entity name standardization 313
11.4.3 Entity relationship standardization and extraction 314
11.4.4 Word pattern 314
11.4.5 Text segmentation 314
11.4.6 Why the split('.!') function Regardless of 316
11.4.7 Use regular expressions to segment sentences 316
11.5 Real-world information extraction 318
11.6 Summary 319
Chapter 12 Start chat (dialog engine) 320
12.1 Language skills 321
12.1.1 Modern method 322
12.1.2 Hybrid method 326
12.2 Pattern matching method 327
12.2.1 AIML-based pattern matching chatbot 328
12.2.2 Network view of pattern matching 334
12.3 Knowledge method 334
12. 4 Retrieval (search) method 336
12.4.1 Context challenge 336
12.4.2 Chatbot 338 based on example retrieval
12.4.3 Chatbot 341 based on search
12.5 Generative method 343
12.5.1 Talk about NLPIA 343
12.5.2 Pros and cons of each method 345
12.6 Four-wheel drive 345
12.7 Design process 347
12.8 Skills 349
12.8.1 use with predictable answers to questions Question 349
12.8.2 To interesting 350
12.8.3 When all else fails, search 350
12.8.4 become popular 350
12 8.5 Become a connector 351
12.8.6 Become emotional 351
12.9 Real world 351
12.10 Summary 352
Chapter 13 Scalability (optimization, parallelization, and batch processing) 353
13.1 Too much ( Data) is not necessarily a good thing 353
13.2 Optimizing NLP algorithm 354
13.2.1 Index 354
13.2.2 Advanced index 355
13.2.3 Advanced index based on Annoy 357
13.2.4 Why use approximate index 361
13.2.5 Index workaround: Discretization 362
13.3 Constant-level memory algorithm 363
13.3.1 gensim 363
13.3.2 Graph calculation 363
13.4 Parallel NLP calculation 364
13.4.1 In Training NLP models on GPU 364
13.4.2 Rent and buy 365
13.4.3 GPU lease options 366
13.4.4 Tensor processing unit TPU 367
13.5 Reduce memory usage during model training 367
13.6 Using TensorBoard to Understand Models 369
13.7 Summary 372
Appendix A NLP Tools Complementing the Book 373
Appendix B Interesting Python and Regular Expressions 380
Appendix C Vectors and Matrices (Basics of Linear Algebra) 385
Appendix D Common Tools and Machine Learning Technique 391
Appendix E Setting up the GPU on Amazon Cloud Services (AWS) 403
Appendix F Partially Sensitive Hashing 415
Resources 421
Glossary 428