NLP basic introduction

What is NLP?
NLP = NLU + NLG
NLU: language, text -> meaning
NLG: meaning -> voice text
Solve the problem: a word with multiple meanings.
How to solve: If there is no context, you can only guess by the probability of word usage; if there is context, add context, which is the information related to the word.
For example: machine translation.
Original: Each word is compared and translated.
Improvement: First perform a rough translation, and then select the translation with the highest probability from the possible translations. Disadvantages: time complexity is high.
Language model: Given a sentence of English e, calculate the probability (e), if it is in line with English grammar, p(e) will be high, if it is a random sentence, p(e) will be low.
Translation model: given a pair of <c,e >, calculate p(f|e), if the semantic similarity is high, then p(f|e) is high, otherwise it is low.
Decoding Algorithm: Given the language model, translation model and f, find the optimal one that maximizes p(e)p(f|e).
NLP classic practical scenarios
1. Question answering system
Knowledge base
2. Sentiment analysis
Stock price prediction, public opinion monitoring, product reviews, event monitoring
3. Machine translation
4. Automatic summarization
5. Chat robot
6. Information extraction (the most important area)
NLP Key technology
1. Natural language processing technology Four dimensions of
semantics (the final goal, NLU, machine learning algorithm) Sentence structure (focus on the relationship between words and words, including syntactic analysis, very dependent on language, but also dependency analysis, analysis What is the relationship between each word) Word (bottom technical participle, part of speech) Sound
2. Word segmentation algorithm
3. Part of speech
4. Named entity recognition
5. Relation extraction