NVIDIA DLI Deep Learning Introductory Training | Three special sessions! !
April 28th/May 19th/May 26th One-The text consists of 1485 words, 2 pictures, and the estimated reading time is 5 minutes.
Hello everyone, let's take a look at the conditional random field today. The conditional random field is a big content and is widely used in Chinese word segmentation, because we have an understanding of the probabilistic graphical model and basic formal language knowledge in the previous article. When we go to learn conditional random fields now, it will be easier (I also read a lot of blogs before writing this article, and found that many bloggers come up to talk about a lot of core formulas, and the previous knowledge is very little. Mention, I don’t think it’s good, it will make a lot of people confused at first).
And I hope to reduce the repetition of pure theoretical knowledge as much as possible in these articles of mine, but use some examples, such as word segmentation, some practical operations, and CRF+ to implement the algorithm by hand. This way everyone may understand it better.
There are about three articles on conditional random fields:
Chapter 1: Let's talk about Chinese word segmentation
The second part: talk about the theory of conditional random field and its application in Chinese word segmentation
Part 3: Code Implementation of Writing Conditional Random Fields
And today's article will talk about Chinese word segmentation together:
In fact, word segmentation technology has been widely used in foreign countries, but in China, because of the particularity of Chinese (English is composed of a single word, while Chinese is composed of a single character), and the previous domestic research is relatively backward (mostly foreign People), until now, building a useful Chinese word segmentation thesaurus is still everyone's pursuit. And Chinese word segmentation is also a difficult and hot spot in the field of natural language processing. Think about it now that more and more people publish information on the Internet , and also obtain information on the Internet. Massive text data requires text information mining, and information extraction, intelligent question answering, and textual tendency analysis are all based on word segmentation. Therefore, the efficiency and accuracy of word segmentation are related to the above work. have a big impact.
Therefore, there are two key issues to be dealt with in word segmentation: segmentation disambiguation resolution and unregistered word recognition
At present, there are three main word segmentation methods:
One is the word segmentation method based on the dictionary, which is also called the mechanical word segmentation method. This method takes a long time. His work idea is to divide the words to be assigned with the entries in a sufficiently large dictionary according to a certain strategy. Matching is just like reciting verses. Some of them can be recited in their minds, but when the question maker comes up with one that has not been recited, they are confused. Therefore, this dictionary-based word segmentation method must require a high-quality dictionary. Support, the problem of unregistered word recognition and ambiguity recognition is simply powerless. But it is fast, efficient, easy to modify, and flexible.
For example: "Improve people's living standards"
The method using the dictionary will output all possibilities:
raise - raise
tall - tall
people - people
people - people's livelihood
Raw — life
live - living water
water - level
flat
After the graph model is formed, it will be:
The second is the method of word segmentation based on statistics. After using the method based on statistics, the effect of Chinese word segmentation has been significantly improved, which is much more powerful than the previous one, and the method of machine learning has been introduced at this time, using The divided words are used as a training corpus, and then a variety of models are selected to learn and decode, and finally the model is trained.
At this time, the protagonist of our article - the conditional random field has begun to come in handy. One idea is to establish a standard machine model of the conditional random field, and then introduce the probability characteristics of the text and domain knowledge to segment the words. The accuracy rate of such word segmentation However, there are too many probabilistic features of text and most of them are customized, so the model established in this way is not only complex but also huge.
And the statistical method is used, which depends to a large extent on the quality of the corpus you choose. Just like we learn machine learning models, if the representativeness of your training set is not obvious enough, the model you train will be easier to pass. Fitting. If your training corpus requirements are high, on the one hand, you have a large amount of calculation, which leads to low efficiency. On the other hand, high-quality corpus cannot be separated from manual screening, which is also a test for manpower.
The third method is based on the combination of statistics and dictionaries. Use, and your dictionary can be used as an internal training corpus, but such a defect is that the adaptability of word segmentation in different fields is not good, and the model needs to be retrained. This is also the effect of stammering word segmentation in some specialized fields. not good.
Original link: https://www.jianshu.com/p/b62cb8a256f2
For more concise and convenient classified articles and the latest course and product information, please move to the newly presented "LeadAI Academy Official Website":
www.leadai.org
Please pay attention to the artificial intelligence LeadAI public account to view more professional articles
Everyone is watching
Application of LSTM model in question answering system
TensorFlow-based neural network solves the problem of user churn overview
The Most Common Algorithm Engineer Interview Questions (1)
Sorting out the most common algorithm engineer interview questions (2)