In the past
In the past, a small evening planted tree planted in a cabin behind the small Xi ~
Why do you want to plant this tree? Yeah because it can help small evening historical experience, and then make a decision to help small evening - so small evening on the current situation can give you peace of mind to write articles friends ~
This tree is this.
At first, small evening to buy a small sapling, it plummeted to the ground.
And then, small evening without watering it, do not give him pouring fertilizer, but let him eat the historical experience. For example today, let it help small evening small evening identify biological picture is meow or donkey. So, here is a lot of historical experience and donkey meow pictures, and small evening to give it a good mark each picture is meow or donkey.
The eve of the election the following several small features to describe each picture:
1, if the head is elongated
2, there is no hair body
3, there is found no beard
4, no ugly ugly
5, Meng not Meng
So, for example small evening will be this picture to the following:
Converted to [False True False False True] The feature vectors (i.e., not long head, hairy, beard not found, not ugly, Meng)
Then the eve of breath to thousands of small pictures feature extraction is over, of course, these pictures are marked good category.
Small evening put them threw small evening beloved friends - little tree
Grow up quickly
Eh? That saplings how to grow it?
Asked the little sapling small evening: "trees trees, how long you want to do next?"
Saplings drink some water, said:. "My every step of the growth is very careful, I long branches from a not long in branches or less per sample This is not a five characteristics Well, I do, will pick out a most valuable feature, as a branch for the first time I grew up. "
Small evening then asked: "What do you want to choose how characteristic it?"
Little Tree said: "You see, ah, for example, you pick out the second feature," who have no hair, "this is a very bad feature we look at why.?" There is no hair. "I count this feature. for a moment, all the animals have hair in 50% of the animals is meow, 50% of the animals are donkeys. and in all no hair of animals, but also 50% of the animals meow, 50% of the animals are donkeys. so "there is no hair" either a value of this feature, the same number of samples in all categories, is entirely equal probability, this feature can not be used for classification ah completely, because no matter what the value of this feature is that it mapping each category are equally likely, so this category is the worst. "
Saplings drink some water, and then said: "But you use a third feature," There is no beard ", the great I figure a bit, animals beard, the meow accounted for 92%, accounting for 8 donkey. %, while animals without a mustache, a meow accounted for 20%, accounting for 80% of the donkey. Thus, if you give me a sample of unknown class, just to see "there is no beard," this feature, then basically you can determine What is the class of this sample! For example, the value of this feature of the sample happens to be 'no beard ", then I say there are 80% sure this is a donkey! of course, if the value of the characteristics of this sample is just" beard, "I have a 92% certainty say meow when this sample! such a great feature, is the preferred course of it! so, I want to grow below the branches is called" whiskers have no branches "!"
Small evening:
"I might buy a fake tree.
I. . . I understand what you mean. You say the best feature of the method of selecting the actual process conditional entropy is calculated. For you said, "There is no hair" This is very bad feature, the value of this feature at each of the various categories are completely equal probability, it is the most chaotic, most random features, we called "conditional entropy greatest features." 1 is the maximum entropy, so calculated "There is no hair" conditional entropy of this feature is certainly 1.
And you say, "There is no beard," this feature, at each of its value, category distribution is random, that is very orderly (Imagine the most ordered state is the next value each, all samples are is the same category, how orderly ah. and when the probability distribution category, etc., is quite a mixed bag of features at each time, what categories are, a mess), conditional entropy of this feature very orderly, calculated affirmation small. The most orderly, the conditional entropy is 0 friends. "
Trees:. "Well, but, my goal is to achieve 100% classification accuracy So, I want to further pick up features.!
Based on that branch bearded, I will this branch in the sample recalculate conditional entropy of the various features (except of course the beard this feature), as before select the maximum conditional entropy as the next feature branches!
This process is repeated, until all the samples in which the branches are in the same category, this will no longer continue to divide the branches. "
Eventually, the trees grew into a tree.
Then the little busy evening for everyone to writing it, but a small evening of good girlfriends Xiaoya took a picture over, she wanted to help her small evening to see if this is a donkey or meow.
But small evening busy Yeah, Not for her, let her backyard tree took pictures and asked friends ~
Xiao Ya ask then what is this tree, decision tree, said: "? It has a beard it."
Xiaoya said: "There!"
Tree: "It Meng it?"
Xiao Ya: "Not Moe!"
Tree: "It is a long face it?"
Xiao Ya: "a long face!"
Tree: "Well, this is just an ass!"
Xiao Ya look ignorant forced to leave under the tree messy (¯∇¯)
What tree magic tree called what?
This tree of professional theory called "ID3 decision tree", ID3 Why is it? Because it is optimum to select a feature by calculating the conditional entropy. The other classical tree tree C4.5, CART decision tree ID3 differs only in that the optimal feature selection algorithm.
to sum up
Then this article sum up, we go through the following steps, which is a simple machine learning / data mining tasks classical processes.
1, the category of the labeled data set with the feature extraction data preprocessing
2, decision tree is trained (classifier / machine learning models)
3, data sets of unknown class data preprocessing and feature extraction
4, using the decision tree category category unknown samples to make decisions
Want to learn more about the ID3 choose the best features of the process (ie calculate the conditional entropy) students can Google or Baidu, "conditional entropy" or refer to "Introduction to data mining" and other machine-learning-related books; you want to learn more about how code to achieve the ID3 students can search for all kinds of CSDN blog or refer to "combat machine learning" and other books focus on code implementation. How to improve the generalization ability of the decision tree (to prevent over-fitting) and other optimization techniques, or other type of decision tree, you can see "Introduction to data mining" and other classic books -
