Principle of DecisionTree Algorithm - Part II Ideas of Decision Tree ID3 Algorithm

Ideas of Decision Tree ID3 Algorithm

The ID3 algorithm uses the information gain size to determine what features the current node should use to construct the decision tree, and uses the calculated maximum gain of information to establish the current node of the decision tree. Here we give a concrete example of information gain calculation. For example, we have 15 samples D and the output is 0 or 1. Among them, 9 outputs are 0, and 6 outputs are 1. There is a feature A in the sample, which takes the values ​​A1, A2 and A3. In the output of the sample with the value A1, there are 3 outputs as 1, 2 outputs are 0, the sample output is A2, 2 outputs are 1, 3 outputs are 0, and the value is A3. In the sample, 4 outputs are 1, and 1 output is 0.

The entropy of sample D is: H(D)=−(9/15log₂9/15+6/15log₂6/15)=0.971

The conditional entropy of the sample D under the feature is:

H(D|A)=5/15H(D1)+5/15H(D2)+5/15H(D3)

          =−5/15(3/5log₂3/5+2/5log₂2/5)−5/15(2/5log₂2/5+3/5log₂3/5)−5/15(4/5log₂4/5+1/5log₂1/5)

          =0.888

The corresponding information gain is I(D,A)=H(D)−H(D|A)=0.083

Let's take a look at what the specific algorithm process is like.

The input is m samples, the sample output set is D, each sample has n discrete features, the feature set is A, and the output is decision tree T.

The process of the algorithm is:

1) Initialization information gain threshold ε

2) Determine whether the sample is the same type of output Di, and if so, return to the single node tree T. The tag category is Di

3) Determine whether the feature is empty. If yes, return to the single-node tree T. The tag category is the category with the largest number of instances of the output category D in the sample.

4) Calculate the information gain of each feature (a total of n) in output A for output D, and select the feature Ag with the largest information gain.

5) If the information gain of the Ag is less than the threshold ε, a single node tree T is returned, and the tag category is the category with the largest number of output class D instances in the sample.

6) Otherwise, according to the different values ​​of the characteristic Ag, Agi divides the corresponding sample output D into different categories Di. Each category produces a child node. The corresponding feature value is Agi. Returns the number T of nodes added.

7) For all child nodes, let D=Di, A=A−{Ag} recursively call 2-6 steps to get the subtree Ti and return.