Generation of decision tree - ID3 algorithm

The origin of the algorithm:

The decision tree algorithm was originally proposed by Hunt Earl B. CLS (Concept Learning System), but it did not give any method to select the optimal feature. Later, J. Ross Quinlan proposed the ID3 algorithm, using [Information Gain] to determine Afterwards, Rosquin optimized and improved the ID3 algorithm to obtain the C4.5 algorithm, and used the information gain ratio to determine the optimal feature. The essence of the two algorithms is similar, but the method of determining the optimal feature is different. The ID3 algorithm is biased towards selecting a certain feature with a large number, and the C4.5 algorithm is biased towards the selection of a certain number of feature units.

ID3 algorithm

Input: training data set D, feature set A, threshold $\in$
output: decision tree T

Determine whether T needs to select features to generate a decision tree:

——If all the instances in D belong to the same class, then T is a single-node tree, recording the instance class $C_k$ Use this as the class mark of the node and return T:
——If all instances in D have no features $(A=\emptyset)$ , then T is a single-node tree, record the category $C_k$ , take it as the class mark of the node, and return T;
Otherwise, calculate the information gain of each feature in A , and select the feature Ag with the largest information gain :

——If the information gain of Ag is less than, then T is a single-node tree, and the category $C_k with the largest number of instances in D is recorded$ , take this as the class mark of the node, and return T;
——Otherwise, according to each possible value of Ag $a_i$ , divide D into several non-empty subsets $D_i$ ,将 $D_i$ The category with the largest number of instances in is used as a mark, and a sub-node is constructed, and T is composed of a node and its sub-nodes, and T is returned;
The i-th child node, with $D_i$ is the training set, A-Ag is the feature set, recursively call the above steps, and get the subtree $T_i$ and return.

The core of the decision tree idea is that when the maximum information gain calculated by each iterative training set is less than the threshold, the single-node tree can be generated to terminate.

Example:

The calculated empirical entropy, empirical conditional entropy, and information gain

of each feature are as follows: the maximum information gain is the feature of your own house, which is divided into two categories: those with a house and those without a house. If you have a house, set a single node to agree to the loan. The training data set without houses is used as the new training data set $D_2$ , and then calculate the information gain from the remaining features, and select the feature with the largest information gain.

Take the feature: age is $A_1$ , has work for $A_2$ , the credit status is $A_3$

信息增益公式：
$g\left(D_{2}, A_{i}\right)=H\left(D_{2}\right)-H\left(D_{2} \mid A_{i}\right)$
经验熵：
$\begin{aligned} H\left(D_{2}\right) &=-\frac{6}{9} \log _{2}\left(\frac{6}{9}\right)-\frac{3}{9} \log _{2}\left(\frac{3}{9}\right) \\ &=0.918 \end{aligned}$
Feature: age

Computation:

青年：

$\begin{aligned} &w_{1}=\frac{\left|D_{21}\right|}{\left|D_{2}\right|}=\frac{4}{9} \\ &H\left(D_{21}\right)=-\frac{1}{4} \log _{2} \frac{1}{4}-\frac{3}{4} \log _{2} \frac{3}{4}=0.811 \end{aligned}$
中年：
$\begin{aligned} &w_{2}=\frac{\left|D_{21}\right|}{\left|D_{2}\right|}=\frac{2}{9} \\ &H\left(D_{22}\right)=0 \end{aligned}$
老年：
$\begin{aligned} &w_{3}=\frac{\left|D_{23}\right|}{\left|D_{2}\right|}=\frac{3}{9} \\ &H\left(D_{23}\right)=-\frac{2}{3} \log _{2} \frac{2}{3}-\frac{1}{3} \log _{2} \frac{1}{3}=0.918 \end{aligned}$
经验条件熵：
$\begin{aligned} H\left(D_{2} \mid A_{1}\right) &=w_{1} H\left(D_{21}\right)+w_{2} H\left(D_{22}\right)+w_{3} H\left(D_{23}\right) \\ &=0.360+0+0.307 \\ &=0.667 \end{aligned}$
Feature: Whether there is a job

Computation:
There is a job:
$\begin{aligned} &w_{1}=\frac{\left|D_{22 }\right|}{\left|D_{2}\right|}=\frac{3}{9} \\ &H\left(D_{22}\right)=0 \end{aligned}$
没有工作：
$\begin{aligned} &w_{2}=\frac{\left|D_{22}\right|}{\left|D_{2}\right|}=\frac{6}{9} \\ &H\left(D_{22}\right)=0 \end{aligned}$
Entropy of empirical conditions:
$H(D_2|A_3)=w_1H(d_{21})+w_2H(D_{22 })=0$

Feature: Credit Profile

Calculation:

非常好：

$\begin{aligned} &w_{1}=\frac{\left|D_{23}\right|}{\left|D_{2}\right|}=\frac{1}{9} \\ &H\left(D_{23}\right)=0 \end{aligned}$
好：
$\begin{aligned} &w_{2}=\frac{\left|D_{23}\right|}{\left|D_{2}\right|}=\frac{3}{9} \\ &H\left(D_{23}\right)=-\frac{2}{3} \log _{2} \frac{2}{3}-\frac{1}{3} \log _{2} \frac{1}{3} \end{aligned}$
老年：
$\begin{aligned} &w_{1}=\frac{\left|D_{23}\right|}{\left|D_{2}\right|}=\frac{5}{9} \\ &H\left(D_{23}\right)=-\frac{1}{5} \log _{2} \frac{1}{5}-\frac{4}{5} \log _{2} \frac{4}{5} \end{aligned}$
经验条件熵：
$\begin{aligned} H\left(D_{2} \mid A_{3}\right) &=w_{1} H\left(D_{21}\right)+w_{2} H\left(D_{22}\right)+w_{3} H\left(D_{23}\right) =0.444 \end{aligned}$
Synthesis:
The comparison found that the feature is: whether there is a job with the largest information gain, so choose it for the next feature.

Through this example, the entire decision tree only uses two features, and their subsets belong to the same category. For those who have a house, they agree to the loan, and for those who do not have a house, they do not agree to the loan. It satisfies a single leaf node, and the decision tree generate.

Generation of decision tree - ID3 algorithm

The origin of the algorithm:

Guess you like