[Machine Learning] Decision Tree Algorithm Attribute Screening Metrics

Decision Tree Algorithm Attribute Filtering Metrics

In information theory , entropy is the average amount of information contained in each received message, also known as information entropy , source entropy , and average self-information . Here, "messages" represent events, samples, or features from a distribution or data stream. The concept of entropy originated in physics and is used to measure the degree of disorder of a thermodynamic system. In information theory , entropy is a measure of uncertainty. In the information world, the higher the entropy, the more information can be transmitted, and the lower the entropy, the less information can be transmitted.

information entropy

In 1948, Shannon introduced the concept of entropy in thermodynamics to measure the amount of information. Entropy is the sum of the entropy of various possibilities with probability as the weight , each possibility can be used $\log \frac{1}{p}=-\log p$ to represent. For information$x$ , the amount of information it contains is
$$\mathrm{I}(x)=\log \frac{1}{p(x)}=-\log p\left(x\right)$$
where$p\left(x\right)$ is information$The\; probability\; of\; x$ happening.

Information entropy is the mathematical expectation of the amount of information , and it is the prior uncertainty before the source sends information, also known as prior entropy. For random variable $X$，$x_1,x_2,\cdots ,x_n$For a simple sample from the population, its probability density distribution is
$$\mathrm{P}(X=x_i)=p_i$$
Sufficient
$$\sum _{i=1}^n{p}_i=1$$ XX
_$X$ 的熵定义为
$$\mathrm{H}(X)={\mathrm{I}}_E(x_1,x_2,\cdots ,x_n)=\sum _{i=1}^n{p}_{i}I(x_i)=-\sum _{i=1}^n{p}_i\log p_i$$
Suppose there are sources $x_1,x_2$, the figure below is the information entropy $H$ with source$x_1$The probability of occurrence $P(x_1)$ , when$P\left(x_1\right)=1$ or$P(x_2)=0$ , then the information entropy$\mathrm{H}=0$ means no uncertainty is sent. And when$P\left(x_1\right)=When\; the\; value\; is\; 0.5,the$ entropy reaches the maximum and the uncertainty is the maximum.

For example, if in a classification system, the category identifier is $c$ , the values ​​are$c_1,c_2,\cdots ,c_n$，$n$ is the total number of categories. Then the entropy of the classification system is:
$$\mathrm{H}(c)=-\sum _{i=1}^{n}p(c_i)\log p(c_i)$$
In particular, if it is a binary classification system, then the entropy of this system is:
$$\mathrm{H}(c)=-p(c_0){\log }_{2}p\left(c_0\right)-p\left(c_1\right)\log p\left(c_1\right)$$
where$p(c_0),p\left(c_1\right)$ are the probabilities of occurrence of the two types of samples, respectively.

information gain

The generation of the decision tree algorithm ID3 uses information gain . Information gain is based on information entropy and self-information theory in information theory , and is defined as IG ( T , a ) ⏞ Information Gain = H ( T ) ⏞ Entropy (parent) − H ( T ∣ a ) ⏞ Weighted Sum of Entropy (Children) = − ∑ i = 1 J pi log ⁡ 2 pi − ∑ ap ( a ) ∑ i = 1 J − Pr ⁡ ( i ∣ a ) log ⁡ 2 Pr ⁡ ( i ∣ a ) \begin{aligned} \overbrace {\ mathrm{IG}(T,a)} ^{\text{Information Gain}} &=\overbrace {\mathrm {H} (T)} ^{\text{Entropy (parent)}}-\overbrace {\mathrm {H} (T|a)} ^{\text{Weighted Sum of Entropy (Children)}}\\ &=-\sum _{i=1}^{J}p_{i}\log _{2} {p_{i}}-\sum _{a}{p(a)\sum _{i=1}^{J}-\Pr(i|a)\log _{2}{\Pr(i| a)}} \end{aligned}
$$\begin{array}{rl}\stackrel{\text{Information Gain}}{\overbrace{IG(T,a)}}& =\stackrel{\text{Entropy (parent)}}{\overbrace{\mathrm{H}(T)}}-\stackrel{\text{Weighted Sum of Entropy (Children)}}{\overbrace{\mathrm{H}(T|a)}}\\ & =-\sum _{i=1}^J{p}_i{\log }_2{p}_i-\sum _{a}p(a)\sum _{i=1}^J-\mathrm{Pr}(i|a){\log }_2\mathrm{Pr}(i|a)\end{array}$$
Take a look at an example given in Wikipedia:

Example The dataset has 4 attributes: outlook (sunny, overcast, rainy), temperature (hot, mild, cool), humidity (high, normal), and windy (true, false), target value play (yes, no), A total of 14 data points.

Outlook	Temperature	Humidity	Windy	Play?
sunny	hot	high	false	no
sunny	hot	high	true	no
overcast	hot	high	false	yes
rainy	mild	high	false	yes
rainy	cool	normal	false	yes
rainy	cool	normal	true	no
overcast	cool	normal	true	yes
sunny	mild	high	false	no
sunny	cool	normal	false	yes
rainy	mild	normal	false	yes
sunny	mild	normal	true	yes
overcast	mild	high	true	yes
overcast	hot	normal	false	yes
rainy	mild	high	true	no

In order to build a decision tree, it is necessary to compare the information gains of four decision trees, and each decision tree is divided by an attribute. The partition with the highest information gain is taken as the first partition, and this process is continued at each child node until its information gain is 0.

If the attribute windy is used for division, two child nodes are generated: the value of windy is true and false. In the current data set, the windy value of 6 data points is true, the play value of 3 points is true, and the play value of 3 points is false; the windy value of the remaining 8 data points is false, and the play value of 6 points is false is true, and the play value of the 2 points is false.

• The information entropy of the child node of windy =true is calculated as:

$${\mathrm{I}}_E([3,3])=-\frac{3}{6}{\log }_2^{}\frac{3}{6}-\frac{3}{6}{\log }_2^{}\frac{3}{6}=-\frac{1}{2}{\log }_2^{}\frac{1}{2}-\frac{1}{2}{\log }_2^{}\frac{1}{2}=1$$

• The information entropy of the child node of windy =false is calculated as:

$$I_E([6,2])=-\frac{6}{8}{\log }_2^{}\frac{6}{8}-\frac{2}{8}{\log }_2^{}\frac{2}{8}=-\frac{3}{4}{\log }_2^{}\frac{3}{4}-\frac{1}{4}{\log }_2^{}\frac{1}{4}=0.8112781$$

The information entropy of this partition (using the attribute windy ) is the weighted sum of the information entropies of the two child nodes:
$${\mathrm{I}}_E([3,3],[6,2\left]\right)=I_E(\text{windy or not})=\frac{6}{14}\cdot 1+\frac{8}{14}\cdot 0.8112781=0.8921589$$
In order to calculate the information gain using the attributewindy, the information entropy of the initial (undivided) data set must be calculated first. Theplayhas 9 yes and 5 no:
$${\mathrm{I}}_E([9,5])=-\frac{9}{14}{\log }_2^{}\frac{9}{14}-\frac{5}{14}{\log }_2\frac{5}{14}=0.940286$$
Thus,the information gainwindy
$$\mathrm{I}\mathrm{G}(\text{windy})={\mathrm{I}}_E([9,5])-I_E([3,3],[6,2])=0.940286-0.8921589=0.0481271$$
The above is just an example of the information gain if divided according to Windy, but what if it is divided according to Temperature, Humidity and Outlook? Similarly,
$$\begin{array}{r}\mathrm{I}\mathrm{G}(\mathrm{o}\mathrm{u}\mathrm{t}\mathrm{l}\mathrm{o}\mathrm{o}\mathrm{k})=0.246\\ \mathrm{I}\mathrm{G}(\mathrm{T}\mathrm{e}\mathrm{m}\mathrm{p}\mathrm{e}\mathrm{r}\mathrm{a}\mathrm{t}\mathrm{u}\mathrm{r}\mathrm{e})=0.029\\ \mathrm{I}\mathrm{G}(\mathrm{H}\mathrm{u}\mathrm{m}\mathrm{i}\mathrm{d}\mathrm{i}\mathrm{t}\mathrm{y})=0.151\end{array}$$
Select the attribute with the largest information gain to divide, and then repeat the above steps until a tree is built; in this example, the attribute of the first branch node is Outlook.

Note : attributes with more values ​​tend to make the data more "pure", and its information gain is greater. The decision tree will first select this attribute as the top/node of the tree; the resulting trained shape is a huge tree with a very shallow depth, which is extremely unreasonable.

information gain rate

Since the information gain prefers attributes with many values ​​(the limit is close to the uniform distribution), the decision tree C4.5 algorithm uses the gain rate to replace the information gain of the ID3 algorithm. The information gain rate is defined as
$$\mathrm{I}{\mathrm{G}}_{\mathrm{r}}=\frac{IG(T,a)}{\mathrm{H}(a)}$$
aa $a$ is the intrinsic property of$IG(T,a)$ is according to attributeaaInformation gain when$a$$H\left(a\right)$ is according to attribute$a$ Divided information entropy.

Therefore, when divided by attribute Outlook, the information entropy is
$${\mathrm{I}}_E([5,4,5])=-\frac{5}{14}{\log }_2^{}\frac{5}{14}-\frac{4}{14}{\log }_2^{}\frac{4}{14}-\frac{5}{14}{\log }_2^{}\frac{5}{14}=1.5774$$
Information gain rate
$$\mathrm{I}{\mathrm{G}}_{\mathrm{r}}(\mathrm{o}\mathrm{u}\mathrm{t}\mathrm{l}\mathrm{o}\mathrm{o}\mathrm{k})=\frac{0.246}{1.5774}=0.15595$$

Gini Coefficient

But both ID3 and C4.5 are based on the entropy model of information theory, which involves a lot of logarithmic operations (in order to avoid this problem). Can the model be simplified without completely losing the benefits of the entropy model? have! The CART (Classification and Regression Tree) classification tree algorithm uses the Gini coefficient instead of the information gain ratio. The Gini coefficient represents the impurity of the model . The smaller the Gini coefficient, the lower the impurity (higher purity) and the better the characteristics.

Similarly, for $n$ category$c_1,c_2,\cdots ,c_n$Sample $T$ ,iiThe probability of occurrence of$i$$p_k$,
$$Gini\left(T\right)=\sum _{i=1}^n{p}_i(1-p_i)=1-\sum _{i=1}^n{p}_i^2$$
In particular, for the binary classification problem, the Gini coefficient is
$$Gini\left(T\right)=\sum _{i=1}^2{p}_i(1-p_i)=1-\sum _{i=1}^2{p}_i^2=2p(1-p)$$
Gini impurity is zero when all samples in a node are of one class.

If feature $a$ has$k$ categories, the Gini coefficient is
$$Gini(T,a)=\sum _{i=1}^k\frac{|T^i|}{|T|}Gini\left(T^i\right)$$
In feature selection,with the smallestas the priority attribute.
$$a^{\ast }={\mathrm{argmin}}_{a\in A}Gini(T,a)$$

References

[1] Super detailed information entropy, information gain, information gain ratio, Gini coefficient

[2] Information Gain_Information Gain Rate_Gini, xiaoxiyouran

[3] Decision tree learning, Wikipedia