Artificial Intelligence-Machine Learning & Artificial Neural Network

machine learning

The main content of the machine learning part is the naive Bayes algorithm and the decision tree algorithm.

Machine learning studies how computers can simulate human learning behavior, acquire new knowledge or new skills, reorganize existing knowledge to improve their own performance, so that computers have human learning capabilities, thereby realizing artificial intelligence.

Machine learning is often classified as:

• Supervised learning: establishing a mapping from input to output when the input and output are known, often used for classification and regression tasks
• Unsupervised learning: There is no correct output, only input, and the model summarizes the characteristic information of the data by itself. It is often used in clustering tasks.
• Reinforcement learning: The agent chooses an action to act on the environment, and the environment gives a reinforcement signal and a changed state. The agent selects the next action based on the reinforcement signal and the current state of the environment, and chooses an action that increases the probability of receiving positive reinforcement.

Decision tree algorithm

The decision tree algorithm is a commonly used classification algorithm in machine learning. In the decision tree, each attribute is used for classification, and there are many ways to select the attributes for each classification. The simplest is the ID3 algorithm, which uses information gain as the basis for selecting attributes.

• Entropy: Entropy represents the degree of chaos of things. The greater the entropy, the greater the degree of chaos. The smaller the entropy, the smaller the degree of chaos. For a random event S, if there are N possibilities and the probability of each possible occurrence is Pi, then the entropy of the event is:
  $$H(S)=-\sum _{i=1}^{N}pi{\log }_{2}pi$$
• Information gain: Information gain represents the degree to which the information of feature A reduces the uncertainty of the information of class D. It is the average entropy of the class that is not divided by a certain attribute minus the entropy divided by different values ​​of a certain attribute. Defined as:
  $$Gain(D,A)=H(D)-\sum _{i=1}^V\frac{|D_i|}{|D|}H(D_i)$$
  where Di is a sample whose value is Ai on attribute A. There are V kinds of values ​​for attribute A.

When building a decision tree, select the attribute with the largest information gain as the attribute to divide the sample. After dividing the sample once, calculate the information gain of each attribute and continue dividing until all attributes are used or the sample is completely divided.

Question

a) Concept question

b)
Calculate the information gain of each attribute to construct a decision tree:
$$H\left(attraction\right)=-\frac{1}{2}lo{g}_2\frac{1}{2}-\frac{1}{2}lo{g}_2\frac{1}{2}=1$$
There are two temperature values, each accounting for half. The attractive ones account for 2/5 (hot) and 3/5 (cold) respectively.

$$H(attraction|temperature)=\frac{1}{2}(-\frac{2}{5}lo{g}_2\frac{2}{5}-\frac{3}{5}lo{g}_2\frac{3}{5})+\frac{1}{2}(-\frac{2}{5}lo{g}_2\frac{2}{5}-\frac{3}{5}lo{g}_2\frac{3}{5})=0.971$$
There are three flavors in total, sweet, sour and salty 4:3:3. The attractive ones account for 1/2, 1 and 0 respectively.

$$H(attractiveness|taste)=0.4(-0.5lor{g}_{2}0.5-0.5lor{g}_{2}0.5)+0+0=$$
There are two values ​​for$$0.4\; servings,\; each\; accounting\; for\; half.\; The\; attractive\; ones\; account\; for\; 1/5\; (large)\; and\; 4/5\; respectively.$$

$$H(attractiveness|weight)=0.5(-0.2lo{g}_{2}0.2-0.8lo{g}_{2}0.8)+0.5(-0.2lo{g}_{2}0.2-0.8lo{g}_{2}0.8)=0.722$$
can calculate the information gain of each attribute as:

$$\begin{gathered} Gain(attraction,\; temperature)=1-0.971=0.029 \\ Gain(attraction,\; taste)=1-0.4=0.6 \\ Gain(attractiveness,\; weight)=1-0.722=0.278 \end{gathered}$$
Therefore, taste should be selected as the attribute to divide the samples to divide the samples. After the division, it is found that the samples with sour and salty tastes have been divided, and the sweet samples can obviously be divided into two categories by the portion size. Therefore, directly select the portion size and continue dividing, you can get the final decision tree:

c)
By querying the decision tree, the "no" result can be predicted.

Naive Bayes algorithm

The Naive Bayes algorithm is commonly used to implement classifiers. It uses training samples to obtain the probability of having a certain feature in a specific category, and calculates the probability that the sample belongs to a specific category under certain feature conditions. For samples that need to be classified, calculate the probability of belonging to each category according to the characteristics. The probability of each category is selected, and the one with the highest probability is selected as the classification result.

The question
first calculates the probability of taking each feature under each category: (According to the answer, you can only calculate the ones you need to use)

P(C=1) = 9/15
P(C=-1) = 6/15
P(X1=1|C=1) = 2/9
P(X1=2|C=1) = 3/9
P (X1=3|C=1) = 4/9
P(X1=1|C=-1) = 3/6
P(X1=2|C=-1) = 2/6
P(X1=3|C =-1) = 1/6
P(X2=S|C=1) = 1/9
P(X2=P|C=1) = 4/9
P(X2=Q|C=1) = 4/9
P(X2=S|C=-1) = 3/6
P(X2=P|C=-1) = 2/6
P(X2=Q|C=-1) = 1/6

Compute the desired probability:
$$\begin{gathered} P(C=1|X1=3,X2=S)=\alpha P(X1=3,X2=S|C=1)=\alpha P(X1=3|C=1\left)P\right(X\_=S|C=1) \\ =0.0296a \\ P(C=-1|X1=3,X2=S)=\alpha P(X1=3,X2=S|C=-1)=\alpha P(X1=3|C=-1\left)P\right(X\_=S|C=-1) \\ =0.333\alpha \end{gathered}$$
so the classifier predicts the category of this sample to be C=-1.

Here αP(X1=3,X2=S|C=1) can be directly written as αP(X1=3|C=1)P(X2=S|C=1) because the naive Bayes algorithm adopts conditions The independence assumption simplifies by assuming that all features are independent from each other.

Artificial neural networks

The main content of the artificial neural network part is the principle of artificial neural network, perceptron algorithm, multi-layer neural network

perceptron algorithm

The goal of the perceptron algorithm is to minimize the loss function, learn the parameters w and b in the model, and find the separation hyperplane that divides the sample.

The mapping relationship from input to output is as follows:
$$Y=F(w1x1+w2x2-\theta )$$
F is the activation function, which takes a step function. When it is less than 0, it is 0, and when it is greater than 0, it is 1.

When training the perceptron, assume that the training round is i, the learning rate is α, the error E(i) is calculated each time, and the parameters are adjusted:

$$\begin{gathered} E\left(i\right)=d(i)-y(i) \\ {w}_1(i+1)=w1(i)+\alpha E(i)x_1(i) \\ {w}_2(i+1)=w2(i)+\alpha E\left(i\right)x_2\left(i\right) \\ \theta (i+1)=\theta (i)+\alpha E\left(i\right)(-1) \end{gathered}$$
where d(i) is the ideal output of the i-th round, and the parameter adjustment part is actually: the learning rate x the inverse of the gradient of the loss function on the parameters, and the loss function on the parameters The gradient of is obtained using the chain derivation rule as:
$$\frac{\partial L}{\partial w_1}=\frac{\partial L}{\partial O}\times \frac{\partial O}{\partial y}\times \frac{\partial y}{\partial w_1}$$
The loss function is usually taken as:
$$\begin{gathered} LOSS=\frac{1}{2}(Correct\; output-actual\; output{)}^2 \\ Seek\; the\; derivative\frac{\partial L}{\partial O}=-(correct\; output-Actual\; output) \end{gathered}$$
and the step function cannot be differentiated, so in the perceptron algorithm, the step function is not multiplied to derivation of the weighted result, so only:
$$\begin{gathered} y=w1x1+w2x2-i \\ Seek\; the\; derivative\frac{\partial y}{\partial w_1}=x_1 \end{gathered}$$
Therefore, the final adjustment parameter is calculated as
$$w_1(i+1)=w1(i)+a\times -(-(Correct\; output-actual\; output)\times x_1(i))=w1\left(i\right)+\alpha E(i)x_1\left(i\right)$$
What is multiplied after α is the opposite direction of the gradient, so it finally becomes multiplied by E(i)x1(i). The adjustment method for finding θ is similar.

Artificial neural networks

The core of this part of the content is the principle of gradient descent and the calculation of error backpropagation. In fact, it is explained more clearly in the PPT part of deep learning. Mainly learn to find the formula for error back propagation.

Question



                 (2)