Sichuan University Software College｜Data Mining Course｜Final Review

basic concept

data mining

The process of exploring useful patterns or knowledge from data sources.

machine learning

Machine learning is the study of computer algorithms that can automatically improve through experience, using data or past experience to optimize the performance standards of computer programs.

Supervised learning & unsupervised learning

Supervised learning derives predictive functions from labeled training data. Labeled training data means that each training instance includes input and desired output.

Given data, predict labels

Unsupervised learning infers conclusions from unlabeled training data. The most typical unsupervised learning is cluster analysis, which can be used to discover hidden patterns or group data during the exploratory data analysis stage.

Given data, find hidden structures/features

Classification & Regression

Regression problem: predict a value

Such as: predicting house prices, future weather conditions, etc.

Classification problem: label something , usually the result is a discrete value

For example: Determine whether the animal in a picture is a cat or a dog

Classification is usually based on regression , and the last layer of classification usually uses the softmax function to determine the category it belongs to.

Association rules, sequence patterns

Association Rule: an important type of rules contained in data

Sequence pattern: Given a set of different sequences, each sequence consists of different elements ordered in order, each element consists of different items

Support of association rules and support of sequence patterns

Rule $X\to The\; support\; of\; Y$ refers to$T$ contains$X\cup Percentage\; of\; transactions\; in\; Y$ :
$$Support=\frac{(X\cup Y).count}{n}$$

Sequence pattern support: the proportion of data containing this sequence in the sequence to the total data

Confidence of association rules

Rule $X\to The\; confidence\; level\; of\; Y$ means that it contains$X$ also contains$The\; number\; of\; transactions\; of\; Y$ accounts for all transactions that containXXPercentage of transactions in$X$
$$Confidence=\frac{(X\cup Y).count}{X.count}$$

Classifier accuracy

$$Accuracy=\frac{correct}{total}$$

Classifier precision & recall

Mixing matrix

	classified as positive	Classify as negative example
Actually a positive example	TP	FN
Actually a negative example	FP	TN

Precision

Precision rate = number of correctly classified positive examples / number of data classified as positive examples
$$p=\frac{TP}{TP+FP}$$

Recall

Recall rate = number of correctly classified positive examples / actual number of positive examples
$$r=\frac{TP}{TP+FN}$$

F-score

F-score is the precision rate ( $p$ ) and recall rate ($r$ )harmonic mean

Improvements in recall are often achieved at the expense of precision, and vice versa; therefore, F-score needs to be used

If you want to make the F-score very high, $p$ and$r$ needs to be very high

$$F=\frac{2}{\frac{1}{p}+\frac{1}{r}}=\frac{2pr}{p+r}$$

Overfitting & Underfitting

overfitting

• The model complexity is higher than the actual problem. The model performs well on the training set but performs poorly on the test set.
• The model does not understand the patterns behind the data and has poor generalization ability.

Prevent overfitting

• Parametric regularization methods that provide regularization constraints , such as L1/L2 regularization
• Methods to achieve lower generalization error through engineering techniques, such as early stopping ( Early stopping ) and Dropout
• Implicit regularization methods that do not directly provide constraints , such as data enhancement , controlling model complexity , reducing the number of features , etc.

Underfitting

• The model cannot achieve low enough error on the training set
• The model has low complexity and performs very poorly on the training set, unable to learn the patterns behind the data.

K-fold cross validation

Data preparation: split the data set into $K$ disjoint equally large data subsets

K iterations :

1. Test set: $K$ subset
2. Training set: remaining $K-1$ subset of

Ensemble learning

Supervised learning algorithms are often described as performing the task of searching a hypothesis space to find a suitable hypothesis that will make good predictions for a specific problem. Even if the hypothesis space contains hypotheses that are well suited to a particular problem, it can be difficult to find a good hypothesis. Ensemble learning combines multiple hypotheses to form a (hopefully) better hypothesis.

clustering

Clustering is to divide a data set into different classes or clusters according to a certain criterion (such as distance) , so that the similarity of data objects in the same cluster is as large as possible, while data that are not in the same cluster The diversity of objects is also as great as possible.

Basic principles issues

distance measure

Minkowski distance

$$dist(\overrightarrow{x_i},\overrightarrow{x_j})=(|\overrightarrow{x_i}-\overrightarrow{x_j}{|}^h+|\overrightarrow{x_{i2}}-\overrightarrow{x_j}{|}^h+\cdots +|\overrightarrow{x_i}-\overrightarrow{x_{jr}}{|}^h{)}^{\frac{1}{h}}$$

Euclidean distance

$$dist(\overrightarrow{x_i},\overrightarrow{x_j})=\sqrt{(\overrightarrow{x_i}-\overrightarrow{x_j}{)}^2+(\overrightarrow{x_{i2}}-\overrightarrow{x_j}{)}^2+\cdots +(\overrightarrow{x_i}-\overrightarrow{x_{jr}}{)}^2}$$

manhattan distance

$$dist(\overrightarrow{x_i},\overrightarrow{x_j})=|\overrightarrow{x_i}-\overrightarrow{x_j}|+|\overrightarrow{x_{i2}}-\overrightarrow{x_j}|+\cdots +|\overrightarrow{x_i}-\overrightarrow{x_{jr}}|$$

Weighted Euclidean distance

Each attribute has a weight that represents its importance relative to other attributes
$$dist(\overrightarrow{x_i},\overrightarrow{x_j})=\sqrt{w_1(\overrightarrow{x_i}-\overrightarrow{x_j}{)}^2+w_2(\overrightarrow{x_{i2}}-\overrightarrow{x_j}{)}^2+\cdots +w_r(\overrightarrow{x_i}-\overrightarrow{x_{jr}}{)}^2}$$

Squared Euclidean distance

Increase the weight of data points that are far away
$$dist(\overrightarrow{x_i},\overrightarrow{x_j})=(\overrightarrow{x_i}-\overrightarrow{x_j}{)}^2+(\overrightarrow{x_{i2}}-\overrightarrow{x_j}{)}^2+\cdots +(\overrightarrow{x_i}-\overrightarrow{x_{jr}}{)}^2$$

Chebyshev distance

The Chebyshev distance definition is appropriate when it is necessary to define that two data points are "different" if they have a different attribute value
$$dist(\overrightarrow{x_i},\overrightarrow{x_j})=max(|\overrightarrow{x_i}-\overrightarrow{x_j}|,|\overrightarrow{x_{i2}}-\overrightarrow{x_j}|,\cdots ,|\overrightarrow{x_i}-\overrightarrow{x_{jr}}|)$$

similarity measure

cosine similarity

By calculating the cosine of the angle between two vectors:
$$cos\theta =\frac{\vec{x}\cdot \vec{y}}{||\vec{x}||\times ||\vec{y}||}$$
The value range of cosine similarity is $[-1,1]$ , the larger the value, the more similar it is.

Jaccard

The Jaccard index is used to measure the similarity in a sample set.

Suppose two data points $\vec{x}$与$\vec{y}$, each of their attributes is a Boolean attribute, then:

• order $a$ is two data points with the same attribute value1 1Number of attributes of$1$
• Let $b$ is xi ⃗ = 1 \vec {x_i}=1among the two data points$\overrightarrow{x_i}=1$ 且 $\overrightarrow{y_i}=$Number of attributes of$0$
• Let $c$ is xi ⃗ = 0 \vec {x_i}=0among the two data points$\overrightarrow{x_i}=0$ 且 $\overrightarrow{y_i}=$Number of attributes of$1$
• Let $d$ is two data points with the same attribute value0 0Number of attributes of$0$

Then the Jaccard distance is:
$$dist(\vec{x},\vec{y})=\frac{b+c}{a+b+c}$$

Association rule mining - Apriori

Close down property

If an itemset satisfies a certain minimum support requirement, then any non-empty subset of the itemset must satisfy the minimum support .

Rule sequence coverage, association rules are used for classification

decision tree

Basic idea

Each internal node of the decision tree represents a test of a certain attribute, each edge represents a test result, and the leaf nodes represent a certain class or the distribution of the class.

The decision-making process of the decision tree needs to start from the root node of the decision tree. The data to be tested is compared with the feature nodes in the decision tree, and the next comparison branch is selected according to the comparison results until the leaf node is used as the final decision result.

Termination condition

1. The samples contained in the current node all belong to the same category and do not need to be divided

2. The current attribute set is empty, or all samples have the same value on all attributes and cannot be divided.

   Mark the current node as a leaf node, and the category of the node contains the most samples.

3. The sample set contained in the current node is empty and cannot be divided.

   Mark the current node as a leaf node, and the category is the category with the most samples in the parent node.

Reasons for pruning

The purpose of pruning is to avoid overfitting of the decision tree model .

Because the decision tree algorithm constantly divides the nodes in order to classify the training samples as accurately as possible during the learning process, this will lead to too many branches in the entire tree (that is, the model is too complex), which will lead to Overfitting .

pruning method

pre-pruning

Pre-pruning : In the process of constructing a decision tree, each node is estimated before partitioning. If the partitioning of the current node cannot improve the generalization performance of the decision tree model, the current node will not be evaluated. The node is divided and the current node is marked as a leaf node.

post-pruning

Post-pruning : First construct the entire decision tree , and then examine the non-leaf nodes from the bottom up . If the subtree corresponding to the node is replaced with a leaf node, it can bring about generalization. To improve performance, replace the subtree with a leaf node.

How to deal with continuous attributes

dichotomy

information gain

Entropy: used to describe "the degree of chaos in a system"

The higher the information entropy of a system, the more disordered it is; the lower the information entropy, the more orderly it is; the higher the information entropy, the greater the amount of information consumed to make it orderly.

Suppose variable x ⃗ = { x 1 , x 2 , ⋯ , xi , ⋯ , xn } \vec x = \{x_1, x_2, \cdots, x_i, \cdots, x_n \}x ={ x1​,x2​,⋯,xi​,⋯,xn​} , where the probability corresponding to each element isp ⃗ = { p 1 , p 2 , ⋯ , pi , ⋯ , pn } \vec p = \{p_1, p_2, \cdots, p_i, \cdots, p_n \}p ​={ p1​,p2​,⋯,pi​,⋯,pn​} , then the corresponding entropy calculation formula is as follows:

$$entropy(\vec{x})=-\sum _{i=1}^n{p}_{i}lo{g}_2(p_i)$$

Information gain (Info-Gain) refers to the reduction in entropy (information gain measures the reduction in information confusion ):

$$Gain(S,A)=entropy(S)-entropy(A)$$

Among them, $SwaAA$ _$A$ represents the data division status before and after the operation respectively.

advantage

• Perform better when facing discrete data with smaller categories
• The generated classification rules are easy to understand and have high accuracy.

shortcoming

• Prefer features with more values
• In the process of constructing the tree, the data set needs to be scanned and sorted multiple times, resulting in inefficiency of the algorithm.
• It is only suitable for data sets that can reside in memory. When the training set is too large to be accommodated in memory, the program cannot run.

Naive Bayes

Basic idea

For a given item to be classified, find out the probability of each category appearing under the condition that this item appears. Whichever one is the largest is considered to be the category to which the item to be classified belongs.

$$\begin{gathered} Pr(C=c_j|A_1=a_1,A_2=a_2,\cdots ,A_{|A|}=a_{|A|}) \\ =\frac{Pr(C=c_j)Pr(A{}_1=a_1,A_2=a_2,\cdots ,A_{|A|}=a_{|A|}|C=c_j)}{Pr(A_1=a_1,A_2=a_2,\cdots ,A_{|A|}=a_{|A|})} \\ =\frac{Pr(C=c_j)Pr(A{}_1=a_1,A_2=a_2,\cdots ,A_{|A|}=a_{|A|}|C=c_j)}{\sum _{k=1}^{|C|}Pr(C=c_k)Pr(A{}_1=a_1,A_2=a_2,\cdots ,A_{|A|}=a_{|A|}|C=c_k)} \end{gathered}$$

According to the conditional independence assumption , we get:

$$Pr(A_1=a_1,A_2=a_2,\cdots ,A_{|A|}=a_{|A|}|C=c_j)=\prod _{i=1}^{|A|}Pr(A_i=a_i|C=c_j)$$

Furthermore, the core formula of the Naive Bayes classification algorithm is obtained :

$$Pr(C=c_j|A_1=a_1,A_2=a_2,\cdots ,A_{|A|})=\frac{Pr(C=c_j)\prod _{i=1}^{|A|}Pr(A_i=a_i|C=c_j)}{\sum _{k=1}^{|C|}Pr(C=c_k)\prod _{i=1}^{|A|}Pr(A_i=a_i|C=c_k)}$$

Then, the prior probability P r ( C = cj ) {\rm Pr}(C=c_j) needs to be estimated from the training data$Pr(C=c_j)$ and conditional probability$Pr(A_i=a_i|C=c_j)$ :
$$\begin{gathered} Pr(C=c_j)=\frac{Belongs\; to\; categoryc{}_j}{The\; total\; number\; of\; examples\; in\; the\; dataset} \\ Pr(A_i=a_i|C=c_j)=\frac{A_i=a_{i}and\; belongs\; to\; categoryc{}_j}{Belongs\; to\; categoryc{}_{j}total\; number\; of\; samples} \end{gathered}$$
Finally, given a test example, determine the most likely class by computing:
$$c=argma{x}_{c_j}Pr(C=c_j)\prod _{i=1}^{|A|}Pr(A_i=a_i|C=c_j)$$

independence assumption

" Data samples are independent and identically distributed " refers to the independence between samples , and the " feature conditional independence assumption " refers to the independence between the internal features of each sample . The two independence assumptions make the model simple and the calculation simple. , although it sacrifices some accuracy, it is worth it in certain usage scenarios.

Data samples are independently and identically distributed

Independent and identical distribution between data samples means that there is no dependency relationship between each data sample point , and there is no temporal relationship (or the temporal relationship is not important), and they are obtained from the same distribution through multiple samplings .

If it is not the same distribution but is generated by multiple distributions, it is a mixture model, typically such as a Gaussian mixture model; if it is not independent and there is some relationship between the samples, then this dependence needs to be modeled into the model , such as weather conditions for several days can be modeled using Markov networks.

characteristic conditional independence assumption

Assume a sample $x$ has$n$ features, the feature conditional independence hypothesis means thatthese features are independent under a specific category.

estimate the probability of producing zero

Add a small sample correction to all probabilities: let $n_{ij}$Same time $A_i=a_i$and $C=c_j$The number of samples, let $n_j$For the training data $C=c_j$The total number of data, then the corrected estimate is:
$$Pr(A_i=a_i|C=c_j)=\frac{n_{ij}+}{n_j+\lambda m_i}$$
Itinakami $m_i$Yes AAThe total number of possible values ​​of$A$$\lambda$ is a factor, generally set to$l=1/n$（$n$ is the total number of training data), when$l=When\; 1$ , the Laplace continuation rate is obtained.

Polynomial Naive Bayes - Text Classification

Support vector machine (SVM)

Suppose there is a hyperplane $\vec{W}\vec{x}+b=0$ , where$\vec{W}$is a normal vector, then point $\overrightarrow{x_i}$To hyperplane $\vec{W}\vec{x}+b=The\; vertical\; Euclidean\; distance\; of\; 0$ is:
$$\frac{|\vec{W}\overrightarrow{x_i}+b|}{||\vec{W}||}$$
where $||\vec{W}||$是$\vec{W}$Euclidean norm of .

SVM maximizes the margin between positive and negative examples $\to$ Improve model robustness

For positive examples, the hyperplane $H_{+}$ 为 $\vec{W}\vec{x}+b=1$ ; For negative examples, the hyperplane$H_{-}$ 为 $\vec{W}\vec{x}+b=-1$。

Support vector: such that $y_i(\vec{W}{\vec{x}}_i+b)-1=$several points of$0$

Define $d_{+}$with $d_{-}$is the hyperplane $\vec{W}\vec{x}+b=The\; nearest\; distance\; between\; 0$ and the positive and negative examples is$d_{+}=d_{-}=\frac{1}{||\vec{W}||}$, that is, $margin=d_{+}+d_{-}=\frac{2}{||\vec{W}||}$。

Maximize margin $margin$ $<=>$ Minimize∣$||\vec{W}|{|}^2/2$ $<=>$ Minimize${\vec{W}}^T\vec{W}/2$

And it needs to satisfy $y_i(\vec{W}\vec{x}+b)\ge 1,i=1,2,\cdots ,n$ ：
$$\begin{gathered} \vec{W}\vec{x}+b\ge 1,\; wheny{}_i=1 \\ \vec{W}\vec{x}+b\le -1,\; wheny{}_i=-1 \end{gathered}$$
, so it can be solved using the standardLagrange multipliermethod, that is:
$$L_p=\frac{1}{2}||\vec{W}||-\sum _{i=1}^n{a}_i[y_i(\vec{W}{\vec{x}}_i+b)-1]$$
where,$a_i\ge 0$ is the Lagrange multiplier.

Dual problem calculation

$$L=\frac{1}{2}\sum _{i=1}^N\sum _{j=1}^N{a}_i{a}_j{y}_i{y}_j(x_i\cdot x_j)-\sum _{i=1}^N{a}_i$$

$$\begin{gathered} {\vec{w}}^{\ast }=\sum _{i=1}^N{a}_i^{\ast }{y}_i\overrightarrow{x_i} \\ {b}^{\ast }=y_j-\sum _{i=1}^N{a}_i^{\ast }{y}_i(\overrightarrow{x_i}\cdot \overrightarrow{x_j}) \end{gathered}$$

Kernel function

Simplify calculations

KNN

When predicting a new value xxWhen$x$ , according to the nearest$What\; categories\; are\; the\; K$ pointsto judgexx?Which category$x\; belongs\; to.$

advantage

1. Simple and easy to use
2. Model training time is fast (KNN algorithm is lazy )
3. Good prediction effect
4. Not sensitive to outliers

shortcoming

1. High memory requirements (because the algorithm stores all training data )
2. The prediction phase can be slow
3. Sensitive to irrelevant features and data size

K-means clustering

Given sample set D = { x 1 ⃗ , x 2 ⃗ , ⋯ , xm ⃗ } D=\{\vec {x_1}, \vec {x_2}, \cdots, \vec {x_m} \}D={ x1​ ​,x2​ ​,⋯,xm​ ​}，“$The\; k$ -means algorithm divides the clusters obtained by clustering intoC = { C 1 , C 2 , ⋯ , C k } C = \{C_1, C_2, \cdots, C_k \}C={ C1​,C2​,⋯,Ck​} Minimize the squared error:
$$E=\sum _{i=1}^k\sum _{\vec{x}\in C_i}||\vec{x}-\overrightarrow{m_i}|{|}_2^2$$
其中 $\overrightarrow{m_i}=\frac{1}{|C_i|}{\sum }_{\vec{x}\in C_i}\vec{x}$is cluster $C_i$mean vector .

选择 k 个“初始均值向量中心”
repeat
	1. 将数据集中每个数据分配到距离最近的“均值向量中心”（一般使用欧氏距离作度量）
	2. 重新计算每个簇的“均值向量中心”
until 没有数据点被重新分配给不同的聚类 ｜ 没有聚类中心再发生变化 ｜ 平方误差局部最小

advantage

1. concise
2. Efficient

shortcoming

1. Applied to data sets where "the mean can be defined" is difficult to apply to categorical data
2. The user needs to specify the number of clusters $k$
3. Very sensitive to outliers

hierarchical clustering

Merged (bottom-up) clustering

• Start: the lowest level of the tree diagram
• Process: Each time clusters in the previous layer are formed by merging the most similar (closest) clusters
• Termination: Stop when all data points are merged into one cluster (root node clustering)

Determine the similarity between the data points of each category by calculating the distance between them (the smaller the distance, the higher the similarity), and combine the two closest data points or categories to generate Clustering tree.

Split (top-down) clustering

• Start: a cluster (root) containing all data points
• Process: Split the root node cluster into some sub-clusters, and each sub-cluster continues to split recursively.
• Termination: Include only one data point in each cluster

Hierarchical clustering method

1. single link method
   1. Distance between two clusters: the distance between the two closest data points in the two clusters
   2. Each step merges those nearest elements with the smallest distance, i.e. the two clusters with the shortest closest data points
   3. Sensitive to noisy data : may cause chain reactions to form long chains
2. Full link method
   1. Distance between two clusters: the maximum value of the pairwise distance between all data points in the two clusters
   2. Each step merges those farthest elements with the shortest distance, i.e. the two clusters with the shortest and farthest data points
   3. sensitive to outliers
3. average link method
   1. A compromise between the sensitivity of full-link protection to outliers and the tendency of single-link methods to form long chains.
   2. Distance between two clusters: the average of the sum of the distances between pairs of data points in the two clusters

Neural Networks

Ensemble classifier

Bagging

Bagging uses bagging sampling to obtain data subsets to train the basic learner.

Usually classification tasks are integrated by voting, while regression tasks are integrated by averaging.

The variance of the results obtained after bagging is smaller .

Specific process

1. A training set is extracted from the original sample set . In each round, n training samples are extracted from the original sample set using the Bootstraping method (in the training set, some samples may be extracted multiple times, while some samples may not be selected once). A total of k rounds of extraction are performed to obtain k Training set (k training sets are independent of each other).
2. Each time a training set is used to obtain a model, k training sets are used to obtain a total of k models . (Note: There is no specific classification algorithm or regression method here. We can use different classification or regression methods according to specific problems, such as decision trees, perceptrons, etc.)
3. Get the result (all models have the same importance)
   1. For classification problems: use the k models obtained in the previous step to vote to obtain the classification results.
   2. For regression problems, calculate the mean of the above model as the final result

Boosting

Boosting refers to converting weak learners into strong learners through a collection of algorithms.

The training method is to use weighted data . Give greater weight to misclassified data in the early stages of training.

The bias of the results obtained after boosting is smaller .

Specific process

1. Linearly combine the base models through an additive model .
2. Each round of training increases the weights of basic models with low error rates , while reducing the weights of models with high error rates .
3. Change the weight or probability distribution of the training data in each round, by increasing the weight of those "misclassified samples" by the weak classifier in the previous round and reducing the weight of the "paired samples" in the previous round. To make the classifier have better effect on misclassified data.

The difference between Bagging and Boosting

Sample selection

Bagging : The training set is selected with replacement from the original set , and each training set selected from the original set is independent.

Boosting : The training set in each round remains unchanged , but the weight of each sample in the training set in the classifier changes. The weights are adjusted based on the classification results of the previous round.

Sample weight

Bagging : Use uniform sampling, with equal weight for each sample

Boosting : Continuously adjust the weight of the sample according to the error rate . The greater the error rate, the greater the weight.

prediction function

Bagging : All prediction functions are equally weighted .

Boosting : Each weak classifier has a corresponding weight , and classifiers with small classification errors will have greater weights .

parallel computing

Bagging : Each prediction function can be generated in parallel

Boosting : Each prediction function can only be generated sequentially , because the latter model parameters require the results of the previous round of models.

semi-supervised learning

In many practical problems, labeled samples and unlabeled samples often exist at the same time, and there are more unlabeled samples, while there are relatively few labeled samples.

Semi-supervised learning: Under the guidance of a small number of sample labels, it can make full use of a large number of unlabeled samples to improve learning performance , avoid the waste of data resources, and at the same time solve the problem of weak generalization ability of supervised learning methods and lack of samples when there are few labeled samples. The problem of inaccurate unsupervised learning methods when guided by labels.

Semi-supervised algorithms can be divided into two categories: direct inference and inductive.

Direct push semi-supervision only contains labeled sample sets and test sample sets, and the test samples are also unlabeled samples . The direct push semi-supervised algorithm first treats test samples as unlabeled samples, then uses labeled samples and unlabeled samples to train the model, and predicts unlabeled samples during the training process. Therefore, the direct push semi-supervised algorithm can only process the current unlabeled samples (test samples) and cannot directly expand out of the sample. For new test samples, the direct-inference semi-supervised algorithm needs to retrain the model to predict its label.

Inductive semi-supervised algorithms use independent test sample sets in addition to labeled and unlabeled sample sets . Inductive semi-supervised algorithms are able to process samples in the entire sample space. Inductive semi-supervised algorithms train learning models on labeled and unlabeled samples. This model can not only predict the labels of training unlabeled samples, but also directly predict the labels of new test samples.