Comparing Classification Models

1. Decision tree - decision tree

1. Definition

Classification and prediction methods, supervised learning algorithms, based on dendrograms, the output results are simple and practical rules. is a series of if-then statements.

2. Solve the problem

classification, regression.

3. Principle

Is a greedy algorithm, that is, performs recursive binary segmentation on the feature space, and the decision tree is composed of nodes and directed edges.

• Internal Node: A feature or attribute.
• Leaf node: a category.

4. Advantages

• Readable, easy to understand and explain. (the structure of the tree has visualization);
• The data required for training is small, the usage cost is exponentially distributed, and the classification speed is fast;
• Ease of evaluating models through static testing;
• Can handle multiple output issues.

5. Disadvantages

• It is easy to produce overly complex models and overfitting;
• Not good at predicting numerical results, difficult to deal with missing data;
• Unstable (achievable by ensemble of decision trees);
• Overfitting problems can arise.

6. Adopt the principle

Principle of loss function minimization.

2. Perceptron model

1. Definition

Let the feature space be and the output space be y = {+1,-1}.

The output is a point in the feature space; the output is the category of the instance.

is a linear classifier.

2. Solve the problem

Binary classification, linearly separable problems.

3. Advantages

The model is simple and easy to implement.

4. Disadvantages 

• Cannot handle linearly inseparable training data perfectly;
• The final iterative algebra is greatly affected by the resulting hyperplane and the data of the training set;
• The goal of the loss function is to reduce all misclassified points and the hyperplane, and it is very likely that some sample points will be very close to the hyperplane in the end. To some extent, the classification effect is not particularly good (support vector machine solves ).

3. Neural network (multi-layer functional neurons)

1. Definition

It is an algorithmic mathematical model that imitates the behavior characteristics of animal neural networks and performs distributed parallel information processing.

2. Solve the problem

Nonlinearly separable problems.

3. Advantages

• It has strong robustness and fault tolerance, and the accuracy of classification is high, which is better than almost all other machine learning algorithms;
• Parallel processing method makes the calculation fast;
• Self-learning, self-organizing, adaptive;
• Can fully approximate any complex nonlinear relationship;
• Possess strong information synthesis ability, able to process quantitative and qualitative information at the same time.

4. Disadvantages

• "Black box" operation, unable to determine the derivation process;
• Time-consuming and labor-intensive, with many algorithm details, difficult to control, and expensive;
• The amount of data required is large;
• If the study time is too long, the purpose of study may not even be achieved.

Five, support vector machine - Support Vector Nachine (SVM)

1. Definition

Defines a linear classifier with the largest margin in the feature space.

2. Solve the problem

Two classification problems.

• Linearly Separable Support Vector Machines: Training data is linearly separable
• Linear Support Vector Machines: The training data is approximately linearly separable
• Nonlinear Support Vector Machines: Training data is linearly inseparable

3. Advantages

• Applicable to small samples, which simplifies the usual classification and regression problems;
• Avoid the "curse of dimensionality": the complexity of calculation depends on the number of support vectors, not the dimensionality of the sample space;
• A small number of support vectors determine the final result, which is insensitive to outliers and has good "robustness";
• High generalization performance
• Can solve nonlinear problems;
• The problem of neural network structure selection and local minimum points can be avoided.

4. Disadvantages

• It is difficult to implement for large-scale training samples;

• It is difficult to solve multi-classification problems, and there is no general solution for nonlinear problems;

• Sensitive to missing data, parameter and kernel function choices.

6. (Naive) Bayesian Classification

1. Definition

Defines a linear classifier with the largest margin in the feature space.

2. Advantages

• Fast speed, supports incremental training;
• Solid mathematical foundation and stable classification efficiency;
• The required estimated parameters are few, less sensitive to missing data, and the algorithm is simple;
• The explanation of what a classifier actually learns is relatively straightforward.

3. Disadvantages

• Need to know the prior probability;

• There is an error rate in the classification decision;

• Unable to handle varying results based on feature combinations.

Seven, K-nearest neighbor - KNN

1. Definition

Classification Algorithms in Supervised Learning

2. Advantages

• Ability to use complex functions for numerical prediction, simple, easy to understand, and easy to implement;
• Only need to save training samples and labels, a reasonable amount of data scaling;
• Fast speed, supports incremental training;
• Not susceptible to small error probability;
• The explanation of what a classifier actually learns is relatively straightforward.

3. Disadvantages

• The choice of K is not fixed;

• All training data must be indispensable;

• Has high computational complexity and memory consumption;

• Finding a reasonable scaling factor is tedious.