SVR (Support Vector Machine) Usage Introduction

Introduction to SVR

Support Vector Machine (SVM for short) is a commonly used supervised learning algorithm, mainly used for classification and regression problems. Its principle is based on the idea of ​​structural risk minimization, and realizes classification or regression tasks by finding an optimal hyperplane in the feature space.

The principle of support vector machine includes the following key concepts:

1. Separating hyperplane: SVM tries to find a hyperplane that can completely separate samples from different classes. For binary classification problems, this hyperplane is a two-dimensional plane; for multi-classification problems, multiple binary classification hyperplanes can be constructed in a one-to-one or one-to-many manner.

2. Support Vectors: Support vectors refer to the sample points closest to the separating hyperplane. They play an important role in determining the hyperplane because they determine the location and spacing of the hyperplane.

3. Margin maximization: The goal of a support vector machine is to find a separating hyperplane that maximizes the distance (ie, the margin) between the support vectors of the two classes and the hyperplane. Maximizing the margin helps improve the robustness and generalization of the model.

4. Kernel function: In practical applications, samples may be linearly inseparable. In order to solve this problem, the support vector machine introduces the concept of kernel function, which maps the samples from the original feature space to the high-dimensional feature space, so that the non-linearly separable problem becomes a linearly separable problem in the high-dimensional space. Commonly used kernel functions include linear kernel, polynomial kernel, and Gaussian radial basis kernel.

5. Regularization parameter: The support vector machine also introduces a regularization parameter C to control the complexity and fault tolerance of the model. Smaller values ​​of C result in larger intervals but less fault tolerance, and larger values ​​of C result in smaller intervals but higher fault tolerance.

By solving the optimization problem of the support vector machine, an optimal hyperplane can be obtained to realize the classification or regression task. Support vector machines have good generalization performance and adaptability in practical applications, and perform well for high-dimensional data and small sample sizes.

1. Introduction to SVR regression

SVR (Support Vector Regression) is the application of Support Vector Machine (SVM) in regression problems. Similar to the SVM classification model, SVR is also a non-probabilistic algorithm. By using the kernel function to map the data to a high-dimensional space, and searching for the optimal interval between the hyperplane and the training data in this space, the obtained regression model.

Different from the traditional regression model, SVR transforms the regression problem into a process of finding a function that approximates the real function. In practical applications, different kernel functions and hyperparameters can be used to adjust the model to obtain better fitting results.

2. Establishment of SVR regression model

The basic steps to build an SVR regression model are as follows:

1. Data preprocessing
The SVR regression model has relatively high requirements on data, and data preprocessing is required. First, the data needs to be standardized so that the mean is 0 and the variance is 1. Secondly, it is necessary to normalize the data and scale the data to the [0,1] range, so as to prevent the data differences between features from affecting the model training effect.

X_train = StandardScaler().fit_transform(X_train)
y_train = np.log1p(y_train) # 取对数归一化，提高拟合效果

2. Model training
You can use the SVR class in the sklearn library to train the model, where you need to specify the kernel function and hyperparameters, for example:

clf = SVR(kernel='rbf', C=1, gamma=0.1, epsilon=0.1)
clf.fit(X_train, y_train)

Among them, the kernel parameter is the kernel function type, C is the regularization parameter, gamma is the bandwidth parameter of the rbf kernel function, and epsilon is the error tolerance parameter.

3. Model evaluation
You can use the mean_squared_error function in the sklearn library to calculate the mean square error (MSE) of the model and evaluate the fitting effect of the model, for example:

y_pred = clf.predict(X_test)
mse = mean_squared_error(y_test, y_pred)

The mse can be compared to the mean and standard deviation to assess how well the model fits.

3. SVR regression model tuning

In SVR regression, parameter tuning is a very important process. Commonly used parameter tuning methods mainly include grid search method and random search method.

1. Grid search method
The grid search method traverses all possible values ​​of hyperparameters, and selects the best combination of hyperparameters to obtain the optimal model. Grid search can be performed using the GridSearchCV class in the sklearn library.

For example, a grid of parameters can be defined specifying different kernel, C and gamma values ​​for model training and evaluation:

param_grid = {
    
    'kernel': ['rbf'], 'C': [0.1, 1, 10], 'gamma': [0.1, 0.01, 0.001]}
grid_search = GridSearchCV(clf, param_grid, cv=5, scoring='neg_mean_squared_error')
grid_search.fit(X_train, y_train)

2. Random search method
Different from the grid search method, the random search method samples from the specified hyperparameter space according to the probability distribution, so as to find the optimal hyperparameter combination faster. Randomized searches can be performed using the RandomizedSearchCV class in the sklearn library.

For example, hyperparameter distributions can be defined to specify the value distribution of different kernel functions, C and gamma for model training and evaluation:

param_distribs = {
    
    'kernel': ['rbf'], 'C': reciprocal(20, 200000), 'gamma': expon(scale=1.0)}
rnd_search = RandomizedSearchCV(clf, param_distributions=param_distribs, n_iter=50, cv=5, scoring='neg_mean_squared_error')
rnd_search.fit(X_train, y_train)

4. SVR Returns to Practical Application

SVR regression can be applied in many fields, such as stock prediction, housing price prediction, and character relationship prediction. Let's take a simple housing price prediction as an example to introduce the practical application of SVR regression.

1. Data collection and processing
First, it is necessary to collect house sample data, including house area, number of rooms, number of bathrooms, number of kitchens, geographical location, etc. Preprocess the data, including feature scaling, normalization, and classification encoding, etc.

2. Model training and parameter adjustment
You can use the SVR class in the sklearn library to train the model, and use the grid search method or random search method to adjust the hyperparameters to obtain the best fitting effect.

param_grid = {
    
    'kernel': ['rbf'], 'C': [0.1, 1, 10], 'gamma': [0.1, 0.01, 0.001]}
grid_search = GridSearchCV(clf, param_grid, cv=5, scoring='neg_mean_squared_error')
grid_search.fit(X_train, y_train)

3. Model testing and performance evaluation
Use test data to test the model, and use indicators such as mean square error (MSE) and R-square value to evaluate the performance of the model. Evaluation can be performed using the mean_squared_error and r2_score functions in the sklearn library:

y_pred = clf.predict(X_test)
mse = mean_squared_error(y_test, y_pred)
r2 = r2_score(y_test, y_pred)

Five, SVR regression summary

This article elaborates on the introduction of SVR regression, model establishment, parameter adjustment and practical application. SVR regression is a very useful regression model that has wide applications in several fields.