Machine Learning (22) SVM Example 3: [Octave] solves binary classification (nonlinear SVM [unclear boundary])

1 Visualization of training set ex6data3.mat

Plot the training set function plotData.m

Same as plotData.m in Machine Learning (20)

Part of the code of the training set visualization script GaussianSVM1.m

%% Initialization

clear ; close all; clc

%% =============== Part 1: Loading and visualizing data================

fprintf('Loading and Visualizing Data ...\n')

% Load from the file ex6data3.mat and find that there are X, y, Xval, yval variable values in the environment:

load('ex6data3.mat');

% Plot the training set data

plotData(X, y);

% Plot cross validation set data

plotData(Xval, yval);

fprintf('Program paused. Press enter to continue.\n');

pause;

Results of the

The left picture shows a total of 211 training set samples, and the right picture shows a total of 200 test set samples. Each sample has 2 input features.

It can be seen that it is still a two-classification problem, but the corresponding boundary is not obvious, so it is necessary to select appropriate parameters C and σ according to the cross-validation set training situation (this is also the content in 4.2, select parameters based on the cross-validation set).

2 Gaussian kernel function of SVM gaussianKernel.m

Same as gaussianKernel.m in Machine Learning (21)

3 Find the optimal C and σ function dataset3Params.m

Here, C and σ are verified successively using a geometric sequence with a common ratio of 3. Each time, the parameter theta is obtained through training based on these two corresponding values, and then the accuracy of the cross-validation set is obtained. By comparing this time with the last time Accuracy, select parameter values with higher accuracy. It is easy to get that the number of loops is the square of length(vec), that is, there is one loop inside and outside.

function [C, sigma] = dataset3Params(X, y, Xval, yval)

% dataset3Params returns the C and sigma selected in exercise part 3

% Return the best C and sigma based on the cross-validation set.

% Input: X training matrix, the number of rows is the number of samples, and the number of columns is the number of input features

% y The output feature vector of the training set is a column vector containing 1 and 0. The number of rows is the number of samples and the number of columns is 1.

% Xval verification sample matrix, the number of rows is the number of samples, and the number of columns is the number of input features

% yval is the verification sample output feature vector, which is a column vector containing 1 and 0. The number of rows is the number of samples and the number of columns is 1.

% Output: Used to select the optimal (C, sigma) learning parameters for support vector machine and RBF kernel

C = 1; %Initialize C, sigma

sigma = 0.3;

% Description: The following code returns the optimal C and Sigma learning parameters found using the cross-validation set.

% You can use svmPredict to predict labels in the cross-validation set. For example,

%      predictions = svmPredict(model, Xval);

% will return predictions on the cross-validation set

% Note: You can use mean(double(predictions ~= yval)) to calculate the prediction error

vec = [0.01 0.03 0.1 0.3 1 3 10 30]';

C = 0.01;

sigma = 0.01;

model= svmTrain(X, y, C, @(x1, x2) gaussianKernel(x1, x2, sigma));

predictions = svmPredict(model,Xval);

meanMin = mean(double(predictions ~= yval));

C_optimal = C;

sigma_optimal = sigma;

for i = 1:length(vec)

    for j = 1:length(vec)

      C = thing(s);

      sigma = thing(j);

      model= svmTrain(X, y, C, @(x1, x2) gaussianKernel(x1, x2, sigma));

      predictions = svmPredict(model,Xval);

      if(meanMin >= mean(double(predictions ~= yval)))

        meanMin = mean(double(predictions ~= yval));

        C_optimal = C;

        sigma_optimal = sigma;

      endif

    endfor

endfor

C = C_optimal;

sigma = sigma_optimal;

endfunction

4 Training function svmTrain.m

Same as svmTrain.m in Machine Learning (20)

5 Nonlinear SVM script GaussianSVM1.m

%% Initialization

clear ; close all; clc

%% =============== Part 1: Loading and visualizing data================

fprintf('Loading and Visualizing Data ...\n')

% Loading from the file ex6data3.mat, you will find that there are X and y variable values in the environment:

load('ex6data3.mat');

% Plot the training set data

plotData (X, y);

fprintf('Program paused. Press enter to continue.\n');

pause;

%% ==================== Part 2: Nonlinear SVM training ====================

% After implementing the kernel, we can now use it to train the SVM classifier.

% Loading from the file ex6data3.mat, you will find that there are X and y variable values in the environment:

fprintf('\nTraining SVM with RBF Kernel (this may take 1 to 2 minutes) ...\n');

load('ex6data3.mat');

% Try different SVM parameters here

[C, sigma] = dataset3Params(X, y, Xval, yval);

% Train SVM

model= svmTrain(X, y, C, @(x1, x2) gaussianKernel(x1, x2, sigma));

visualizeBoundary(X, y, model);

fprintf('Program paused. Press enter to continue.\n');

pause;