Machine Learning(week 7~11)

Support Vector Machines

Large Margin Classification

1. Optimization Objective

Support vector machine (SVM): a supervised learning algorithm, sometimes gives cleaner, more powerful ways of learning complex non-linear algorithm than logistic regression and neural network.

Modify logistic regression to get SVM

In logistic regression: (as we have known)

If y=1, we want h_θ(x)≈1, i.e. θ^Tx>>0;

If y=0, we want h_θ(x)≈0, i.e. θ^Tx<<0.

A single example’s contribution to the overall cost:

when y=1 or y=0, only one of the terms matters: (z=θ^Tx)

Build a new function: a straight line and a flat line, joining at x=1 (when y=1, the function is called cost₁(z).) or x=-1 (when y=0, the function is called cost₀(z).).

Replace these terms with cost₁(z) and cost₀(z) in the logistic regression cost function:

A minimization problem for SVM: