[Machine Learning] Basics of Supervised Algorithms

The concept of supervised learning

• The main task of supervised learning is to predict the label according to the characteristics of the data object , and the corresponding learning algorithm needs to learn from experience
• Experience comes from labeled training data , which is randomly sampled data from a collection of data objects, also called samples .

feature group (features)

---

Definition : feature group In supervised learning tasks, the vector x = ( x 1 , x 2 , . . . , xn ) ∈ R nx=(x_{1},x_{2},...,x_{n})\in R^ n $x=(x_1,x_2,...,x_n)\in R^n$ , is called the feature group of the object, that is, the feature vector. Let$X\subseteq R^n$ is the set of all possible values ​​of the feature group, called$X$ isthe sample space.

Label

---

Definition : In a regression problem, the training data contains a numerical label $y\in R$ ; in$In\; the\; k$ -ary classification problem, the training data contains a vector label$y\in [0,1{]}^k$。设$Y$ is all possible label values, called$Y$ is the label space.
Classification problems:
For example: handwritten digit recognition problem:$y\in [0,1{]}^{10}$

1. Vector label for number 3: $y=(0,0,0,1,0,0,0,0,0,0,0)$
2. Scalar label for number 3: y = 3 , y ∈ Y , Y = { 0 , 1 , 2 , 3 , . . . , 9 } y=3,y\in Y,Y=\{0,1,2,3,...,9\}y=3,y∈Y,Y={ 0,1,2,3,...,9}

Model

---

Definition : Let $X$ is the sample space,$Y$ is the label space,$\varphi$的$X->$The mappingsetof$Y$$\varphi$ isthe model spaceh ∈ ϕ h\in \phiin any model space$h\in \varphi$ , called amodel

supervised learning tasks

---

Definition : given sample space $X$ , label space$Y$ , unknown characteristic distribution$D$ and label distribution{ D x : x ∈ X } \{D_{x}:x\in X\}{ Dx​:x∈X } ,the supervised learning taskis to train amodel$h$ , which is calculated from$X$到YYAmapping$of\; Y$ , denoted as:$$h:X->$$The eigenvector xx$$of\; Y$$ for any sample$x$ , given by$h\left(x\right)$ as pairxxThe prediction of the label of$x\; .$$$\hat{y}=h(x)$$

loss function

---

Definition : Let $X$ is the sample space,$Y$ is the label space, the loss function is a from$Y\times Y$ to positive real number$R+$ function$$l:Y\times Y->R^{+}$$ : and satisfy the following properties, for any$y\in Y$ ,有$$l(y,y)=0Specifies$$ : 0-1 Let$$l(y,\hat{y})=\{\begin{array}{r}0,ify=\hat{y}\\ 1,otherwise\end{array}$$
其中：$\hat{y}=h(x),x\in X,\hat{y}\in and$ ($y$ is the true value of the label of the sample,$\hat{y}$is the label prediction value for sample x)

For example: square loss function $$l(y,\hat{y})=(y-\hat{y}{)}^2,y,\hat{y}\in R$$

Test data and model metrics (test error)

---

In supervised learning, given a sample space $X$ , label space$Y$ ,the unknown characteristic distribution$D$ and label distribution{ D x : x ∈ X } , \{D_x:x\in X\},{ Dx​:x∈X } , assuming that the model type output by the supervised learning algorithm is$h$ . Given a set of data$$T=\{(x^{(1)},y^{(1)}),(x^{(2)},y^{(2)}),...,(x^{(t)},y^{(t)})\}$$其中，$x^{\left(1\right)},x^{\left(2\right)},...,x^{(t)}\sim D,$ forXXAccording to the characteristic distributionDD$in\; X$$Independent\; sampling\; of\; D$ , and for any$1\le i\le t,$有$y^{(i)}\sim D_{x^{(i)}},$ will$T$ isthe test data set, with model$h$ in the test dataset$T$上的平均损失$$L_T(h)=\frac{1}{t}\sum _{i=1}^{t}l(h(x^{(i)},y^{\left(i\right)}))$$as modelhhA measure of the effect ofh$When$
the test data size is sufficiently large, the empirical loss approximates the expected loss well.

Generally speaking, given a data set (it is not clear about its distribution $D$ ), first randomly divide it into a test set and a training set in a certain proportion, so that both sets will satisfy the distribution$D$ , train the model$h$ , compute the loss on the test set, as modelhhThe effect measure of$h$

Empirical Loss Minimization Algorithm Architecture (Training Error)

Given a loss function $l,$ and given a set of data$$S=\{(x^{(1)},y^{(1)}),(x^{(2)},y^{(2)}),...,(x^{(t)},y^{(t)})\}$$其中，$x^{(1)},x^{\left(2\right)},...,x^{(t)}\sim D,$ forXXAccording to the characteristic distributionDD$in\; X$$Independent\; sampling\; of\; D$ , and for any$1\le i\le t,$有$y^{(i)}\sim D_{x^{(i)}},$ the$S$ isthe training data sethhtrained on the training data set$h$ inthe training data$S$上的平均损失$$L_S(h)=\frac{1}{t}\sum _{i=1}^{t}l(h(x^{(i)},y^{\left(i\right)}))$$as model$h$ 'sempirical loss
unconstrained empirical loss minimization algorithm architecture:
  given sample space$X$ , label space$Y$ , model space$\Phi$ ,
  loss function:$l:Y\times Y->R^{+}$
  Enter:$m$ training data$S=\{(x^{(1)},y^{(1)}),(x^{(2)},y^{(2)}),...,(x^{(m)},y^{\left(m\right)})\}$
  Output model:$h_s={\mathrm{argmin}}_{h\in \varphi }{L}_s\left(h\right)$
features:$h_s\left(x\right)$ in the training dataSSExperience loss on$S$$L_s(h)=0$ , the unconstrained empirical loss minimization algorithm is prone tooverfittingproblems
ps: Unconstrained refers to the space ($\varphi$ ) Unconstrained (model space composed of all qualified models)
overfitting example:
  Suppose the sample space X=[-1,1], the feature distribution D is a uniform distribution on X, the label space Y=ℝ, the label distribution Dx=N(x, 0.3) is a normal distribution with an expectation of x and a standard deviation of 0.3, and the loss function is a square loss function.

Overfitting: small training error, large test error!

1. $h_s\left(x\right)$ inthe training dataSSExperience lossLSon$S\; (\; h\; )\; L\_S(h)$$L_S\left(h\right)$ ---- training error
2. $h_s\left(x\right)$ intest dataSSExperience lossLTon$S\; (\; h\; )\; L\_T(h)$$L_T\left(h\right)$ ----Test error
   Overfitting: small training error, large test errorUnderfitting
   : relatively large training errorGeneralization ability: small training error, small test error
   

We work on the generalization ability of the model, preventing the method of fitting:

1. Introduce model assumptions (make appropriate assumptions about label distribution or model structure, such as the above example, which can be assumed to be a linear model)
2. Regularization ( $L_1、L_2$Regularization)
3. $dropout$ (for deep neural networks)
4. Expand the training sample

EXAMPLE OF EXPERIENCE LOSS MINIMIZATION: BLOCK CLASSIFICATION

---

from sklearn.datasets._samples_generator import make_blobs
import pandas as pd
import matplotlib.pylab as plt
import numpy as np
from sklearn.model_selection import train_test_split


# 感知器算法
class Perception:
    def __init__(self):
        self.b = None
        self.w = None

    def fit(self, X, y):
        m, n = X.shape
        w = np.zeros((n, 1))
        b = 0
        done = False
        while not done:
            done = True
            for i in range(m):
                x = X[i].reshape(1, -1)
                if y[i] * (x.dot(w) + b) <= 0:
                    w = w + y[i] * x.T
                    b = b + y[i]
                    done = False
        self.w = w
        self.b = b

    def predict(self, x):
        return np.sign(x.dot(self.w) + self.b)


# 构建数据集合
X, y = make_blobs(n_samples=100, centers=2, n_features=2, cluster_std=0.6, random_state=0)
y[y == 0] = -1
data = pd.DataFrame(X, columns=['x1', 'x2'])
data['y'] = y
# 根据题意：划分不同的test_size
fig, axes = plt.subplots(nrows=2, ncols=3, figsize=(12, 8))
axes = axes.ravel()
ax1 =fig.add_subplot(131)
ax1.plot(data['x1'][data['y'] == 1], data['x2'][data['y'] == 1], "bs", ms=3)
ax1.plot(data['x1'][data['y'] == -1], data['x2'][data['y'] == -1], "rs", ms=3)
ax1.set_title('Original Data')


for i, test_size in enumerate([0.5, 0.4, 0.3, 0.2], start=1):
    print(i,test_size)
    # 划分数据集（训练集，测试集）
    X_train, X_test, y_train, y_test = train_test_split(data[['x1', 'x2']], data['y'], test_size=test_size)
    # 训练模型
    model = Perception()
    model.fit(np.array(X_train), np.array(y_train))
    w = model.w
    b = model.b

    # 作图
    if i == 3:
        i = i+2
    ax = axes[i]
    ax.plot(data['x1'][data['y'] == 1], data['x2'][data['y'] == 1], "bs", ms=3)
    ax.plot(data['x1'][data['y'] == -1], data['x2'][data['y'] == -1], "rs", ms=3)
    ax.set_title('Test Size: {:.1f}'.format(test_size))
    x_0 = np.linspace(-1, 3.5, 200)
    line = -w[0] / w[1] * x_0 - b / w[1]
    ax.plot(x_0, line)

plt.subplots_adjust(hspace=0.3)
plt.show()

in conclusion:

1. Under all test set ratios, the perceptron model showed a good classification effect, and successfully divided the data set into two clusters.
2. When the test set ratio is 0.2, the performance of the model is the best, and the classification effect is closest to the real dividing line.

regularization algorithm

---

Foreword:

• In the process of minimizing empirical loss , a reasonable choice of model assumptions is an effective way to avoid overfitting
• How to deal with the possibility of overfitting despite the choice of model assumptions? regularization algorithm

Common strategies for regularization:

• $L_1$Regularization
• $L_2$Regularization

Occam's Razor: Don't Multiply Entities Unnecessarily

For example: each $n$ -ary linear function$$h(x)=<w,x>+b,x\in R^n$$ can use$n+1$ parameter means:$$w=(w_1,w_2,...,w_n)andb$$ use$h_w$Display reason $w=(w_1,w_2,...,w_n)$ The model represented by this set of parameters.
In machine learning,use the parameter wwThe norm of$w$$h_w$Complexity $$L_{1}Norm:|w|=|w_1|+|w_2|+...+|w_n|$$$$L_{2}Norm:||w||=\sqrt{w_1^2+w_2^2+...+w_n^2}$$

ps: what we are after is $h_w$The complexity is low, so it is $The\; w$ vectorhas fewandsmall

---

$$L_{1}Norm:|w|=|w_1|+|w_2|+...+|w_n|$$

• Every non-zero parameter $w_j$Both bring size $\lambda |w_j|$ punishment
• Algorithm $L_s$When the value is close, choose $L_1$A model with a smaller norm

---

$$L_{2}Norm:||w||=\sqrt{w_1^2+w_2^2+...+w_n^2}$$

• Every non-zero parameter $w_j$Both bring size $\lambda w_j^2$punishment