[Practical Machine Learning] 3.3 Linear Model

linear regression

Example of Linear Regression - House Price Prediction

• Suppose there are three features: $x_1=$ #beds，$x_2=$ #baths，$x_3=$ #living sqft，
• Suppose the predicted value is the weighted sum of all input features: $y=w_1{x}_1+w_2{x}_2+w_3{x}_3+b$
• Weight $w_1,w_2,w_3$and offset $b$ will be learned from the training data

General form of linear regression

In general, given data $x=\left[x_1,x_2,\dots ,x_p\right]$ , that is, each sample is expressed as$p-$ dimensional features, then the linear model predicts

$$\hat{y}=w_1{x}_1+w_2{x}_2+\dots +w_p{x}_p+b=⟨\mathbf{w},\mathbf{x}⟩+b$$

其实$\mathbf{w},x$ are all long as$the\; vector\; of\; p$ ,$w$ and$b$ are parameters that can be learned.

If the above formula is to be implemented through code, it can be written as follows:

# weight w has shape (p,1)
# bias b is a scalar
# data x has shape (p,1)
y_hat = (x*w).sum + b

objective function

Suppose the collection of $nUndersubjective$ valueX$\mathbf{X}={\left[{\mathbf{x}}_1,x_2,\dots ,x_n\right]}^T\in {\mathbb{R}}^{n\times p}$，$X$ is a$n$ row$A\; matrix\; of\; p$ columns, while$X$ input function$\mathbf{y}={\left[y_1,\dots ,y_n\right]}^T\in {\mathbb{R}}^n$。

Goal: Minimize the mean square error (MSE)

$$\begin{array}{rl}{\mathbf{w}}^{\ast },{\mathbf{b}}^{\ast }& =\underset{\mathbf{w},b}{\mathrm{argmin}}\ell (\mathbf{X},\mathbf{y},\mathbf{w},b)\\ & =\underset{w,b}{\mathrm{argmin}}\frac{1}{n}\sum _{i=1}^n{\left(y_i-⟨{\mathbf{x}}_i,\mathbf{w}⟩-b\right)}^2\end{array}$$

linear classification

The output of regression is a continuous real number, but for classification the output is a prediction of the class.

Multi-category classification:

• If we want to output multiple types of objects, we can output a vector. Specifically, the output can be $The\; vector\; of\; m$ , the iiin the vector$The\; i$ element reflects the classification as theiiThe confidence (probability) of$i\; ,\; the\; higher\; the\; value,\; the\; more\; likely\; it\; belongs\; to\; this\; category,\; and\; the\; lower\; the\; less\; likely\; it\; is.$
• We can use a linear model $o_i=⟨\mathbf{x},{\mathbf{w}}_i⟩+b_i$, where $x$ is the feature of the data,$w_i$is a long $A\; vector\; of\; p$ , representing the corresponding class$The\; parameters\; that\; i$ can learn,$b_i$Indicates this type of offset. Then for class $i$ , the degree of confidence is$o_i$. because $m$ each category, so there are a total of$mpcsoi$ o_i_$o_i$。
• 标签$\mathbf{y}=\left[y_1,y_2,\dots ,y_m\right]$ , where only one$y_i=1$ , all others are$0$ , which is one-hot encoding (one-hot encoding), indicates that it belongs to the$class\; i$ .
• We want to minimize the mean squared error (MSE) $\frac{1}{m}\Vert \mathbf{o}-\mathbf{y}{\Vert }_2^2$
• The predicted category is argmax ⁡ i { oi } i = 1 m \operatorname{argmax}_i\left\{o_i\right\}_{i=1}^margmaxi​{ oi​}i=1m​, indicating that the predicted result is the $class\; i$ , this$i$ need to use$o_i$maximum.

Softmax regression (Softmax regression)

Because our goal is to make all $o$ sum$y$ is the same, but when actually doing the classification, we don't care about$o$ , we only hope that the confidence of the true category is large and strong enough. So in order to make our model more focused on the correct class and not caring about other classes, we propose to use softmax.

• First, you need to convert the predicted score into a probability. $o_i$The output is a real number, between negative infinity and positive infinity. If you want to convert it into a probability, you must make it greater than or equal to 0, and all probabilities add up to 1.

$$\hat{\mathbf{y}}=\mathrm{softmax}(\mathbf{o})\text{ where }{\hat{y}}_i=\frac{\exp \left(o_i\right)}{{\sum }_{k=1}^m\exp \left(o_k\right)}$$

O_exp = torch.exp(O)
partition = O_exp.sum(1, keepdim=True)
Y = O_exp / partition

Although nonlinear changes are used here, it is still a linear model. Because the model is looking for the largest y ^ i \hat y_i when making a decision${\hat{y}}_i$Equivalent to the largest $o_i$，即${\mathrm{argmax}}_i{\hat{y}}_i={\mathrm{argmax}}_i{o}_i$

• If you want to compare probabilities $\hat{y}$和yyThe difference between$y\; ,\; we\; can\; use\; cross-entropy\; (cross-entropy):$

$$H(\mathbf{y},\hat{y})=\sum _i-y_i\log \left({\hat{y}}_i\right)=-\log {\hat{y}}_y$$

Because actually $y_i$Only one of them is 1, and the rest are all 0, so the cross entropy can be simplified as $-\log {\hat{y}}_y$。$-\log$ is a decreasing function, so in order to minimize the cross entropy, lety ^ y \hat{y}$\_y$${\hat{y}}_y$maximum. Then this model ultimately only cares about our predicted probability on the correct category , and doesn't care much about other values.

References

3.3 The simplest and most commonly used linear model [Stanford 21 Fall: Practical Machine Learning Chinese Edition]_哔哩哔哩_bilibili

3.1. Linear Regression — Dive into Deep Learning 1.0.0-beta0 documentation

https://c.d2l.ai/stanford-cs329p/_static/pdfs/cs329p_slides_4_3.pdf

[Machine Learning] Re-understanding Linear Regression - 1 - Maximum Likelihood Estimation_哔哩哔哩_bilibili