deep learning|1|softmax regression

Series Article Directory

---

foreword

Softmax regression (softmax regression) is actually the general form of logistic regression, logistic regression is used for binary classification, and softmax regression is used for multi-classification

---

softmax regression is a single-layer neural network with multiple outputs

Neural Network: All neurons in the output layer are linearly connected to all neurons in the input layer. no middle layer

For any output neuron $y_j$, Toyu $y_j=\sum w_{i,j}{x}_i+bj$

Furthermore, the softmax regression model can be written as:

$y=Wx+b$

Suppose $x\in {\mathbb{R}}^{n\times 1}$, $y\in R^{m\times 1}$,

$Then$

$W\in {\mathbb{R}}^{m\times n},b\in R^{m\times 1}$

---

The output of Softmax regression is not a real value, but a probability between 0 and 1

The essential difference between softmax regression and linear regression is that the output has different meanings

Take iris as an example: $y_1,y_2,y_3$represent x \pmb{x} respectivelyProbability$that\; x$ belongs to Iris iris, Iris versicolor, or Iris virginia.

Since it is a probability, then $y_1,y_2,y_3$The range of values:

1. 应当在1的范围之内
2. 和应当为1

Treating it with linear regression does not guarantee $y_1,y_2,y_3$The range of values ​​satisfies the above conditions.

The output needs to be softmax processed.

$y_i=\frac{e^{y_i}}{{\sum }_{j=1}^3{e}^{y_j}}$

The above formula is the softmax function .

Obviously, $y_i\in (0,1]$ , y can never be equal to 0, but may be very close to 0.

Example 1: Given an array {-0.5, 0, 10}, calculate their softmax output

$e^{-0.5}=0.6065,e^0=1,e^{100}=22026.4658$

${\sum }_{j=1}^3{e}^{y_j}=22028.0723$

$y_1=0.00003$

$y_2=0.00004$

$y_3=0.99993$

Example 2: Given an array {0.5, 0.8, 0.4}, calculate their softmax output

$e^{0.5}=1.649,e^{0.8}=2.226,e^{0.4}=1.492$

${\sum }_{j=1}^3{e}^{y_j}=5.367$

$y_1=0.307$

$y_2=0.415$

$y_3=0.278$

In summary, a more complete softmax regression model can be written as

$y=softmax(Wx+b)$

---

What kind of data do we expect the softmax model to output?

Linear regression uses (mean square error, MSE) loss

$l=0.5(\hat{y}-y{)}^2$

Its purpose is to make the predicted value closer to the real value. Obviously, this doesn't apply to softmax regression models .

In the softmax regression model, the output is a discrete value, and each value represents the probability of a category.

We expect:

• The predicted class of the correct class is higher $\to$ close to 1

• The predicted class of the wrong class is lower $\to$ close to 0

Then in the extreme case, our expected output should be a string of numbers consisting of 1 and 0, of which there is only one element that is 1, and the position of 1 is in the correct category.

$$y_i=\{\begin{array}{ll}0,& category\ne i\\ 1,& category=i\end{array}$$

Example: In the iris data set, the labels of Iris Iris, Iris Versicolor, and Iris Virginia are respectively 1, 2, and 3. Then in the softmax model, the best outputs we expect are:

Yamanashio = [1, 0, 0]

color iris = [0, 1, 0]

Virginia Kite = [0, 0, 1]

–

So for a given softmax output, how do we measure the gap between him and lable?

cross entropy loss

Unlike MSE or L1 norm loss, cross-entropy loss can more sensitively reflect the gap between the predicted result and the real value:

The greater the difference between the results

$\downarrow$

The greater the loss

$\downarrow$

The multiplication of the model will be larger

$\downarrow$

The greater the intensity of parameter adjustment

The specific form of cross entropy loss

$H(p,q)={\sum }_i-p_i\text{log}(q_i)$

where $p,q$ denotes the probability of the output and the true value, respectively.

Example 1

The actual value is the first type of iris, and its value in the first dimension is 1; the predicted result is [0.3, 0.2, 0.7], and its value in the first dimension is 0.3.

Then you can calculate the cross entropy $H([1,0,0],[0.3,0.2,0.7])=-(\text{0.3})\approx 1.7$

Example 2

The actual value is the first type of iris, and its value in the first dimension is 1; the predicted result is [0.6, 0.2, 0.2], and its value in the first dimension is 0.6.

Then you can calculate the cross entropy $H([1,0,0],[0.6,0.2,0.2])=-(\text{0.6})\approx 0.74$

Example 3

The actual value is the first type of iris, and its value in the first dimension is 1; the predicted result is [0.8, 0.1, 0.1], and its value in the first dimension is 0.8.

Then you can calculate the cross entropy $H([1,0,0],[0.8,0.1,0.1])=-\left(\text{0.6}\right)\approx 0.32$

Obviously, the more accurate the prediction, the closer the distribution to the true probability, and the lower the cross-entropy.
Therefore, it is very suitable to use cross entropy as a loss function.

–

Theoretical summary

1. expression

$$\left[\begin{array}{c}{o}_1\\ ⋮\\ o_n\end{array}\right]=\left[\begin{array}{ccc}{w}_{11}& \cdots & w_1\\ ⋮& \ddots & ⋮\\ w_n& \cdots & w_{mn}\end{array}\right]\times \left[\begin{array}{c}{x}_1\\ ⋮\\ x_m\end{array}\right]+\left[\begin{array}{c}{b}_1\\ ⋮\\ b_n\end{array}\right]$$

Written in matrix form is:

$$O=Wx+b$$

The final result is softmax:

$$y=\left[\begin{array}{c}{y}_1\\ ⋮\\ y_n\end{array}\right]=\text{softmax}(o)=\frac{1}{{\sum }_i{e}^{o_i}}\left[\begin{array}{c}{e}^{o_1}\\ ⋮\\ e^{o_n}\end{array}\right]$$

2. Loss function
   crossover Loss function
   $$L(y,\hat{y})=-\sum _i{y}_i\text{log}\hat{y}=-\text{log}{\hat{y}}_{i|y_i=1}$$