"Hands-on Learning Deep Learning" Notes (2) Linear Neural Network

3. Linear neural network

3.1 Linear regression

3.1.1 Introduction

  1. Regression is a class of methods for modeling the relationship between one or more independent variables and a dependent variable . Linear regression is based on several simple assumptions: ① the relationship between the independent variable and the dependent variable is linear; ② noise is allowed but follows a normal distribution.
  2. The concept of training data set /training set, sample /data point/data sample, label/target (target trying to predict), feature/covariate (independent variable on which prediction is based), use $n$ represents the number of samples in the data set, and the pair index is$The\; sample\; of\; i$ , the input is expressed as${\mathbf{x}}^{(i)}=[x_1^{(i)},x_2^{\left(i\right)}{]}^{\top }$ , the corresponding label is$y^{\left(i\right)}$ .
  3.The linear assumptioncontains weights and biases, which is an affine transformation of the input features. Put all the features into the vector$\mathbf{w}\in {\mathbb{R}}^{In\; d}$ , the concise representation of the linear model is obtained:$\hat{y}=w^{\top }\mathbf{x}+b$ , and then get the model representation of the entire data set:$\hat{y}=\mathbf{Xw}+b$ , to get the best model parameter$w$ and$b$ , two more things are needed:
(1) A measure of model quality -loss function
$$L(\mathbf{w},b)=\frac{1}{n}\sum _{i=1}^n{l}^{(i)}(\mathbf{w},b)=\frac{1}{n}\sum _{i=1}^n\frac{1}{2}({\mathbf{w}}^{\top }{\mathbf{x}}^{(i)}+b-y^{(i)}{)}^2\text{\hspace{1em}( 1 )}$$
$${\mathbf{w}}^{\ast },b^{\ast }=\underset{\mathbf{w},b}{\mathrm{argmin}}L(w,b)\text{\hspace{1em}( 2 )}$$
  The solution of linear regression can be simply expressed by a formula, but this method has strict restrictions on the problem, so it cannot be widely used in deep learning, so the following is needed— (2) A method that can update the model
toimprove A method for model prediction quality. For example, the gradient descent method (Gradient Descent) can calculate the derivative of the loss function with respect to the model parameters, but in actual execution, a small batch of samples is usually randomly selected every time the update needs to be calculated. This variant is called: small batchrandom gradient descent.
  Randomly select a mini-batch$B$ , predetermine a positive number$\eta$ ,define a simple solution:
$$(\mathbf{w},b)\leftarrow (w,b)-\frac{}{|\mathcal{B}|}\sum _{i\in \mathcal{B}}{\partial }_{(\mathbf{w},b)}{l}^{(i)}(\mathbf{w},b)\text{\hspace{1em}( 3 )}$$
  where$|B|$ represents the number of samples in each mini-batch, also known as the batch size (batch size),$\eta$ represents the learning rate, which is manually set in advance, and these parameters that can be adjusted but not updated during the training process are calledhyperparameters.
  4.generalization; use the model to makepredictions;
  5.Vectorization acceleration: When training the model, the calculation is vectorized, so that the linear algebra library is used instead of the expensive Python For loop to reduce the time required for the program to run .
  6.Normal distribution and square loss: After considering the noise in the observation information and setting it to obey the normal distribution, there is$y={\mathbf{w}}^{\top }\mathbf{x}+b+ϵ$ ,index$ϵ\sim \mathcal{N}(0,p^2)$,Maximum Likelihood Estimation...
  7.From linear regression to deep network: Neural network covers a wealth of models, and linear models can be described in the same way as neural networks. The following uses the "layer" symbol to rewrite this model. (Note: The values ​​​​of weights and biases are hidden in the figure below)

Linear regression is a single layer neural network

  For linear regression, each input is connected to an output, and this transformation is called a fully -connected layer or dense layer. The idea of ​​these algorithms is partly due to our study of real biological nervous systems .

3.1.2 Implementation of linear regression from scratch

3.1.3 Simple implementation of linear regression

3.2 softmax regression

3.2.1 Introduction

  1. [ Classification problem ] Regression can be used to predict how many problems, and we may also be interested in classification problems , not asking "how much", but asking "which one". There are two subtly different problems: ① differentiate only the "hard" categories; ② get the "soft" categories, i.e. the probability of belonging to each category.
  2. [ One-Hot Encoding ] Statisticians long ago invented a simple way to represent categorical data: One-Hot Encoding , which is a vector with as many components as categories, whereby the label $y$ can be expressed as:
$y\in \{(1,0,0),(0,1,0),(0,0,1)\}$ .
  3. [Multi-output model] Taking a 4-pixel picture to distinguish three categories as an example, in order to estimate the probability of each category, amulti-output model:
$$\begin{gathered} o_1=x_1{w}_{11}+x_2{w}_{12}+x_3{w}_{13}+x_4{w}_{14}+b_1 \\ {o}_2=x_1{w}_{21}+x_2{w}_{22}+x_3{w}_{23}+x_4{w}_{24}+b_2 \\ {o}_3=x_1{w}_{31}+x_2{w}_{32}+x_3{w}_{33}+x_4{w}_{14}+b_3\text{\hspace{1em}( 4 )} \end{gathered}$$
  The neural network diagram is described as follows. It can be seen that the softmax regression is also a single-layer neural network, and the output layer is also a fully connected layer.

softmax regression is a single layer neural network

  4. [ Parameter overhead ] In deep learning, the fully connected layer is ubiquitous, for any $d$ inputs andqqA fully connected layer with$q$ outputs, the parameter overheadis$O\left(dq\right)$ , which may not be acceptable in practice, this cost can be reduced to$\mathcal{O}(\frac{dq}{n})$ , the hyperparameter$n$ can be flexibly specified to balance parameter saving with model effectiveness (Zhang et al., 2021).

  5. [ softmax function ] If you want to treat the output as a probability, you must ensure that the output on any data is non-negative and sums to 1. The softmax function can convert unnormalized predictions into non-negative numbers and sum to 1, while maintaining the model's derivability, the formula is as follows:
$$\hat{\mathbf{y}}=\text{softmax}\left(o\right)where{\hat{y}}_j=\frac{\exp (o_j)}{{\sum }_k\exp (o_k)}\text{\hspace{1em}( 5 )}$$
  In this process,${\mathrm{argmax}}_j{\hat{y}}_j={\mathrm{argmax}}_j{o}_j$, although softmax is nonlinear, the output of softmax regression is still determined by the affine transformation of the input features, so softmax regression is a linear model .
  6. [ Vectorization of small batches ], speed up $X$和W \mathbf{W}Matrix-vector multiplication of$W.$
  7. [Loss function], the softmax function gives a vector$\hat{y}$, that is, "for a given input $The\; conditional\; probability\; of\; each\; class\; of\; x$ ", the optimization goal is to maximize$Let\; P(Y|X)$ , and then have the following function:
$$-\log P(Y|X)=\sum _{i=1}^n-\log P({and}^{(i)}|{\mathbf{x}}^{(i)})=\sum _{i=1}^{n}l(y^{(i)},{\hat{\mathbf{y}}}^{\left(i\right)})\text{\hspace{1em}( 6 )}$$
$$l(y^{\left(i\right)},{\hat{y}}^{\left(i\right)})=-\sum _{j=1}^q{y}_j\log {\hat{y}}_j\text{\hspace{1em}( 7 )}$$
  The loss function in (7) is usually calledcross-entropy loss, which is one of the most commonly used losses for classification problems. Since the power is always greater than zero, the resulting probability must be greater than zero, and theoretically the loss function cannot be further minimized.
  8. [Derivative Calculation] Substituting formula (5) into formula (7), we can get:
$$l(y,\hat{\mathbf{y}})=-\sum _{j=1}^q{y}_j\log \frac{\exp (o_j)}{{\sum }_k\exp (o_k)}=\log \sum _{k=1}^q\exp \left(o_k\right)-\sum _{j=1}^q{y}_j{o}_j\text{\hspace{1em}( 8 )}$$
get the loss relative to any unnormalized prediction$o_j$Definitions:
$${\partial }_{o_j}l(y,\hat{\mathbf{y}})=\frac{\exp (o_j)}{{\sum }_{k=1}^q\exp (o_k)}-y_j=\text{softmax}(\mathbf{o}{)}_j-y_j\text{\hspace{1em}( 9 )}$$
  9. [Calculation of entropy], the core idea is to quantify the information content in the data, this value is called distribution$The\; entropy\; of\; P$ (entropy), the calculation method is$H[P]={\sum }_j-P(j)\log P\left(j\right)$ , nat, bit...
  [Compression and Prediction] Imagine a data stream that needs to be compressed, if it is easy to predict the next data, it is easy to be compressed, and when we can't completely predict every time, we will feel "Surprised", Shannon decided to use the amount of information$\log \frac{1}{P(j)}=-\log P\left(j\right)$ to quantify this surprise, we assign a lower probability to an event, the greater our surprise, the greater the amount of information contained, and the above entropy is when the assigned probability really matches the data generation process information expectations.
  [Understanding of entropy and cross entropy] Put entropy$H\left(P\right)$ imagined as "the degree of surprise experienced by a person who knows the true probability", the cross entropy$H(P,Q)$ is "the subjective probability is$An\; observer\; of\; Q$ sees with probability$Expected\; surprise\; when\; P$ generates data",$P=$The cross-entropy reaches the minimum at$Q$
  [Cross-entropy classification objective] The cross-entropy classification objective can be considered from two aspects: ① Maximize the likelihood of observed data; ② Minimize the surprise required to convey the label.

  10. [ Model prediction evaluation ] Use the class with the highest predicted probability as the output class, and if the prediction agrees with the actual class, the prediction is correct. Use precision (the ratio between the number of correct predictions and the total number of predictions) to evaluate the performance of the model.

3.2.2 Implementation of softmax regression from scratch

3.2.3 Simple implementation of softmax regression

3.3 Image Classification Dataset

  MNIST ( LeCun et al., 1998 ) is one of the widely used data sets in image classification, but it is too simple as a benchmark data set. Fashion-MNIST ( Xiao et al., 2017 ) data set is similar but more complex. It is a clothing Classification dataset, which consists of 10 categories of images, each category consists of 6000 images in the training set and 1000 images in the test set.

""" 导入所需库 """
%matplotlib inline
import torch
import torchvision
from torch.utils import data
from torchvision import transforms
from d2l import torch as d2l

d2l.use_svg_display()

""" 读取数据集 """
# 通过ToTensor实例将图像数据从PIL类型变换成32位浮点数格式，
# 并除以255使得所有像素的数值均在0～1之间
trans = transforms.ToTensor()
mnist_train = torchvision.datasets.FashionMNIST(
    root="../data", train=True, transform=trans, download=True)
mnist_test = torchvision.datasets.FashionMNIST(
    root="../data", train=False, transform=trans, download=True)

len(mnist_train), len(mnist_test) # (60000, 10000)
mnist_train[0][0].shape # torch.Size([1, 28, 28])

""" 在数字标签索引及其文本名称之间转换 """
def get_fashion_mnist_labels(labels):  #@save
    """返回Fashion-MNIST数据集的文本标签"""
    text_labels = ['t-shirt', 'trouser', 'pullover', 'dress', 'coat',
                   'sandal', 'shirt', 'sneaker', 'bag', 'ankle boot']
    return [text_labels[int(i)] for i in labels]

""" 可视化样本图像 """
def show_images(imgs, num_rows, num_cols, titles=None, scale=1.5):  #@save
    """绘制图像列表"""
    figsize = (num_cols * scale, num_rows * scale)
    _, axes = d2l.plt.subplots(num_rows, num_cols, figsize=figsize)
    axes = axes.flatten()
    for i, (ax, img) in enumerate(zip(axes, imgs)):
        if torch.is_tensor(img):
            # 图片张量
            ax.imshow(img.numpy())
        else:
            # PIL图片
            ax.imshow(img)
        ax.axes.get_xaxis().set_visible(False)
        ax.axes.get_yaxis().set_visible(False)
        if titles:
            ax.set_title(titles[i])
    return axes

""" 展示图像 """
X, y = next(iter(data.DataLoader(mnist_train, batch_size=18)))
show_images(X.reshape(18, 28, 28), 2, 9, titles=get_fashion_mnist_labels(y));

In order to make it easier to read the training set and test set, Pytorch's built-in data iterator is generally used instead of creating it from scratch. The data iterator reads a small batch of data (batch_size) in each iteration, and can randomly shuffle the samples to ensure unbiasedness.

batch_size = 256
def get_dataloader_workers():  #@save
    """使用4个进程来读取数据"""
    return 4

train_iter = data.DataLoader(mnist_train, batch_size, shuffle=True,
                             num_workers=get_dataloader_workers())
""" 统计时间 """
timer = d2l.Timer()
for X, y in train_iter:
    continue
f'{
      
      timer.stop():.2f} sec'

Integration components

def load_data_fashion_mnist(batch_size, resize=None):  #@save
    """下载Fashion-MNIST数据集，然后将其加载到内存中"""
    trans = [transforms.ToTensor()]
    if resize:
        trans.insert(0, transforms.Resize(resize))
    trans = transforms.Compose(trans)
    mnist_train = torchvision.datasets.FashionMNIST(
        root="../data", train=True, transform=trans, download=True)
    mnist_test = torchvision.datasets.FashionMNIST(
        root="../data", train=False, transform=trans, download=True)
    return (data.DataLoader(mnist_train, batch_size, shuffle=True,
                            num_workers=get_dataloader_workers()),
            data.DataLoader(mnist_test, batch_size, shuffle=False,
                            num_workers=get_dataloader_workers()))

train_iter, test_iter = load_data_fashion_mnist(32, resize=64)
for X, y in train_iter:
    print(X.shape, X.dtype, y.shape, y.dtype)  # torch.Size([32, 1, 64, 64]) torch.float32 torch.Size([32]) torch.int64
    break