[Deep Learning] Overview of Machine Learning (2) Gradient Descent Method of Optimization Algorithm (Batch BGD, Stochastic SGD, Mini-batch)

Article directory

1. Basic concepts
2. Three elements of machine learning

1. Basic concepts

Machine learning: Through algorithms, machines can learn patterns from large amounts of data to make decisions on new samples.
Machine learning is the process of learning (or "guessing") general rules from limited observation data, and can generalize the summarized rules to unobserved samples.
Insert image description here

2. Three elements of machine learning

Machine learning methods can be roughly divided into three basic elements: Model, learning criterion, optimization algorithm.

1. Model

a. Linear model

$f(\mathbf{x}; \boldsymbol{\theta}) = \mathbf{w}^T \mathbf{x} + b$

b. Nonlinear model

The generalized nonlinear model can be written as multiple nonlinear basis functions $\boldsymbol{\phi}(\mathbf{x})$ 的线性组合： $f(\mathbf{x}; \boldsymbol{\theta}) = \mathbf{w}^T \boldsymbol{\phi}(\mathbf{x}) + b$ 使用， $\boldsymbol{\phi}(\mathbf{x}) = [\phi_1(\mathbf{x}), \phi_2(\mathbf{x}), \ldots, \phi_K(\mathbf{x}) ]^T$ Koreyu $A vector composed of K$ nonlinear basis functions, parameters $\boldsymbol{\theta}$ Including weight direction $\mathbf{w}$ sum offset $b$ .
Results $\boldsymbol{\phi}(\mathbf{x})$ The truth is that it can be learned. Base function, example:

$\phi_k(\mathbf{x}) = h(\mathbf{w}_k^T \boldsymbol{\phi}'(\ mathbf{x}) + b_k)$ in that, $h(\cdot)$ It's definitely a linear function, < /span> $\boldsymbol{\phi}'(\mathbf{x})$ Function, $\mathbf{w}_k$ sum $b_k$ is a learnable parameter, then the model $f(\mathbf{x}; \boldsymbol{\theta})$ is equivalent to the neural network model.

2. Learning Guidelines

a. Loss function

b. Risk minimization criteria

[Deep Learning] Overview of Machine Learning (1) Three Elements of Machine Learning - Model, Learning Criteria, and Optimization Algorithm

3. Optimization

The machine learning problem is transformed into an optimization problem

Once the training set is determined $\mathcal{D}$ , create space $\mathcal{F}$ and learning criteria, the next task is to find the optimal model through the optimization algorithm $\mathbf{x}, \boldsymbol{\theta}^*)$ . The training process of machine learning is essentially the process of solving optimization problems.

a. Parameters and hyperparameters

Optimization can be divided into two aspects: parameter optimization and hyperparameter optimization:

Number improvement: $(\mathbf{x}; \boldsymbol{\theta})$ 中的 $\boldsymbol{\theta}$ are called parameters of the model, and these parameters are learned through the optimization algorithm. These parameters can be updated iteratively through algorithms such as gradient descent to minimize the loss function.
Hyperparameter optimization: In addition to learnable parameters $\boldsymbol{\theta}$ , there is also a type of parameters used to define the model structure or optimization strategy. These parameters are called hyperparameters. For example, the number of categories in the clustering algorithm, the learning rate in the gradient descent method, the coefficient of the regularization term, the number of layers of the neural network, and the kernel function in the support vector machine are all hyperparameters. Unlike learnable parameters, the selection of hyperparameters is usually a combinatorial optimization problem and is difficult to automatically learn through optimization algorithms. Usually, the setting of hyperparameters is based on experience or continuous trial and error adjustment of a set of hyperparameter combinations through search methods.

b. Gradient descent method

In machine learning, one of the simplest and commonly used optimization algorithms is the gradient descent method. Gradient descent is used to minimize a function, usually a loss function or risk function. The gradient of this function with respect to the model parameters (weights) points in the direction where the function value increases the fastest. The gradient descent method uses this information to update the parameters so that the function value gradually decreases.

Iterative formula of gradient descent method

$\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \alpha \frac{\partial \mathcal{R}_{\mathcal{D}}(\boldsymbol{\theta})}{\partial \boldsymbol{\theta}}$

in:

$\boldsymbol{\theta}_t$ is the parameter value at the (t)th iteration.
$\alpha$ is the learning rate, which controls the step size of parameter update.
$\mathcal{R}_{\mathcal{D}}(\boldsymbol{\theta})$ is a risk function or a loss function, expressed in the training set ( \mathcal{D}).

The goal of the gradient descent method is to minimize the risk function by iteratively adjusting parameters.

Specific parameter update formula

The parameter update formula can be specified as:

$\boldsymbol{\theta}_{t+1} = \boldsymbol{\theta}_t - \alpha \frac{1}{N} \sum_{n=1}^{N} \frac{\partial \mathcal{L}(y^{(n)}, f(\mathbf{x}^{(n)}; \boldsymbol{\theta}))}{\partial \boldsymbol{\theta}}$

in:

$N$ is the number of samples in the training set.
$\mathcal{L}(y^{(n)}, f(\mathbf{x}^{(n)}; \ball symbol{\theta }))$ is the loss function, which represents the model pair sample $Prediction error for n$ .

Learning rate choice

Learning rate $\alpha$ is a key hyperparameter that affects the step size of parameter update. It is important to choose an appropriate learning rate. A learning rate that is too small may cause the convergence speed to be too slow, while a learning rate that is too large may cause the parameters to diverge during the optimization process.

An improvement on the gradient descent method is to use adaptive learning rate variants such as Adagrad, RMSprop, Adam, etc. These algorithms can automatically adjust the learning rate according to the historical gradient of the parameters, thereby more flexibly adapting to the update needs of different parameters.

c. Stochastic gradient descent

Insert image description here

Batch Gradient Descent (BGD)

In the batch gradient descent method, each iteration must calculate the gradient on the entire training set, and then update the model parameters, which results in overcomes the high computational cost and memory requirements on large-scale data sets. Its iterative update rules are as follows:

$\theta_{t+1} = \theta_t - \alpha \nabla \mathcal{R}_{\mathcal{D}}(\theta_t)$

in that, $\alpha$ 是学习率， $\nabla \mathcal{R}_{\mathcal{D}}( \theta_t)$ is the loss function on the entire training set with respect to the parameters $\theta_t$ gradient.

Stochastic Gradient Descent (SGD)

Stochastic gradient descent reduces computational cost by estimating the gradient using only one sample in each iteration. Its iterative update rules are as follows:

$\theta_{t+1} = \theta_t - \alpha \nabla \mathcal{L}(\theta_t, \mathbf{x}_i, y_i)$

使用， $\nabla \mathcal{L}(\theta_t, \mathbf{x}_i, y_i)$ This is the original version $(\mathbf{x}_i, y_i)$ is about the parameter $\theta_t$ gradient.

Mini-batch Gradient Descent

To trade off the computational cost and the accuracy of gradient estimation, mini-batch gradient descent is often used. This method uses a mini-batch of samples in each iteration to estimate the gradient, thus achieving both computational efficiency and gradient accuracy.

$\theta_{t+1} = \theta_t - \alpha \nabla \mathcal{L}_{\text{batch}}(\theta_t, \text{Batch})$

其中， $\nabla \mathcal{L}_{\text{batch}}(\theta_t, \text{Batch})$ This collection of books $\text{Batch}$ is about the parameters $\theta_t$ gradient.

Advantages of SGD

Computational efficiency: Compared with the batch gradient descent method, SGD has lower computational cost and is especially more practical on large-scale data sets.
Online learning: SGD has the property of online learning, requiring only one sample for each iteration, allowing the model to gradually adapt to new data.
Escape from the local minimum: Since the samples used in each iteration are different, SGD helps to escape from the local minimum, making it more likely to find the global optimal solution.

Challenges of SGD

Instability: The update of each iteration in SGD may be affected by a single sample, resulting in large fluctuations in the update direction.
Learning rate adjustment: Choosing an appropriate learning rate is crucial to the performance of SGD. A learning rate that is too large may cause instability, while a learning rate that is too small may make the model converge slowly.
Parameter adjustment required: The performance of SGD depends on the selection of hyperparameters such as learning rate and mini-batch size, and parameter adjustment is required.

In practice, techniques such as learning rate decay and momentum methods are usually used to improve the performance of SGD.