Derivatives of Scalars and Vectors

1. Scalar Derivatives

y	a (constant)	$${\mathrm{x}}^n$$	exp(x)	log(x)	sin(x)
$$\frac{dy}{dx}$$	0	$${\mathrm{n}\mathrm{x}}^n$$	exp(x)	$$\frac{1}{x}$$	$$\cos x$$

2. Vector derivatives

y	a	x	Ax	$$x^T$$ A	au	Au	u+v
$$\frac{\partial x}{\partial y}$$	0	I	Ax	$$x^T$$	$$a\frac{\partial u}{\partial y}$$	$$A\frac{\partial u}{\partial y}$$	$$\frac{\partial u}{\partial x}+\frac{\partial v}{\partial x}$$

3. Linear regression

3.1 Square loss

Assuming y is the true value, $\hat{y}$Is the estimated value, square loss: $$\ell (y,\hat{y})=\frac{1}{2}(y-\hat{y}{)}^2$$

3.2 Training data

Assuming there are n samples, record $$X=[x_1,x_2,...,x_n{]}^T$$ ,$$Y=[y_1,y_2,...,y_3{]}^T$$

3.3 Parameter Learning

训练损失：$$\ell (X,y,w,b)=\frac{1}{2n}\sum _{i=1}^n(y_i-<x_i,w>-b{)}^2=\frac{1}{2n}{||y-Xw-b||}^2$$

Minimize the loss to learn the parameters w*,b*:

$${\mathbf{w}}^{\ast },{\mathbf{b}}^{\ast }=\arg \underset{\mathbf{w},b}{\min }\ell (\mathbf{X},\mathbf{y},\mathbf{w},b)$$

3.4 Display solution

The infinitesimal equation
$\ell (X,y,w)=\frac{1}{2n}||y-w\mathbf{X}|{|}^2=\frac{\partial \ell (\mathbf{X},y,w)}{\partial \mathbf{w}}=\frac{1}{n}(y-\mathbf{X}w{)}^{T}X$

Since it is a convex function, the optimal solution satisfies the point where the derivative is equal to 0, that is:

$$\frac{\partial \ell (\mathbf{X},y,b)}{\partial \mathbf{w}}=0$$

$$\Rightarrow \frac{1}{n}(y-Xw{)}^{T}X=0$$

$$\Rightarrow w^{\ast }=({\mathbf{X}}^T\mathbf{X}{)}^{-1}{\mathbf{X}}^{T}y$$

3.5 Gradient Descent

pick an initial value $w_0$, repeat iteration parameters $$t=1,2,3,...$$ , making w obtain a minimum value:
$$W_t=W_{t-1}-the\frac{\partial \ell }{\partial W_{t-1}}$$

• The loss function value will increase along the gradient direction, so $$-\frac{\partial \ell }{\partial W_{t-1}}$$
• $\eta$ : learning rate, hyperparameter, control gradient descent step size

So for the hyperparameter-learning rate, how to choose the learning rate properly?

3.5.1 Mini-batch Gradient Descent

In the actual training process, gradient descent is rarely used directly, but a variant of gradient descent is used, for example: small batch random descent , because for each descent, the loss function is the average loss after deriving the entire sample. Therefore, to find a gradient, it is necessary to recalculate the entire sample set, and it may take several minutes or even hours in a deep neural network model. Therefore, the calculation cost of a gradient is huge.

So can we randomly sample b samples $$i_1,i_2,i_3,...$$ to find the approximate loss?

The answer is yes, this is mini-batch stochastic gradient descent:
$$\frac{1}{b}\sum _{i\in I_b}\ell ({\mathbf{X}}_i,{\mathbf{y}}_i,\mathbf{w})$$

in,

• b is the batch size, which is another important hyperparameter relative to the learning rate

So how to choose the batch size?

• Can not be too small: the amount of calculation is too small, not suitable for GPU parallel computing
• Can't be too large: too large increases memory consumption or wastes computation (when all samples are the same)

4. Summary

• Gradient descent is to continuously update the parameters to solve the problem by continuously approaching the point where the loss function is the smallest along the opposite direction of the gradient.
• The mini-batch stochastic gradient descent algorithm is the default solution algorithm for deep learning
• Two important hyperparameters are the learning rate $\eta$ and batch size b