Deep Learning - Li Hongyi's first lesson 2020

Li Hongyi Deep Learning Course

Predict Pokémon's combat power

Regression

• Market Forecast - Predict what the stock price will be tomorrow?
• self-driving car - predict the steering wheel angle
• Recommendation - Purchase Likelihood (Recommendation System)

$$f\left(x\right(Pokémon\left)\right)\_\_=y{}^{\prime }CPafterevolution{}^{\prime }$$

$x_{cp}:Combat\; powerbeforeevolution,x{}_s:speciesx{\_}_{hp}:health,x\_{}_w:weight,x_h:height$

Step 1. Model

A set of function … ————→ Model ( f1, f2, f3 … )

linear Model :
$$y=b+w\cdot x_{cp}$$
$w$ and $b$ are parameters (can be any value)
$$y=b+\sum w_i{x}_i$$
$x_i$ : an attribute of input $X$ (feature). —— X X Various properties of$X$

$w_i$ : weight

$b$ : bias

Step 2. Goodness of function

function input : function output (scalar) :

​ $x^1,x^2,x^3...$ ${\hat{y}}^1,{\hat{y}}^2,{\hat{y}}^3...$

Loss function L :

• input : a function
• output : how bad it is

$$L(f)=L(w,b)=\sum _{n=1}^{10}({\hat{y}}^n-(b+w\cdot x_{cp}^n){)}^2$$

estimation error : estimated $y$ based on input function .

In measuring a set of $w$, $b$ is good or bad.

Step 3. Gradient Descent

Best function :

( pick the ‘‘best’’ function)
$$\begin{gathered} f^{\ast }=argmi{n}_{f}L(f) \\ {w}^{\ast },b^{\ast }=argmi{n}_{w,b}L(w,b)=argmi{n}_{w,b}\sum _{n=1}^{10}({\hat{y}}^n-(b+w\cdot x_{cp}^n){)}^2 \end{gathered}$$
Consider loss function $L(w)$ with one parameter $w$ .
$$w^{\ast }=argmi{n}_{w}L\left(w\right)$$
Differentiable numbers can be passed back for gradient descent

• ( Randomly ) Pick an initial value $w^0$
• compute

$$\begin{gathered} \frac{dL}{dw}{|}_{w=w^0} \\ -the\frac{dL}{dw}{|}_{w=w^0} \\ {w}^1=w^0-the\frac{dL}{dw}{|}_{w=w^0} \end{gathered}$$

$\eta$ is called ‘learning rate’ .

• Many iteration

Local optimal, not global optimal .

How obout two parameters ?
$$w^{\ast },b^{\ast }=argmi{n}_{w,b}L(w,b)$$

• ( Randomly ) Pick an initial value $w^0,b^0$
• compute

$$\begin{gathered} \frac{dL}{dw}{|}_{w=w^0,b=b^0} \\ \frac{dL}{db}{|}_{w=w^0,b=b^0} \\ {w}^1=w^0-the\frac{dL}{dw}{|}_{w=w^0,b=b^0} \\ {b}^1=b^0-the\frac{dL}{db}{|}_{w=w^0,b=b^0} \end{gathered}$$

$$▽={\left[\begin{array}{r}\frac{\alpha L}{aw}\\ \frac{\alpha L}{\alpha b}\end{array}\right]}_{gradient}$$

The linear regression, the loss function $L$ is convex. ( No local optimal )

Formulation of $\frac{\alpha L}{aw}$ and $\frac{\alpha L}{\alpha b}$
$$\begin{gathered} L(w,b)=\sum _{n=1}^{10}({\hat{y}}^n-(b+w\cdot x_{cp}^n){)}^2 \\ \frac{\alpha L}{aw}=\sum _{n=1}^{10}2({\hat{y}}^n-(b+w\cdot x_{cp}^n))(-x_{cp}^n) \\ \frac{\alpha L}{\alpha b}=\sum _{n=1}^{10}2({\hat{y}}^n-(b+w\cdot x_{cp}^n))(-1) \end{gathered}$$

Model Selection

$$\begin{gathered} Model1:y=w_{1}x+b \\ Model2:y=w_{1}x+w_{2}x+b \\ Model3:y=w_{1}x+w_{2}x+w_{3}x+b \\ ... \end{gathered}$$

A more complex model does not always lead to better performance on testing data. This is overfitting.

Let’s collect more data. There is more hidden factors influence the previous model. : the type of pokeman

Back to Step 1: Redesign the Model

$x_s$ = species of $x$

X ——→
$$\begin{gathered} if{x}_s=Pidgey:y=b_1+w_1\cdot x_{cp} \\ if{x}_s=Weedle:y=b_2+w_2\cdot x_{cp} \\ if{x}_s=Caterpie:y=b_3+w_3\cdot x_{cp} \\ if{x}_s=Eevee:y=b_4+w_4\cdot x_{cp} \end{gathered}$$
——→ $y$
$$\begin{gathered} y=b_{1}d(x_s=Pidey)+w_1\cdot d(x_s=Pidey)x_{cp} \\ +... \\ +b_{4}d(x_s=Eevee)+w_4\cdot d(x_s=Eevee)x_{cp} \end{gathered}$$

$$d(x_s=Pidey)=\{\begin{array}{rc}1,& if{x}_s=Pidey\\ 0,& otherwise\end{array}$$

Are there any other hidden factors?

Back to Step 2: Regulazation

$$y=b+\sum w_i{x}_i$$

$$L=\sum _n({\hat{y}}^n-(b+\sum w_i{x}_i){)}^2+l\sum (w_i{)}^2$$

training error + regularization

$b$ has nothing to do with the smoothness of the function, so bbis not considered when regularizing$b$

• The functions with smaller $w_i$ are better. $w_i$Koshi Kogoshi smooth.
• Training error: larger $\lambda$ , considering the training error less.

$The\; larger\; the\; \lambda$ , the smoother it is, but it cannot be too smooth

why smooth function are preferred?

The smoothing function has little effect on input clutter. if some noises corrupt input $x_i$ when testing, a smooth function has less influence.

where are the errors from?

• bias
• variance

simpler model is less influenced by the sample data.

• simple model → small variance, large bias ( underfitting )
• complex model → large variance, small bias ( overfitting )

Complex models contain simple models

For bias, redesign your model:

• add more features as input
• a more complex model

what to do with large variance?

• more data (collect real data, generate fake data) —— very effective, but not always practical
• regularization