Numerical stability gradient explosion gradient disappearance + model initialization and activation function hands-on deep learning v2 pytorch

Enterprise 2022-04-05 02:43:44 views: null

1. Numerical stability gradient explosion gradient disappears

insert image description here

insert image description here

insert image description here

2. Make training more stable

insert image description here

f(x) = x

insert image description here

3. QA

nan is generally caused by dividing by 0; inf means infinity, or infinitesimal
sigmoid will easily cause the gradient to disappear, because the value of sigmoid is in the range(0,1)
The normal distribution is easier to derive
4 * sigmoid(x) - 2can improve stability, because when near 0, makingf(x)=x

refer to

https://www.bilibili.com/video/BV1u64y1i75a?p=1

Guess you like

Origin blog.csdn.net/zgpeace/article/details/123932629

Numerical stability gradient explosion gradient disappearance + model initialization and activation function hands-on deep learning v2 pytorch

[Hands-on Deep Learning v2 Li Mu] Study Notes 09: Numerical Stability, Model Initialization, Activation Function

Numerical stability of deep learning models - explanation of gradient decay and gradient explosion

Deep learning - gradient disappearance, gradient explosion

Suppress gradient exception initialization parameters (to prevent gradient disappearance and gradient explosion)

Python deep learning 027: what is gradient, gradient disappearance, gradient explosion and how to solve it

The principle of Batch Normalization and gradient disappearance and gradient explosion

Softmax regression + loss function + image classification dataset hands-on deep learning v2 pytorch

(2) PyTorch Deep Learning: Gradient Descent

Let’s talk about gradient disappearance and gradient explosion again

"Hands-on Deep Learning Pytorch Edition" 5.3 Delayed initialization

[Deep Learning] Chapter 5: Looking back at the network architecture - activation function - loss function - gradient descent

Gradient of activation function and Loss

CNN deep convolutional neural network AlexNet hands-on deep learning v2 pytorch

Network with parallel connections GoogLeNet / Inception V3 hands-on deep learning v2 pytorch

Hands-on learning and deep learning -Softmax classification model _pytorch

(3) PyTorch Deep Learning: Backpropagation Gradient Descent

Multi-layer perceptron + code implementation - hands-on deep learning v2 pytorch

Network in the network NiN hands-on deep learning v2 pytorch

Hands-on deep learning v2 pytorch using block network VGG

Classic convolutional neural network LeNet hands-on deep learning v2 pytorch

CNN convolutional neural network pooling layer Pooling hands-on deep learning v2 pytorch

Dropout drop method hands-on deep learning v2 pytorch

Weight decay Weight Decade hands-on deep learning v2 pytorch

Padding and stride stride in convolutional layers hands-on deep learning v2 pytorch

Actual Kaggle Competition: Dog Breed Recognition (ImageNet Dogs) Hands-on Deep Learning v2 pytorch

Object detection object detection object recognition and coco dataset hands-on deep learning v2 pytorch

Gradient calculating backward learning function Pytorch

Gradient disappears, gradient explosion

Gradient or gradient disappears explosion

Recommended

The United States plans to restrict the export of large AI models to China and Russia

Apple to reach agreement with OpenAI to bring ChatGPT to iPhone

Ranking

whisper-webui installation tutorial is silky and easy to use

[Base] Laravel concepts laravel basis, the custom service provider: Contracts, ServiceContainer, ServiceProvider, Facades relations

Import torchvision error problem solving DLL: module not found

observer & watch & notify = pub & sub

A small turntable program [HTML + CSS + JS]

CorelDRAW 2018 shortcuts Daquan

Supervise el botón de menú para lograr un gatillo de presión prolongada

JS将时间秒转换成天小时分钟秒的字符串

RIP basic configuration

[Deleted] solution to a problem a few questions (Noip1994)

Daily

More

2024-05-11(32)

2024-05-10(34)

2024-05-09(32)

2024-05-08(18)

2024-05-07(34)

2024-05-06(6)

2024-05-05(0)

2024-05-04(18)

2024-05-03(8)

2024-05-02(0)