[Pytorch] study notes - the activation function

What is the activation function

Sentence Activation: is to let the neural network can describe nonlinear problems step, the neural network becomes more powerful

1. The activation function nonlinear factors are added to solve the problem can not be solved linear model.
2. excitation function taking into account the constraints received linear, curved linear function is breaking
3. Further it is actually a non-linear function. For example relu, sigmoid, tanh. These nested tool breaking bend in the original on the results, the original linear force to distort the results. also has an output y such nonlinear features.

Use common functions

1. Your neural network layer only two or three, not a lot of time, for the hidden layer, use any activation function, casually breaking the bend is possible, there will be particularly affected
2. When you use special multilayer neural networks, in turn breaking when the play are not free to choose the weapon because it would involve an explosion gradient, gradient go away.

Excitation function using scene

1. In a small layer structure, we can try many different activation function
2. Convolutional neural networks neural network convolution convolution layer, the recommended excitation function is RELU
3. Recurrent Neural Networks in circulation in neural networks, recommended is tanh or relu

Various activation functions 'breaking bend effect'

import torch
import torch.nn.functional as F     # 激励函数都在这
from torch.autograd import Variable
import matplotlib.pyplot as plt
# 做一些假数据来观看图像

# 函数的作用是，返回一个一维的tensor（张量），这个张量包含了从start到end，分成steps个线段得到的向量。常用的几个变量
x = torch.linspace(-5, 5, 200)#生成线性的空间
x = Variable(x)#转化成张量

x_np = x.data.numpy()   # 换成 numpy array, 出图时用

# 几种常用的 激励函数

y_relu = F.relu(x).data.numpy()
y_sigmoid = F.sigmoid(x).data.numpy()
y_tanh = F.tanh(x).data.numpy()
y_softplus = F.softplus(x).data.numpy()
# y_softmax = F.softmax(x)  softmax 比较特殊, 不能直接显示, 不过他是关于概率的, 用于分类

#将掰弯后的线性函数显示出来
plt.figure(1, figsize=(8, 6))
plt.subplot(221)
plt.plot(x_np, y_relu, c='red', label='relu')
plt.ylim((-1, 5))
plt.legend(loc='best')

plt.subplot(222)
plt.plot(x_np, y_sigmoid, c='red', label='sigmoid')
plt.ylim((-0.2, 1.2))
plt.legend(loc='best')

plt.subplot(223)
plt.plot(x_np, y_tanh, c='red', label='tanh')
plt.ylim((-1.2, 1.2))
plt.legend(loc='best')

plt.subplot(224)
plt.plot(x_np, y_softplus, c='red', label='softplus')
plt.ylim((-0.2, 6))
plt.legend(loc='best')

plt.show()