Hands-on Deep Learning - Deep Learning Basics
- Deep Learning Basics: Linear Neural Networks, Multilayer Perceptrons
- Convolutional Neural Networks: LeNet, AlexNet, VGG, Inception, ResNet
- Recurrent Neural Networks: RNN, GRU, LSTM, seq2seq
- Attention mechanism: Attention, Transformer
- Optimization algorithms: SGD, Momentum, Adam
- High Performance Computing: Parallel, Multi-GPU, Distributed
- Computer Vision: Object Detection, Semantic Segmentation
- Natural Language Processing: Word Embeddings, BERT
1 Introduction to Deep Learning
Machine Learning is a powerful technique that can learn from experience. By taking the form of observational data or interacting with the environment, machine learning algorithms gain experience and their performance improves over time.
Parameter (Parameter) can be regarded as a knob, we can turn the knob to adjust the behavior of the program. After any procedure for adjusting parameters, it is called a model. The collection of all the different programs (input-output mappings) generated by manipulating parameters is called a "model family". A metaprogram that uses a dataset to select parameters is called a learning algorithm.
2 Data operation + data preprocessing
2.1 N-dimensional array
N-dimensional arrays are the main data structure for machine learning and neural networks
2.2 Create an array
Creating an array requires: shape, data type and value of each element
2.3 Accessing elements
2.4 Data manipulation
import torch
# 张量表示一个数值组成的数组，可能有多个维度
x = torch.arange(12)
print(x) # tensor([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11])
# 通过张量的shape属性来访问张量的形状和元素总数
print(x.shape) # torch.Size([12])
print(x.numel()) # 12
# reshape函数改变一个张量的形状而不改变元素数量和元素值
x = x.reshape(3, 4)
print(x)
'''
tensor([[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11]])
'''
# 创建全0或者全1的常量
x = torch.zeros((2, 3, 4))
print(x)
'''
tensor([[[0., 0., 0., 0.],
[0., 0., 0., 0.],
[0., 0., 0., 0.]],
[[0., 0., 0., 0.],
[0., 0., 0., 0.],
[0., 0., 0., 0.]]])
'''
x = torch.ones((2, 3, 4))
print(x)
'''
tensor([[[1., 1., 1., 1.],
[1., 1., 1., 1.],
[1., 1., 1., 1.]],
[[1., 1., 1., 1.],
[1., 1., 1., 1.],
[1., 1., 1., 1.]]])
'''
# 通过提供包含数值的Python列表为张量中的每个元素赋值
x = torch.tensor([[1, 2, 3], [4, 5, 6]])
print(x)
'''
tensor([[1, 2, 3],
[4, 5, 6]])
'''
# 将多个张量拼接起来
x = torch.arange(12, dtype=torch.float32).reshape((3, 4))
print(x)
'''
tensor([[ 0., 1., 2., 3.],
[ 4., 5., 6., 7.],
[ 8., 9., 10., 11.]])
'''
y = torch.tensor([[2.0, 1, 4, 3], [1, 2, 3, 4], [4, 3, 2, 1]])
print(y)
'''
tensor([[2., 1., 4., 3.],
[1., 2., 3., 4.],
[4., 3., 2., 1.]])
'''
# 按行拼接
z0 = torch.cat((x, y), dim=0)
print(z0)
'''
tensor([[ 0., 1., 2., 3.],
[ 4., 5., 6., 7.],
[ 8., 9., 10., 11.],
[ 2., 1., 4., 3.],
[ 1., 2., 3., 4.],
[ 4., 3., 2., 1.]])
'''
# 按列拼接
z1 = torch.cat((x, y), dim=1)
print(z1)
'''
tensor([[ 0., 1., 2., 3., 2., 1., 4., 3.],
[ 4., 5., 6., 7., 1., 2., 3., 4.],
[ 8., 9., 10., 11., 4., 3., 2., 1.]])
'''
# 即使形状不同，仍然可以通过广播机制来执行按元素操作
a = torch.arange(3).reshape((3, 1))
print(a)
'''
tensor([[0],
[1],
[2]])
'''
b = torch.arange(2).reshape((1, 2))
print(b)
'''
tensor([[0, 1]])
'''
print(a + b)
'''
tensor([[0, 1],
[1, 2],
[2, 3]])
'''
2.5 Data preprocessing
import os
import pandas as pd
import torch
os.makedirs(os.path.join('.', 'data'), exist_ok=True)
data_file = os.path.join('.', 'data', 'house_tiny.csv')
with open(data_file, 'w') as f:
f.write('NumRooms,Alley,Price\n')
f.write('NaN,Pave,127500\n')
f.write('2.0,NaN,106000\n')
f.write('4.0,NaN,178100\n')
f.write('NaN,NaN,140000\n')
data = pd.read_csv(data_file)
print(data)
inputs, outputs = data.iloc[:, 0:2], data.iloc[:, 2]
inputs = inputs.fillna(inputs.mean())
print(inputs)
'''
NumRooms Alley
0 3.0 Pave
1 2.0 NaN
2 4.0 NaN
3 3.0 NaN
'''
print(outputs)
'''
0 127500
1 106000
2 178100
3 140000
'''
# 对于inputs中的类别值或离散值，将NaN视为一个类别
inputs = pd.get_dummies(inputs, dummy_na=True)
print(inputs)
'''
NumRooms Alley_Pave Alley_nan
0 3.0 1 0
1 2.0 0 1
2 4.0 0 1
3 3.0 0 1
'''
x, y = torch.tensor(inputs.values), torch.tensor(outputs.values)
print(x, y)
'''
tensor([[3., 1., 0.],
[2., 0., 1.],
[4., 0., 1.],
[3., 0., 1.]], dtype=torch.float64)
tensor([127500, 106000, 178100, 140000])
'''
2.6 reshape and view are the same
reshape example
import torch
a = torch.arange(12)
print(a)
'''
tensor([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11])
'''
b = a.reshape((3, 4))
print(b)
'''
tensor([[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11]])
'''
b[:] = 2
print(a)
'''
tensor([2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2])
'''
print(b)
'''
tensor([[2, 2, 2, 2],
[2, 2, 2, 2],
[2, 2, 2, 2]])
'''
view example
import torch
a = torch.arange(12)
print(a)
'''
tensor([ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11])
'''
b = a.view((3, 4))
print(b)
'''
tensor([[ 0, 1, 2, 3],
[ 4, 5, 6, 7],
[ 8, 9, 10, 11]])
'''
b[:] = 2
print(a)
'''
tensor([2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2])
'''
print(b)
'''
tensor([[2, 2, 2, 2],
[2, 2, 2, 2],
[2, 2, 2, 2]])
'''
3 Linear algebra
3.1 Scalars
3.2 Vectors
3.3 Matrix
u = torch.tensor([3.0, 4.0])
# L2范数
print(torch.norm(u)) # tensor(5.)
# L1范数
print(torch.abs(u).sum()) # tensor(7.)
3.4 Summing by a specific axis
import torch
a = torch.ones((2, 5, 4))
print(a.shape) # torch.Size([2, 5, 4])
# 按照第0个维度求和
print(a.sum(axis=0).shape) # torch.Size([5, 4])
# 按照第1个维度求和
print(a.sum(axis=1).shape) # torch.Size([2, 4])
# 按照第2个维度求和
print(a.sum(axis=2).shape) # torch.Size([2, 5])
# 按照第0个维度求和，并且将第0个维度变为1
print(a.sum(axis=0, keepdims=True).shape) # torch.Size([1, 5, 4])
# 按照第1个维度求和，并且将第1个维度变为1
print(a.sum(axis=1, keepdims=True).shape) # torch.Size([2, 1, 4])
# 按照第2个维度求和，并且将第2个维度变为1
print(a.sum(axis=2, keepdims=True).shape) # torch.Size([2, 5, 1])
4 Matrix Calculation
4.1 Scalar derivatives
4.2 Subderivatives
4.3 Gradient
5 Automatic differentiation
5.1 Vector chain rule
5.2 Automatic Derivation
5.3 Two modes of automatic derivation
