pytorch进阶

使用pytorch构建神经网络系列

第二章 第二节 pytorch进阶

---

1.Broadcasting

key idea:
-Insert 1 dim ahead
-Expand dims with size 1 to same size
-Feature maps:[4,32, 14,14]
-Bias:[32,1,1] ==> [1,32,1,1] ==>[4,32, 14,14]

Match from Last dim！！

• If current dim = 1， expand to same
• if either has no dim， insert one dim and expand to same
• otherwise，NOT broadcasting-able

2.Merge/split

cat
合并（其他dim必须一致）

a = torch.rand(4,28,28)
b = torch.rand(5,28,28)
torch.cat([a,b],dim = 0).shape

torch.Size([9, 28, 28])

stack
creat new dim：

a = torch.rand(28,28)
b = torch.rand(28,28)
torch.stack([a,b],dim = 0).shape

torch.Size([2, 28, 28])

split

a = torch.rand(5,28,28)
aa, bb = a.split([2,3], dim = 0)
aa.shape, bb.shape

(torch.Size([2, 28, 28]), torch.Size([3, 28, 28]))

Chunk: by num

a = torch.rand(6,28,28)
aa, bb = a.chunk(2, dim = 0)
aa.shape, bb.shape

(torch.Size([3, 28, 28]), torch.Size([3, 28, 28]))

3.数学运算

matmul

=@

a = torch.full([2,2],8, dtype = torch.float)
b = torch.ones(2,2)
torch.matmul(a, b)
a@b

tensor([[16., 16.],
    [16., 16.]])

a = torch.rand(4, 784)
w = torch.rand(512, 784)
(a@w.t()).shape

torch.Size([4, 512])

实现并行相乘

a = torch.rand(4, 3, 28, 32)
b = torch.rand(4, 3, 32, 28)
torch.matmul(a, b).shape

torch.Size([4, 3, 28, 28])

Power

a = torch.full([2,2], 3, dtype = float)
a.pow(2)

tensor([[9., 9.],
    [9., 9.]], dtype=torch.float64)

aa = a**2
aa

tensor([[9., 9.],
    [9., 9.]], dtype=torch.float64)

aa.sqrt()
aa**(0.5)

tensor([[3., 3.],
    [3., 3.]], dtype=torch.float64)

aa.rsqrt() #平方根倒数

tensor([[0.3333, 0.3333],
    [0.3333, 0.3333]], dtype=torch.float64)

Exp / log

a = torch.exp(torch.ones(2,2)) # e的1次方
a

tensor([[2.7183, 2.7183],
    [2.7183, 2.7183]])

torch.log(a) #默认以e为底

tensor([[1.0000, 1.0000],
    [1.0000, 1.0000]])

Approximation

• .floor() .ceil() 向下取整 和向上取整
• .round() 四舍五入
• .trunc() .frac()整数部分与小数部分

a = torch.tensor(3.1415)
a.floor(),a.ceil(),a.trunc(),a.frac(),a.round()

(tensor(3.), tensor(4.), tensor(3.), tensor(0.1415), tensor(3.))

clamp

grad = torch.rand(2,3)*19
grad

tensor([[ 0.2305,  0.0978,  0.6458],
    [ 4.6174,  9.2011, 11.8627]])

最小值设置为10

grad.clamp(10)

tensor([[10.0000, 10.0000, 10.0000],
    [10.0000, 10.0000, 11.8627]])

设置在（0，10）区间

grad.clamp(5,10)

tensor([[ 5.0000,  5.0000,  5.0000],
    [ 5.0000,  9.2011, 10.0000]])

4.属性统计

norm

a = torch.full([8],1, dtype = float)
b = a.view(2,4)
c = a.view(2,2,2)
a,b,c

(tensor([1., 1., 1., 1., 1., 1., 1., 1.], dtype=torch.float64),
 tensor([[1., 1., 1., 1.],
         [1., 1., 1., 1.]], dtype=torch.float64),
 tensor([[[1., 1.],
          [1., 1.]],
 
         [[1., 1.],
          [1., 1.]]], dtype=torch.float64))

L1范数

a.norm(1),b.norm(1),c.norm(1)

(tensor(8., dtype=torch.float64),
 tensor(8., dtype=torch.float64),
 tensor(8., dtype=torch.float64))

L2范数

a.norm(2),b.norm(2),c.norm(2)

(tensor(2.8284, dtype=torch.float64),
 tensor(2.8284, dtype=torch.float64),
 tensor(2.8284, dtype=torch.float64))

指定维度：

b.norm(2,dim = 0)

tensor([1.4142, 1.4142, 1.4142, 1.4142], dtype=torch.float64)

mean, sum, min, max, prod(累积)

a = torch.arange(1,9).view(2,4).float()
a

tensor([[1., 2., 3., 4.],
        [5., 6., 7., 8.]])

a.mean(), a.sum(), a.min(), a.max(), a.prod()

(tensor(4.5000), tensor(36.), tensor(1.), tensor(8.), tensor(40320.))

最大值最小值索引

a.argmax(), a.argmin() #默认铺平 可以加上dim = ，指定维度

(tensor(7), tensor(0))
dim / keepdim

a = torch.rand(4,10)
a.max(dim = 1)

torch.return_types.max(
values=tensor([0.8951, 0.9522, 0.6955, 0.9705]),
indices=tensor([1, 6, 0, 9]))

a.max(dim = 1, keepdim = True)

torch.return_types.max(
values=tensor([[0.8951],
        [0.9522],
        [0.6955],
        [0.9705]]),
indices=tensor([[1],
        [6],
        [0],
        [9]]))

top-k/k-th

a.topk(3, dim =1, largest = True)

torch.return_types.topk(
values=tensor([[0.8951, 0.8779, 0.8563],
        [0.9522, 0.9264, 0.8587],
        [0.6955, 0.6641, 0.5106],
        [0.9705, 0.9102, 0.7651]]),
indices=tensor([[1, 0, 5],
        [6, 4, 9],
        [0, 7, 1],
        [9, 5, 8]]))

第k小

a.kthvalue(10, dim =1)

torch.return_types.kthvalue(
values=tensor([0.8951, 0.9522, 0.6955, 0.9705]),
indices=tensor([1, 6, 0, 9]))

compare
<,<=,>=,>,!=,==
返回1或0

5.Tensor advanced operation

where
torch.where(condition, x, y) ⇒ tensor

probility = torch.rand(2,2)
a = torch.zeros(2, 2)
b = torch.ones(2, 2)
probility,a,b

(tensor([[0.9443, 0.5826],
         [0.3918, 0.5968]]),
 tensor([[0., 0.],
         [0., 0.]]),
 tensor([[1., 1.],
         [1., 1.]]))

torch.where(probility > 0.5, b, a)

tensor([[1., 1.],
        [0., 1.]])

gather
查表
torch.gather(input, dim, index, out = None) ==> Tensor

prob = torch.randn(4,10)
idx = prob.topk(dim = 1, k = 2)
idx[1]

tensor([[3, 4],
        [5, 2],
        [1, 4],
        [6, 3]])

label = torch.arange(10) + 50
label

tensor([50, 51, 52, 53, 54, 55, 56, 57, 58, 59])

torch.gather(label.expand(4,10), dim = 1, index = idx[1])

tensor([[53, 54],
        [55, 52],
        [51, 54],
        [56, 53]])

代码内容出自网易云课程，本篇blog是对该课程进行简单记录和总结