note

Article directory

note
1. Creation of tensor
Two, tensor addition, subtraction, multiplication and division
3. torch.argmax() function
Four, gather function
- Chestnut 1
- Small chestnut 2: If each row needs to index multiple elements:
4. Operations for a certain dimension
5. Operations such as changing dimensions, splicing, and stacking
Reference

1. Creation of tensor

torch.tensorThe data will be copied, and you can use it if you don’t want to copy it torch.Tensor.detach().
If you want to get numpy array data, you can use torch.from_numpy()shared memory

# 1. tensor
torch.tensor(data, dtype=None, device=None,requires_grad=False)
data - 可以是list, tuple, numpy array, scalar或其他类型
dtype - 可以返回想要的tensor类型
device - 可以指定返回的设备
requires_grad - 可以指定是否进行记录图的操作，默认为False

# example 1
torch.tensor([[0.1, 1.2], [2.2, 3.1], [4.9, 5.2]])
tensor([[ 0.1000,  1.2000],
[ 2.2000,  3.1000],
[ 4.9000,  5.2000]])

# example 2
torch.tensor([0, 1]) # Type inference on data
tensor([ 0, 1])

# example 3
torch.tensor([[0.11111, 0.222222, 0.3333333]],
dtype=torch.float64,
device=torch.device(‘cuda:0’)) # creates a torch.cuda.DoubleTensor
tensor([[ 0.1111, 0.2222, 0.3333]], dtype=torch.float64, device=‘cuda:0’)

torch.tensor(3.14159) # Create a scalar (zero-dimensional tensor)
tensor(3.1416)

torch.tensor([]) # Create an empty tensor (of size (0,))
tensor([])

# 2. 从numpy中获得数据
torch.from_numpy(ndarry)

# 3. 创建特定数值的tensor
torch.zeros(*sizes, out=None, …)# 返回大小为sizes的零矩阵
1
torch.zeros_like(input, …) # 返回与input相同size的零矩阵
torch.ones(*sizes, out=None, …) #f返回大小为sizes的单位矩阵
torch.ones_like(input, …) #返回与input相同size的单位矩阵
torch.full(size, fill_value, …) #返回大小为sizes,单位值为fill_value的矩阵
torch.full_like(input, fill_value, …) 返回与input相同size，单位值为fill_value的矩阵
torch.arange(start=0, end, step=1, …) #返回从start到end, 单位步长为step的1-d tensor.
torch.linspace(start, end, steps=100, …) #返回从start到end, 间隔中的插值数目为steps的1-d tensor
torch.logspace(start, end, steps=100, …) #返回1-d tensor ，从10start到10end的steps个对数间隔

# 4. 随机生成
torch.normal(mean, std, out=None)
torch.rand(*size, out=None, dtype=None, …) #返回[0,1]之间均匀分布的随机数值
torch.rand_like(input, dtype=None, …) #返回与input相同size的tensor, 填充均匀分布的随机数值
torch.randint(low=0, high, size,…) #返回均匀分布的[low,high]之间的整数随机值
torch.randint_like(input, low=0, high, dtype=None, …) #
torch.randn(*sizes, out=None, …) #返回大小为size,由均值为0，方差为1的正态分布的随机数值
torch.randn_like(input, dtype=None, …)
torch.randperm(n, out=None, dtype=torch.int64) # 返回0到n-1的数列的随机排列

Two, tensor addition, subtraction, multiplication and division

torch.mm: For multiplication of two matrices (excluding vectors). If the dimensions are (l, m) and (m, n) multiplied
torch.bmm: Multiplication for 3D vectors with batches. If the dimensions are (b, l, m) and (b, m, n) multiplied
torch.mul: Used for pixel-by-pixel multiplication (dot product) of two matrices of the same dimension. If the dimensions are (l, m) and (l, m) multiplied
torch.mv: Used for multiplication between matrices and vectors (matrix first, vector last). If the dimension is (l,m) and (m) are multiplied, the dimension of the result is (l).
torch.matmul: It is used to multiply two tensors (the last two dimensions satisfy the dimension of matrix multiplication) or the multiplication between matrix and vector, because it has a broadcasting mechanism (broadcasting, automatically supplementing dimensions). For example, the dimensions are (b,l,m) and (b,m,n); (l,m) and (b,m,n); (b,c,l,m) and (b,c,m, n); (l, m) and (m) are multiplied, etc. [Its functions include torch.mm, torch.bmm and torch.mv]
@operator: it works liketorch.matmul
*operator: it works liketorch.mul
einsum(Einstein summation convention, that is, the Einstein summation convention) usage:
$c_{i k}=\sum_j a_{i j} b_{j k}$ is written as follows:

c = np.dot(a, b)                 # 常规
c = np.einsum('ij,jk->ik', a, b) # einsum

再微天 $c_{\mathrm{kl}}=\sum_{\mathrm{i}} \sum_{\mathrm{j}} \mathrm{a}_{\mathrm{ijk}} \mathrm {b}_{\mathrm{ijl}}$ ： c = np.einsum('ijk,jkl->kl', a, b)

# 对数运算
torch.log(input, out=None)  # y_i=log_e(x_i)
torch.log1p(input, out=None)  #y_i=log_e(x_i+1)
torch.log2(input, out=None)   #y_i=log_2(x_i)
torch.log10(input,out=None)  #y_i=log_10(x_i)

# 幂函数
torch.pow(input, exponent, out=None)  # y_i=input^(exponent)

# 指数运算
torch.exp(tensor, out=None)    #y_i=e^(x_i)
torch.expm1(tensor, out=None)   #y_i=e^(x_i) -1

3. torch.argmax() function

(1) torch.argmax(input, dim=None, keepdim=False)Return the serial number of the maximum value of the specified dimension;
(2) dimThe given definition is: the demention to reduce. That is, dimthe index of this dimension becomes the maximum value of this dimension.

import torch
a=torch.tensor([
              [
                  [1, 5, 5, 2],
                  [9, -6, 2, 8],
                  [-3, 7, -9, 1]
              ],
 
              [
                  [-1, 7, -5, 2],
                  [9, 6, 2, 8],
                  [3, 7, 9, 1]
              ]])
b=torch.argmax(a,dim=1)
print(a)
print(a.shape)
print(b)

(1) In this example, tensor(2, 3, 4), because it is dim=1to remove the second dimension, it becomes tensor(2, 4), and turns each 3x4 array into a 1x4 array.

[1, 5, 5, 2],
[9, -6, 2, 8],
[-3, 7, -9, 1]

A 3×4 matrix as shown above,Take the maximum value of each columnThe corresponding subscript, the row of the maximum value of the first column in a[0] is 1, the row of the maximum value of the second column is 2, the maximum value of the third column is 0, and the maximum value of the fourth column is The value row is marked as 1, so [1, 2, 0, 1] is finally output, and the maximum value of each column is taken, and the result is:

tensor([[[ 1,  5,  5,  2],
         [ 9, -6,  2,  8],
         [-3,  7, -9,  1]],

        [[-1,  7, -5,  2],
         [ 9,  6,  2,  8],
         [ 3,  7,  9,  1]]])
torch.Size([2, 3, 4])
tensor([[1, 2, 0, 1],
        [1, 0, 2, 1]])

(1) If it is changed dim=2, the third dimension will be removed,That is, take the maximum value of each rowThe corresponding subscript results in tensor(2, 3).

import torch
a=torch.tensor([
              [
                  [1, 5, 5, 2],
                  [9, -6, 2, 8],
                  [-3, 7, -9, 1]
              ],
 
              [
                  [-1, 7, -5, 2],
                  [9, 6, 2, 8],
                  [3, 7, 9, 1]
              ]])
b=torch.argmax(a,dim=2)
print(b)
print(a.shape)
"""
tensor([[2, 0, 1],
        [1, 0, 2]])
torch.Size([2, 3, 4])
"""

Four, gather function

torch.gather(input, dim, index, *, sparse_grad=False, out=None) → Tensor

torch.gather()Function: Use index to index the value at a specific position of the input
dim = 1to indicate the horizontal direction.

For a three-dimensional tensor, its output is:

out[i][j][k] = input[index[i][j][k]][j][k]  # if dim == 0
out[i][j][k] = input[i][index[i][j][k]][k]  # if dim == 1
out[i][j][k] = input[i][j][index[i][j][k]]  # if dim == 2

Chestnut 1

For example, there are now 4 sentences (sentences of different lengths), the current sequence labeling problem needs to be labeled with a label for each word, and the labels are as follows:

input = [
    [2, 3, 4, 5],
    [1, 4, 3],
    [4, 2, 2, 5, 7],
    [1]
]

The lengths are 4, 3, 5, 1, respectively, where the labels of the first sentence are 2, 3, 4, 5. In NLP, it is generally necessary to pad sentences of different lengths to the same length (using 0 for padding), so the result after padding:

input = [
    [2, 3, 4, 5, 0, 0],
    [1, 4, 3, 0, 0, 0],
    [4, 2, 2, 5, 7, 0],
    [1, 0, 0, 0, 0, 0]
]

import torch
input = [
    [2, 3, 4, 5, 0, 0],
    [1, 4, 3, 0, 0, 0],
    [4, 2, 2, 5, 7, 0],
    [1, 0, 0, 0, 0, 0]
]
input = torch.tensor(input)
length = torch.LongTensor([[4], [3], [5], [1]])
# index之所以减1，是因为序列维度从0开始计算的
out = torch.gather(input, 1, length - 1)
print(out)

The result of out is as follows. For example, the first line of length is [4], that is, the fourth element of the first line of input is found to be 5 (here, length-1the subscript is calculated from 1).

tensor([[5],
        [3],
        [7],
        [1]])

Small chestnut 2: If each row needs to index multiple elements:

>>> t = torch.Tensor([[1,2],[3,4]])

1  2
3  4
>>> torch.gather(t,1,torch.LongTensor([[0,0],[1,0]])
1  1
4  3
[torch.FloatTensor of size 2x2]

4. Operations for a certain dimension

mean
softmax
BN
LN

5. Operations such as changing dimensions, splicing, and stacking

import torch
x = torch.arange(12)
# tensor([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11])
x1 = x.reshape(3, 4)  # 改变维度
x2 = x.reshape(-1, 4)
x3 = torch.zeros((2, 3, 4))
x4 = torch.ones((2, 3, 4)) # 所有元素都为1
# 正态分布
x5 = torch.randn(3, 4)
x6 = torch.tensor([[2, 1, 4, 3], [1, 2, 3, 4], [4, 3, 2, 1]])

x = torch.tensor([1.0, 2, 4, 8])
y = torch.tensor([2, 2, 2, 2])
# 都是按元素操作,注意**是求幂运算
print(x + y, x - y, x * y, x / y, x ** y)

X = torch.arange(12, dtype=torch.float32).reshape((3,4))
Y = torch.tensor([[2.0, 1, 4, 3], [1, 2, 3, 4], [4, 3, 2, 1]])
# 每行（上下）拼接, dim=1为左右拼接
print(torch.cat((X, Y), dim=0), "\n", torch.cat((X, Y), dim=1))

# 判断每个位置是否相同
X == Y

# 广播机制, 两个矩阵维度不同（数学上不能按元素相加），通过广播（a赋值列，b赋值行）后相加
a = torch.arange(3).reshape((3, 1))
b = torch.arange(2).reshape((1, 2))
print(a + b)

# 切片和索引， 和numpy差不多
X[-1], X[1:3]
X[1, 2]
X[0:2, :] = 12  # 赋值

viewTo change the dimension, you can pass a parameter of -1 in one of the dimensions, and it will be calculated automatically.

import torch
a = torch.arange(1, 7)
print(a)

b = a.view(2, 3)
print(b)

c = a.view(3, -1)
print(c)

flattenFlattening operation

input1 = torch.tensor(range(2*3*4*5)).view(2, 3, 4, 5)
# input1.shape
torch.flatten(input1, start_dim = 1, end_dim=2).shape
# torch.Size([2, 12, 5])

repeat_interleaveIt is to copy the elements in the tensor n times along a certain dimension

import torch

x = torch.tensor([[1, 2, 3],[4,5,6]])

x1 = x.repeat_interleave(3,0)
print("x1:\n", x1)

x2 = x.repeat_interleave(3,1)
print("x2:\n",x2)

x1:
 tensor([[1, 2, 3],
        [1, 2, 3],
        [1, 2, 3],
        [4, 5, 6],
        [4, 5, 6],
        [4, 5, 6]])
x2:
 tensor([[1, 1, 1, 2, 2, 2, 3, 3, 3],
        [4, 4, 4, 5, 5, 5, 6, 6, 6]])

Process finished with exit code 0

Other functions:

torch.lerp(star, end, weight) : 返回结果是out= star t+ (end-start) * weight
torch.rsqrt(input) : 返回平方根的倒数
torch.mean(input) : 返回平均值
torch.std(input) : 返回标准偏差
torch.prod(input) : 返回所有元素的乘积
torch.sum(input) : 返回所有元素的之和
torch.var(input) : 返回所有元素的方差
torch.tanh(input) ：返回元素双正切的结果
torch.equal(torch.Tensor(a), torch.Tensor(b)) ：两个张量进行比较，如果相等返回true
torch.max(input)： 返回输入元素的最大值
torch.min(input) ： 返回输入元素的最小值
element_size() ：返回单个元素的字节
torch.from_numpy(obj)，利用一个numpy的array创建Tensor。注意，若obj原来是1列或者1行，无论obj是否为2维，所生成的Tensor都是一阶的，若需要2阶的Tensor，需要利用view()函数进行转换。
torch.numel(obj)，返回Tensor对象中的元素总数。
torch.ones_like(input)，返回一个全1的Tensor，其维度与input相一致
torch.cat(seq, dim)，在给定维度上对输入的张量序列进行连接操作
torch.chunk（input, chunks, dim）在给定维度(轴)上将输入张量进行分块
torch.squeeze(input)，将input中维度数值为1的维度去除。可以指定某一维度。共享input的内存
torch.unsqeeze(input, dim)，在input目前的dim维度上增加一维
torch.clamp(input, min, max)，将input的值约束在min和max之间
torch.trunc(input)，将input的小数部分舍去

insert image description here

Reference

[1] Thoroughly understand the operations of a certain dimension in torch.tensor (mean, Softmax, batch norm, layer norm)

[Pytorch Basic Tutorial 39] Torch commonly used tensor processing functions