A tensor is a data structure that is very similar to arrays and matrices. A tensor is a data structure that is very similar to arrays and matrices.A tensor is a data structure very similar to arrays and matrices .
In PyTorch, we use tensors to encode the input and output of a model, as well as the parameters of the model .
1. Initialization of tensor
torch.tensor(data, dtype=None, device=None, requires_grad=False, pin_memory=False) → Tensor
·data: The data type of data can be a list, a tuple, a numpy array ndarray, a scalar (also called a scalar), and other data types.
·dtype: This parameter is optional and defaults to None. If not set, the generated Tensor data type will copy the data type of the parameter passed in data. For example, if the data type in data is float, the data type will be generated by default. A Tensor for torch.FloatTensor.
·device: This parameter is optional and defaults to None. If not set, memory will be allocated for the generated Tensor on the current device.
·requires_grad: This parameter is optional and defaults to False. If it is False, the created Tensor cannot perform gradient operations. If it is changed to True, the gradient can be calculated.
·pin_memory: This parameter is optional and defaults to False. If set to True, the current Tensor will be allocated in fixed memory, but it only applies to Tensors in the CPU.
设置require_grad参数的函数
Tensor.requires_grad_(requires_grad=True) → Tensor
Change if autograd should record operations on this tensor: sets this tensor’s requires_grad attribute in-place. Returns this tensor.
The role of with torch.no_grad
Under this module, the requirements_grad of all calculated tensors are automatically set to False, which can greatly save video memory.
Even if a tensor (named x) requires_grad = True, calculated in with torch.no_grad, the new tensor (named w) obtained from x requires_grad is False, and grad_fn is also None, that is, w will not be differentiated . An example is shown below:
x = torch.rand(10, 5, requires_grad = True)
y = torch.rand(10, 5, requires_grad = True)
with torch.no_grad():
w = x + y
print(w.requires_grad)
print(w.grad_fn)
#False
#None
data=[[1,2],[3,4]]
x_data=torch.tensor(data)
2. Converted from Numpy array
np_array=np.array(data)
x_np=torch.from_numpy(np_array)
t = torch.ones(5)
print(f"t: {
t}")
n = t.numpy()#反之Numpy array也可以由tensor转化而来
print(f"n: {
n}")
#t: tensor([1., 1., 1., 1., 1.])
#n: [1. 1. 1. 1. 1.]
t.add_(1)
print(f"t: {
t}")
print(f"n: {
n}")
#t: tensor([2., 2., 2., 2., 2.])
#n: [2. 2. 2. 2. 2.]
#CPU上的张量和NumPy数组可以共享它们的底层内存位置，所以改变一个的同时可以改变另一个。
3. Converted from the shape of another tensor
x_ones = torch.ones_like(x_data)
print(f"Ones Tensor: \n {
x_ones} \n")
x_rand = torch.rand_like(x_data, dtype=torch.float)
print(f"Random Tensor: \n {
x_rand} \n")
#Ones Tensor:
# tensor([[1, 1],
# [1, 1]])
#Random Tensor:
# tensor([[0.5900, 0.9701],
# [0.3483, 0.4137]])
4. Define tensor with random values or constants and with the help of shape tuples
shape = (2,3,)
rand_tensor = torch.rand(shape)
ones_tensor = torch.ones(shape)
zeros_tensor = torch.zeros(shape)
print(f"Random Tensor: \n {
rand_tensor} \n")
print(f"Ones Tensor: \n {
ones_tensor} \n")
print(f"Zeros Tensor: \n {
zeros_tensor}")
#Random Tensor:
# tensor([[0.8330, 0.5998, 0.8980],
# [0.6941, 0.3293, 0.7102]])
#
#Ones Tensor:
# tensor([[1., 1., 1.],
# [1., 1., 1.]])
#
#Zeros Tensor:
# tensor([[0., 0., 0.],
# [0., 0., 0.]])
torch.Tensor(data): Convert the input data into torch.FloatTensor()
torch.Tensor() can create an empty FloatTensor, but an error will be reported when using torch.tensor().
a=torch.tensor(1)
print(a)
print(a.type())
a=torch.Tensor(1)
print(a)
print(a.type())
#tensor(1)
#torch.LongTensor
#tensor([1.0010e-38])
#torch.FloatTensor
2. Properties of tensor
Tensor.shape[i] represents the value of the i-th dimension
tensor = torch.rand(3,4)
print(f"Shape of tensor: {
tensor.shape}")
print(f"Shape of tensor: {
tensor.shape[0]}")
print(f"Shape of tensor: {
tensor.shape[1]}")
print(f"Datatype of tensor: {
tensor.dtype}")
print(f"Device tensor is stored on: {
tensor.device}")
print(tensor.requires_grad) #表示autograd时是否需要计算此tensor的梯度，默认False
print(tensor.grad) #存储梯度的值，初始为None
print(tensor.grad_fn) #反向传播时，用来计算梯度的函数
print(tensor.is_leaf) #该张量节点在计算图中是否为叶子节点
'''
当这个 tensor 是用户创建的时候，它是一个叶子节点，
当这个 tensor 是由其他运算操作产生的时候，它就不是一个叶子节点
'''
print(f"mean of tensor: {
Tensor.mean(tensor)}")
print(f"variance of tensor: {
Tensor.var(tensor)}")
#Shape of tensor: torch.Size([3, 4])
#Shape of tensor: 3
#Shape of tensor: 4
#Datatype of tensor: torch.float32
#Device tensor is stored on: cpu
#False
#None
#None
#True
requires_grad: True if the gradient needs to be calculated for the tensor, False otherwise. When we create a tensor, we can specify requires_grad as True (default is False)
grad_fn: grad_fn is used to record how the variables come from to facilitate calculation of gradients. y = x*3, y.grad_fn records the process of calculating y from x. Directly create the leaf node with grad_fn=None
grad: After executing backward(), check the gradient value of x through x.grad.
For an in-depth discussion of requires_grad, grad, grad_fn, and is_leaf, refer to this article
3. Operation of tensor
1. Tensor is created on CPU by default, we need to use the .to method to explicitly move the tensor to GPU (after checking the availability of GPU)
if torch.cuda.is_available():
tensor = tensor.to("cuda")
2 Indexing, slicing, transpose (permute, transpose) operations
tensor = torch.ones(4, 4)
print(f"First row: {
tensor[0]}")
print(f"First column: {
tensor[:, 0]}")
print(f"Last column: {
tensor[..., -1]}")#此处用:或...都行
tensor[:,1] = 2
print(tensor)
#First row: tensor([1., 1., 1., 1.])
#First column: tensor([1., 1., 1., 1.])
#Last column: tensor([1., 1., 1., 1.])
#tensor([[1., 2., 1., 1.],
# [1., 2., 1., 1.],
# [1., 2., 1., 1.],
# [1., 2., 1., 1.]])
'''
def transpose(self, dim0, dim1): -> Tensor
def permute(self, *dims): -> Tensor
transpose：只能选择tensor中两个维度进行转置
permute：可以让tensor按照任意指定维度顺序进行转置
'''
tensor1=tensor.transpose(0,1)#例原本坐标为(0,1,1,0)的点就会变成(1,0,1,0)
tensor2=tensor.permute(1,0,3,2)#例原本坐标为(0,1,1,0)的点就会变成(1,0,0,1)
Note: But def transpose(self, *axes) in numpy: the method is to let numpy.array() transpose in any specified dimension order
3. Squeeze and unsqueeze operations, reducing dimensionality and increasing dimensionality
t o r c h . u n s q u e e z e ( i n p u t , d i m ) → T e n s o r \text torch.unsqueeze(input, dim) → Tensor torch.unsqueeze(input,dim)→Tensor
Returns a new tensor with a dimension of size one inserted at the specified position.
The returned tensor shares the same underlying data with this tensor.
A dim value within the range [-input.dim() - 1, input.dim() + 1) can be used. Negative dim will correspond to unsqueeze() applied at dim = dim + input.dim() + 1.
>>> x = torch.tensor([1, 2, 3, 4])#shape为4
>>> torch.unsqueeze(x, 0)#shape变为1×4
tensor([[ 1, 2, 3, 4]])
>>> torch.unsqueeze(x, 1)#shape变为4×1
tensor([[ 1],
[ 2],
[ 3],
[ 4]])
t o r c h . s q u e e z e ( i n p u t , d i m = N o n e ) → T e n s o r \text torch.squeeze(input, dim=None) → Tensor torch.squeeze(input,dim=None)→Tensor
Returns a tensor with all the dimensions of input of size removed.
For example, if input is of shape: ( A × 1 × B × C × 1 × D ) (A \times 1 \times B \times C \times 1 \times D) (A×1×B×C×1×D) then the out tensor will be of shape: ( A × B × C × D ) (A \times B \times C \times D) (A×B×C×D)
When dim is given, a squeeze operation is done only in the given dimension. If input is of shape: ( A × 1 × B ) (A \times 1 \times B) (A×1×B), squeeze(input, 0) leaves the tensor unchanged, but squeeze(input, 1) will squeeze the tensor to the shape ( A × B ) (A \times B) (A×B)
>>> x = torch.zeros(2, 1, 2, 1, 2)
>>> x.size()
torch.Size([2, 1, 2, 1, 2])
>>> y = torch.squeeze(x)
>>> y.size()
torch.Size([2, 2, 2])
>>> y = torch.squeeze(x, 0)#leaves the tensor unchanged
>>> y.size()
torch.Size([2, 1, 2, 1, 2])
>>> y = torch.squeeze(x, 1)
>>> y.size()
torch.Size([2, 2, 1, 2])
4. Arithmetic operation torch.matmul, ±*/
The basic four arithmetic operations
a + b are equivalent to torch.add()
a - b are equivalent to torch.sub()
a * b are equivalent to torch.mul()
a / b are equivalent to torch.div() for
direct addition, subtraction, multiplication, division, and exponentiation , square root torch.sqrt
a=10000**(torch.arange(0, 256, 2).float() / 256)
The * method operation is mul , and the / method operation is also similar and classified as follows
(1). The result of Tensor*scalar k is each element of Tensor multiplied by k.
>>> a = torch.ones(3,4)
>>> a
tensor([[1., 1., 1., 1.],
[1., 1., 1., 1.],
[1., 1., 1., 1.]])
>>> a * 2
tensor([[2., 2., 2., 2.],
[2., 2., 2., 2.],
[2., 2., 2., 2.]])
(2)Tensor* One-dimensional tensor
Tensor* The result of the row vector is that each column is multiplied by the value of the corresponding column of the row vector. The
result of Tensor* and the column vector is that each row is multiplied by the value of the corresponding row of the column vector.
>>> a = torch.ones(3,4)
>>> a
tensor([[1., 1., 1., 1.],
[1., 1., 1., 1.],
[1., 1., 1., 1.]])
>>> b = torch.Tensor([1,2,3,4])
>>> b
tensor([1., 2., 3., 4.])
>>> a * b
tensor([[1., 2., 3., 4.],
[1., 2., 3., 4.],
[1., 2., 3., 4.]])
>>> b = torch.Tensor([1,2,3]).reshape((3,1))
>>> b
tensor([[1.],
[2.],
[3.]])
>>> a * b
tensor([[1., 1., 1., 1.],
[2., 2., 2., 2.],
[3., 3., 3., 3.]])
(3)Tensor1*Tensor2
If the dimensions of the two tensors to be multiplied are inconsistent, but the dimensions of Tensor2 are equal to the latter dimensions of Tensor1, they can be multiplied.
When multiplying, Tensor2 will be expanded into the shape of Tensor1 in a repeated manner.
When the dimensions of Tensor1 and Tensor2 are the same, the values of the corresponding dimensions need to be the same. If the values of the corresponding dimensions are different, one of them must be 1, otherwise an error will be reported.
The + method operation is add , and the - method operation is also similar and classified as follows
(1) (2) The situation is the same as above
(3) Tensor1+Tensor2
If the dimensions of the two tensors added are inconsistent, but the dimensions of Tensor2 are the same as the latter dimensions of Tensor1 If the corresponding values are equal, they can be added.
When adding, Tensor2 will be expanded into the shape of Tensor1 in a repeated manner.
a = torch.ones([8, 4, 5, 6])
b = torch.ones([5, 6])
c = a+b
print(c.shape)#(8, 4, 5, 6)
a = torch.ones([8, 4, 5, 6])
b = torch.ones([8, 1, 5, 6])
c = torch.ones([8, 2, 5, 6])
a+b可以。相加过程也是先将b重复扩充为与a相同shape
a+c报错
matmul operation
y1 = tensor @ tensor.T
y2 = tensor.matmul(tensor.T)#类似矩阵乘法
y3 = torch.rand_like(y1)
torch.matmul(tensor, tensor.T, out=y3)
#y1,y2,y3是相同的
z1 = tensor * tensor
z2 = tensor.mul(tensor)#两个tensor的对应位置相乘
z3 = torch.rand_like(tensor)
torch.mul(tensor, tensor, out=z3)
#z1,z2,z3是相同的
#tensor([[1., 4., 1., 1.],
# [1., 4., 1., 1.],
# [1., 4., 1., 1.],
# [1., 4., 1., 1.]])
5、torch.reshape(input, shape) → Tensor
'''
Parameters
input (Tensor) – the tensor to be reshaped
shape (tuple of python:ints) – the new shape
A single dimension may be -1, in which case it’s inferred from the remaining dimensions and
the number of elements in input.
一个单一的维度可能是-1，在这种情况下，它是由其余的维度和输入的元素数量推断出来的。
'''
应用举例：假设当我们的dataloader的batch_size设置为64。并且经过卷积（out_channels=6）之后，
我们需要使用tensorboard可视化，而彩色图片的writer.add.images(output)的彩色图片是in_channels=3的。
那么则需要对卷积后的图片进行reshape，Torch.size(64,6,30,30)---->torch.size(-1,3,30,30)，-1的意思为最后自动计算其batch_size。
输出通道由6变成了3，从而每一个通道的数量增加，因而结果为torch.size(128,3,30,30)
6. Tensor.view(*shape) → Tensor, sometimes you need to add the contiguous() method before use
Returns a new tensor with the same data as the self tensor but of a different shape.
x = torch.randn(4, 4)
x.size()#torch.Size([4, 4])
y = x.view(16)
y.size()#torch.Size([16])
z = x.view(-1, 8) # the size -1 is inferred from other dimensions
z.size()#torch.Size([2, 8])
Key example: For a 3 × 4-dimensional tensor → t, perform reshape (t, (4, 3)) and t = t. transpose (− 1, − 2). Although the new dimensions obtained are consistent, among them The values at the corresponding positions are inconsistent. reshape and view simply fill in the values according to the new dimension. transpose is the key to transposing the matrix\color{red}. For example: for a 3×4 dimension tensor→t, reshape(t, (4,3)) and t=t.transpose(-1,-2), although the new dimensions obtained are consistent, the values at the corresponding positions are inconsistent. reshape and view simply fill in the values according to the new dimensions, and transpose is the Transpose matrixKey example: for a 3×4- dimensional t e n sor→t , res ha p e ( t ,(4,3 )) and t=t.transpose(−1,− 2 ) , although the new dimensions obtained are consistent, the values at the corresponding positions are inconsistent. res ha p e and vi e w simply fill in the values according to the new dimensions, and t r an s p ose is transposing the matrix.
7、torch.cat、torch.split
torch.cat(tensors, dim=0, *, out=None) → Tensor
Concatenates the given sequence of seq tensors in the given dimension. All tensors must either have the same shape (except in the concatenating dimension) or be empty.
Parameters:
tensors (sequence of Tensors) – any python sequence of tensors of the same type. Non-empty tensors provided must have the same shape, except in the cat dimension.
dim (int, optional) – the dimension over which the tensors are concatenated
dim=-1 means the first dimension from the last
>>> x = torch.randn(2, 3)
>>> x
tensor([[ 0.6580, -1.0969, -0.4614],
[-0.1034, -0.5790, 0.1497]])
>>> torch.cat((x, x, x), 0)#shape是6×3
tensor([[ 0.6580, -1.0969, -0.4614],
[-0.1034, -0.5790, 0.1497],
[ 0.6580, -1.0969, -0.4614],
[-0.1034, -0.5790, 0.1497],
[ 0.6580, -1.0969, -0.4614],
[-0.1034, -0.5790, 0.1497]])
>>> torch.cat((x, x, x), 1)#shape是2×9
tensor([[ 0.6580, -1.0969, -0.4614, 0.6580, -1.0969, -0.4614, 0.6580,
-1.0969, -0.4614],
[-0.1034, -0.5790, 0.1497, -0.1034, -0.5790, 0.1497, -0.1034,
-0.5790, 0.1497]])
8、torch.clamp(input, min=None, max=None, *, out=None) → Tensor
Clamps all elements in input into the range [ min, max ]. Letting min_value and max_value be min and max, respectively, this returns:
Calculation formula: yi = min ( max ( xi , min_value i ) , max_value i ) Calculation formula: y_i = \min(\max(x_i, \text{min\_value}_i), \text{max\_value} _i)Calculation formula: yi=min(max(xi,min_valuei),max_valuei)
means that the numbers outside the range of min and max are limited to min or max, and the numbers within the range remain unchanged.
If min is None, there is no lower bound. Or, if max is None there is no upper bound.
>>> a = torch.randn(4)
>>> a
tensor([-1.7120, 0.1734, -0.0478, -0.0922])
>>> torch.clamp(a, min=-0.5, max=0.5)
tensor([-0.5000, 0.1734, -0.0478, -0.0922])
>>> min = torch.linspace(-1, 1, steps=4)
>>> torch.clamp(a, min=min)
tensor([-1.0000, 0.1734, 0.3333, 1.0000])
9、torch.linspace(start, end, steps, *, out=None, dtype=None, layout=torch.strided, device=None, requires_grad=False) → Tensor
Creates a one-dimensional tensor of size steps whose values are evenly spaced from start to end, inclusive. That is, the value are:
( start , start + end − start steps − 1 , … , start + ( steps − 2 ) ∗ end − start steps − 1 , end ) (\text{start}, \text{start} + \frac{\text{end} - \text{start}}{\text{steps} - 1}, \ldots, \text{start} + (\text{steps} - 2) * \frac{\text{end} - \text{start}}{\text{steps} - 1}, \text{end}) (start,start+steps−1end−start,…,start+(steps−2)∗steps−1end−start,end)
>>> torch.linspace(3, 10, steps=5)
tensor([ 3.0000, 4.7500, 6.5000, 8.2500, 10.0000])
>>> torch.linspace(-10, 10, steps=5)
tensor([-10., -5., 0., 5., 10.])
>>> torch.linspace(start=-10, end=10, steps=5)
tensor([-10., -5., 0., 5., 10.])
>>> torch.linspace(start=-10, end=10, steps=1)
tensor([-10.])
10. repeat and expand
*Tensor.repeat(sizes) → Tensor
Repeats this tensor along the specified dimensions.
>>> x = torch.tensor([1, 2, 3])
>>> x.repeat(4, 2)#把x看作1×3了,然后1乘4,3乘2
tensor([[ 1, 2, 3, 1, 2, 3],
[ 1, 2, 3, 1, 2, 3],
[ 1, 2, 3, 1, 2, 3],
[ 1, 2, 3, 1, 2, 3]])
>>> x.repeat(4, 2, 1).size()#把x看作1×1×3了,然后1乘4,1乘2,3乘1
torch.Size([4, 2, 3])
*Tensor.expand(sizes) → Tensor
Returns a new view of the self tensor with singleton dimensions expanded to a larger size.
Passing -1 as the size for a dimension means not changing the size of that dimension.
Tensor can be also expanded to a larger number of dimensions, and the new ones will be appended at the front. For the new dimensions, the size cannot be set to -1.
>>> x = torch.tensor([[1], [2], [3]])
>>> x.size()
torch.Size([3, 1])
>>> x.expand(3, 4)
tensor([[ 1, 1, 1, 1],
[ 2, 2, 2, 2],
[ 3, 3, 3, 3]])
>>> x.expand(-1, 4) # -1 means not changing the size of that dimension
tensor([[ 1, 1, 1, 1],
[ 2, 2, 2, 2],
[ 3, 3, 3, 3]])
11、torch.sum(input, dim, keepdim=False, *, dtype=None)
Returns the sum of each row of the input tensor in the given dimension dim. If dim is a list of dimensions, reduce over all of them.
Parameters:
·input (Tensor) – the input tensor.
·dim (int or tuple of ints, optional) – the dimension or dimensions to reduce. If None, all dimensions are reduced.
·keepdim (bool) – whether the output tensor has dim retained or not.
Keyword Arguments:
·dtype (torch.dtype, optional) – the desired data type of returned tensor. If specified, the input tensor is casted to dtype before the operation is performed. This is useful for preventing data type overflows. Default: None.
>>>a=torch.arange(3*2*2).view(2,2,3)
>>>print(a)
tensor([[[ 0, 1, 2],
[ 3, 4, 5]],
[[ 6, 7, 8],
[ 9, 10, 11]]])
>>>b=torch.sum(a,(1,2))
>#把(0,0,0)、(0,0,1)、(0,0,2)、(0,1,0)、(0,1,1)、(0,1,2)的相加，以此类推
>>>print(b)
tensor([15, 51])
>>>c=torch.sum(a,0)#即把坐标为(0,0,0)和(1,0,0)的相加，以此类推
>>>print(c)
tensor([[ 6, 8, 10],
[12, 14, 16]])
12、torch.cumprod
>>>a=torch.Tensor([[0.1,0.2],[0.3,0.4]])
>>>b=torch.cumprod(a,dim=0)#第0维所以就是坐标为(0,0)和(1,0)的进行相乘
tensor([[0.1000, 0.2000],
[0.0300, 0.0800]])
>>>b=torch.cumprod(a,dim=1)
tensor([[0.1000, 0.0200],
[0.3000, 0.1200]])
13. torch.flatten(n), flatten the dimensions from the nth dimension onward.
agg = tensor.sum().item()
print(agg, type(agg))
tensor.add_(5)
#In-place operations：Operations that store the result into the operand are called in-place.
They are denoted by a _ suffix.