The Eye of Depth Pytorch Punch (2): Pytorch Tensor and Tensor Creation

Foreword

Tensor is not only a basic data structure in Pytorch, which represents a multi-dimensional array, but also a way to automatically find gradients. The framework of this note is mainly derived from the depth of the eye , and has made some related expansions. The content includes: the meaning of tensors, the shape of 0-dimensional, 1-dimensional, 2-dimensional, 3-dimensional, 4-dimensional, 5-dimensional tensors; several parameters of tensors and the method of automatically finding gradients; several commonly used methods of creating tensors.

Pytorch tensor

In various deep learning frameworks, tensors are generally defined as multi-dimensional arrays, with the purpose of pushing vectors and matrices to higher dimensions. Tensors in Pytorch are both basic data structures and tools for automatically finding gradients.

Tensor

Scalars, vectors, matrices, etc. are all tensors. The scalar is a 0-dimensional tensor, the vector is a 1-dimensional tensor, and the matrix is a 2-dimensional tensor. The grayscale images we usually see are 2-dimensional tensors, and the color images are 3-dimensional tensors. A number of three-dimensional tensors can constitute a four-dimensional tensor, a number of four-dimensional tensors can constitute a five-dimensional tensor, and so on. As shown in the following figure (modified from a blog , this blog system introduces the various meanings of tensor)

Variable

Variable is a data type in torch.autograd, which is mainly used to encapsulate Tensor for easy automatic derivation. In Pytorch 0.4.0 and later, Variable was incorporated into Tensor. torch.autograd.Variable contains the following 5 attributes:

data: used to store tensor, is the data ontology.

grad: The gradient value of data.

grad_fn: record the method used when creating the tensor ( function -multiplication: addition :) , used for back propagation gradient calculation . grad_fn points to the function that created the Tensor, if a certain Tensor was created by the user, it points to: None .grad_fn = <MulBackward0>,grad_fn = <AddBackward0>

requires_grad: indicates whether a gradient is required. For a tensor that requires guidance, the requirements_grad attribute must be True, that is: the requirements_grad attribute of the requires = Ture.self-defined tensor is False by default, and the requirements_grad attribute of the tensor of the weight w in the neural network layer defaults to True .

is_ leaf: indicates whether it is a leaf node (tensor), the node created by the user is called a leaf node (tensor), such as X and W. is_leaf is False, that is is_leaf = Ture,, it is not a leaf node when : is_leaf is a leaf node when is_leaf is True. The gradient value of the leaf node can be retained after back propagation, and the gradient value of the non-leaf node is: None .

Pytorch tensor

As mentioned earlier, in Pytorch 0.4.0 and later versions, Tensor already contains Variable content, so it should contain all the properties of Variable. The Pytorch tensor has 8 attributes as shown in the figure below, 5 of which are as above, and the remaining 3 are as follows:

dtype: Tensor data type, divided into three categories of floating point, integer and Boolean, a total of 9 data types, as shown in the following table (quoted from pytorch official website ), of which 32-bit floating-point and 64-bit integer are the most commonly used. Image preprocessing result default: torch.float32，image label default:torch.int64.

type of data	dtype	CPU tensor	GPU tensor
32-bit floating point	`torch.float32` or `torch.float`	`torch.FloatTensor`	`torch.cuda.FloatTensor`
64-bit floating point	`torch.float64` or `torch.double`	`torch.DoubleTensor`	`torch.cuda.DoubleTensor`
16-bit floating point	`torch.float16` or `torch.half`	`torch.HalfTensor`	`torch.cuda.HalfTensor`
8-bit integer (unsigned)	`torch.uint8`	`torch.ByteTensor`	`torch.cuda.ByteTensor`
8-bit integer (signed)	`torch.int8`	`torch.CharTensor`	`torch.cuda.CharTensor`
16-bit integer (signed)	`torch.int16` or `torch.short`	`torch.ShortTensor`	`torch.cuda.ShortTensor`
32-bit integer (signed)	`torch.int32` or `torch.int`	`torch.IntTensor`	`torch.cuda.IntTensor`
64-bit integer (signed)	`torch.int64` or `torch.long`	`torch.LongTensor`	`torch.cuda.LongTensor`
Boolean	`torch.bool`	`torch.BoolTensor`	`torch.cuda.BoolTensor`

shape: The shape of the tensor. For example, (32,3,448,448) is a 32 * 3 * 448 * 448 4-dimensional tensor, which is equivalent to 32 RGB images of 448 * 448. The stacking method can be derived from the above tensor diagram.
device: The device where the tensor is located, GPU or CPU, and the tensor can only be accelerated on the GPU.

Commonly used Pytorch tensor creation methods

★torch.tensor()

torch.tensor(data, dtype=None, device=None, requires_grad=False, pin_memory=False)

data: The initial data of the tensor. It can be list, tuple, NumPy's n-dimensional array ( ndarray ), scalar and other types.

dtype: Returns the data type of the tensor. The default value is None, which means it is consistent with the data data type.

device: The required device to return the tensor. Default value: None, which means use the current device. If it is a CPU tensor, the device is the current CPU, and if it is a cuda tensor, the device is the current GPU.

requires_grad: whether to calculate the gradient, the default is False.

pin_memory: Whether the returned tensor is allocated in fixed memory, only applicable to CPU tensor, which involves efficiency issues. Default value: False.

Code:

# 输入列表，数据类型是int64
a = [[1, 2, 3, 4], [1, 2, 3, 4], [1, 2, 3, 4], [1, 2, 3, 4]]
t = torch.tensor(a)
print(t, '\n', t.dtype)

result:

tensor([[1, 2, 3, 4],
        [1, 2, 3, 4],
        [1, 2, 3, 4],
        [1, 2, 3, 4]]) 
 torch.int64

Code:

# 输入列表，数据类型是float32
a = [[1.2, 2, 3, 4], [1, 2, 3, 4], [1, 2, 3, 4], [1, 2, 3, 4]]
t = torch.tensor(a)
print(t, '\n', t.dtype)

result:

tensor([[1.2000, 2.0000, 3.0000, 4.0000],
        [1.0000, 2.0000, 3.0000, 4.0000],
        [1.0000, 2.0000, 3.0000, 4.0000],
        [1.0000, 2.0000, 3.0000, 4.0000]]) 
 torch.float32

Code:

# 输入一个浮点数
a = 3.1415926535
t = torch.tensor(a, dtype=torch.int64)
print(t, '\n', t.dtype)

result:

tensor(3) 
 torch.int64

Code:

# 输入一个浮点数，设备选择cuda，由于要从cpu先转换到GPU，所以结果输出较慢
a = 3.1415926535
t = torch.tensor(a, dtype=torch.float64, device='cuda')
print(t, '\n', t.dtype)

Result: Different GPUs can be selected according to cuda number

tensor(3.1416, device='cuda:0', dtype=torch.float64) 
 torch.float64

torch.from_numpy()

torch.from_numpy(ndarray)

The tensor returned by torch.from_numpy (ndarray) and ndarray share the same memory , ie: modifications to the tensor will be reflected on the ndarray, and vice versa. And the returned tensor cannot be resized. Currently, the function accepts dtypes as numpy.float64, numpy.float32, numpy.float16, numpy.int64, numpy.int32, numpy.int16, numpy.int8, numpy.uint8, and the ndarray.numpy.bool，

Code:

a = np.array([1, 2, 3, 4])
t = torch.from_numpy(a)
print(t)
t[0] = 10
print('modify t print a:', a)
a[3] = -5
print('modify a print t:', t)

Result: The shared memory is confirmed

tensor([1, 2, 3, 4], dtype=torch.int32)
modify t print a: [10  2  3  4]
modify a print t: tensor([10,  2,  3, -5], dtype=torch.int32)

★torch.zeros()

torch.zeros(*size, out=None, dtype=None, layout=torch.strided, device=None, requires_grad=False)

size: The shape of the tensor, which is a sequence of integers, such as (448,448).

out: output tensor to the address pointed by out

layout : The layout of tensors in memory, currently supported torch.strided(dense tensors), and provide experimental support torch.sparse_coo(sparse COO tensors), default:torch.strided,it is also the most commonly used layout form.

Code:

out_t = torch.tensor(3.14)
t = torch.zeros(4, out=out_t)
print(t.dtype)
print('address of t:', id(t), '\n', t)
print('address of out_t:', id(out_t), '\n', out_t)

Result: The value and the allocated memory address are the same, the same data, but the name is different. Default data type, float32

torch.float32
address of t: 2377593164568 
 tensor([0., 0., 0., 0.])
address of out_t: 2377593164568 
 tensor([0., 0., 0., 0.])

torch.zeros_like()

torch.zeros_like(input, dtype=None, layout=None, device=None, requires_grad=False)

input: Creating the input tensor shape the same amount of all-0.

Code:

a = torch.tensor([1, 2, 3, 4])
t = torch.zeros_like(a)
print('a:', a)
print('t:', t)

result:

a: tensor([1, 2, 3, 4])
t: tensor([0, 0, 0, 0])

★torch.ones()

torch.ones(*size, out=None, dtype=None, layout=torch.strided, device=None, requires_grad=False)

torch.ones_like()

torch.ones_like(input, dtype=None, layout=None, device=None, requires_grad=False)

★torch.empty()

torch.empty(*size, out=None, dtype=None, layout=torch.strided, device=None, requires_grad=False, pin_memory=False)

Returns a tensor filled with uninitialized data.

Code:

a = torch.empty(4, 2)
print(a)

result:

tensor([[1.3563e-19, 4.7393e+30],
        [1.4312e+13, 1.7753e+28],
        [1.9208e+31, 4.6114e+24],
        [3.1036e+27, 7.1941e+28]])

torch.empty_like()

torch.empty_like（input，dtype = None，layout = None，device = None，require_grad = False ）

torch.full()

torch.full(size, fill_value, out=None, dtype=None, layout=torch.strided, device=None, requires_grad=False)

full_value: The value filled into the tensor, must be a number .

Code:

a = torch.full((3, 3), 3.14)
print(a)

result:

tensor([[3.1400, 3.1400, 3.1400],
        [3.1400, 3.1400, 3.1400],
        [3.1400, 3.1400, 3.1400]])

torch.full_like()

torch.full_like(input, fill_value, out=None, dtype=None, layout=torch.strided, device=None, requires_grad=False)

★ torch.arange()

torch.arange(start=0, end, step=1, out=None, dtype=None, layout=torch.strided, device=None, requires_grad=False)

Returns a one-dimensional equivalence tensor of the following formula

start ( Number ): Start value. Default: .0

end ( Number ): End value

step ( Number ): The gap between each pair of adjacent points, that is, the tolerance. Defaults:1

Code:

a = torch.arange(2, 10)
print(a)
b = torch.arange(3, 10, 6)
print(b)

result:

tensor([2, 3, 4, 5, 6, 7, 8, 9])
tensor([3, 9])

★torch.linspace()

torch.linspace（start，end，steps = 100，out = None，dtype = None，layout = torch.strided，device = None，require_grad = False ）

Create a one-dimensional tensor containing steps of equidistant point values between start and end.

start ( python: float ): start value

end ( python: float ): end value

The number of points to sample between steps ( python: int ) and . Defaults:startend100

Code:

a = torch.linspace(1, 10, 8)
print(a)

Result: The spacing is (end-start) / (steps-1)

tensor([ 1.0000,  2.2857,  3.5714,  4.8571,  6.1429,  7.4286,  8.7143, 10.0000])

torch.logspace()

Create a contained within the , steps one-dimensional tensor of the value of the number of equidistant points between

torch.logspace（start，end，steps = 100，base = 10.0，out = None，dtype = None，layout = torch.strided，device = None，require_grad = False ）

base ( python: float ): – the base of the logarithmic function. Defaults:10.0

Code:

a = torch.logspace(1, 10, 8)
print(a)

result:

tensor([1.0000e+01, 1.9307e+02, 3.7276e+03, 7.1969e+04, 1.3895e+06, 2.6827e+07,
        5.1795e+08, 1.0000e+10])

torch.eye()

torch.eye（n，m = None，out = None，dtype = None，layout = torch.strided，device = None，require_grad = False ）

Create a 2D diagonal tensor. If it is a square matrix, only need to set the number of rows

n: number of lines

m: number of columns

Code:

a = torch.eye(3, 4)
print(a)

result:

tensor([[1., 0., 0., 0.],
        [0., 1., 0., 0.],
        [0., 0., 1., 0.]])

torch.seed()

Used to set the random number seed to an uncertain random number

★torch.normal

torch.normal(mean, std, *, generator=None, out=None)

A tensor is formed by sampling from a normal distribution function with mean mean and standard deviation std. There are four modes:

mean is a scalar and std is a scalar: a specific number of values are sampled in the normal distribution determined by the given mean and std to form a tensor. The size of the formed tensor needs to be specified, such as forming a (2,1,4) the amount.

Code:

a = torch.normal(2, 3, size=(2, 2, 4))
print(a)

result:

tensor([[[ 3.4884,  1.9780, -1.2065,  2.6255],
         [ 0.6016,  5.3134, -0.4075, -0.4173]],

        [[-2.2090, -1.3162, -0.8148, -1.2134],
         [-1.3311,  1.8877,  6.2911,  1.7421]]])

mean is a tensor, std is a scalar: size cannot be set, and each element of the tensor mean is the mean, and the scalar std is the variance, and a value is taken from the normal distribution to form a tensor. The tensor size is the same as the tensor mean.

Code:

mean = torch.tensor([[1., 2., 3.],
                     [4., 5., 6.]])
a = torch.normal(mean, 1.)
print(a)

result:

tensor([[3.5307, 1.9584, 4.2141],
        [3.5544, 4.3479, 5.6270]])

mean is a scalar and std is a tensor: similar to the previous one.

mean is a tensor, and std is a tensor: the two tensors must have the same size, and the elements have a one-to-one correspondence.

Code:

mean = torch.tensor([[1., 2., 3.],
                     [4., 5., 6.]])
std = torch.tensor([[1., 0., 1.],
                     [0., 1., 0.]])
a = torch.normal(mean, std)
print(a)

result:

tensor([[0.6861, 2.0000, 5.1926],
        [4.0000, 5.6284, 6.0000]])

torch.rand()

torch.rand（* size，out = None，dtype = None，layout = torch.strided，device = None，require_grad = False ）

The generated tensor is filled with uniformly distributed random numbers on the interval [0, 1) , and the shape of the tensor is defined by variable parameters . There is also a correspondingsizetorch.rand_like()

torch.randint ()

torch.randint（low = 0，high，size，*，generator = None，out = None，dtype = None，layout = torch.strided，device = None，require_grad = False ）

The generated tensor is filled with uniformly distributed random numbers on the interval [low, high]and the shape of the tensor is defined by variable parameters . There is also a correspondingsizetorch.randint_like()

torch.randn()

torch.randn（* size，out = None，dtype = None，layout = torch.strided，device = None，require_grad = False ）

The generated tensor is filled with random numbers on a standard normal distribution, and the shape of the tensor is defined by variable parameters . There is also a correspondingsizetorch.randn_like()

For more methods, click here .