[Youcans Deep Learning 06] PyTorch Introductory Tutorial: Basic Operations of Tensors

Welcome to pay attention to the "deep learning of youcans" series, which is being continuously updated...
[Deep learning of youcans 01] Miniconda of the installation environment
[Deep learning of youcans 02] PyTorch CPU version installation and environment configuration
[Deep learning of youcans 03] PyTorch CPU version installation And environment configuration
[youcans deep learning 04] PyTorch introductory tutorial: basic knowledge
[youcans deep learning 05] PyTorch introductory tutorial: quick start
[youcans deep learning 06] PyTorch introductory tutorial: basic operations of tensors

1. Tensor and its data structure

"Tensor" is the basic data structure of PyTorch.

In PyTorch, tensors are mainly used to store and transform data. The data structure and operation of tensor are similar to Numpy multidimensional array ndarray, but it can be used on GPU to accelerate calculation, which is more suitable for deep learning.

A tensor is a multidimensional array, which is a high-dimensional extension of scalars, vectors, and matrices. A scalar is a 0-dimensional tensor, a vector is a 1-dimensional tensor, and a matrix is ​​a 2-dimensional tensor.

You can ndimcheck the dimensionality of a tensor with , check the shape of a tensor with shapeor , check the data type with , and check the number of elements in a tensor with .size()dtypenumel()

# 导入PyTorch
import pytorch

# (1) 通过数据直接创建Tensor
x = torch.tensor([[1.0, 2.0, 3.6], [0.0, 4.0, 7.0]])
print(x)

# (2) 查看张量的维数，形状和类型
print("x.ndim:", x.ndim)  # 张量的维数
print("x.shape:", x.shape)  # 张量的形状
print("x.size:", x.size())  # 张量的形状
print("x.numel:", x.numel())  # 张量的元素个数
print("x.dtype:", x.dtype)  # 张量的数据类型

The output is:

tensor([[1.0000, 2.0000, 3.6000],
[0.0000, 4.0000, 7.0000]])
x.ndim: 2
x.shape: torch.Size([2, 3])
x.size: torch.Size([2, 3])
x.numel: 6
x.dtype: torch.float32

Description: torch.SizeIt is essentially a tuple (typle) and supports various operations on tuples.

Note that although a 0-dimensional tensor has only one element, it is not a number.

# (3) 0 维张量只有一个元素，但不是数值类型
x = torch.tensor([2009])
print("x = ", x)  # x 是张量
print("x.ndim:", x.ndim)  # 张量的维数
print("x.shape:", x.shape)  # 张量的形状
print("x.type:", x.type())  # x 的类型是张量

n = 2009
print("n = ", n)  # n 是标量，不是张量，没有维数、形状
print("type(n):", type(n))  # n 的类型是整型数值

Output result:

x = tensor([2009])
x.ndim: 1
x.shape: torch.Size([1])
x.type: torch.LongTensor
n = 2009
type(n): <class ‘int’>

A standard Python data structure is a single-level memory object that holds data and metadata.

The data structure of PyTorch is designed hierarchically, its framework supports interoperability, and the computationally intensive part of the kernel is migrated to the C/C++ backend through ATen and Caffe2.

PyTorch's main user-facing data structure is a THTensor object, which holds information about dimensions, offsets, steps, etc., and also stores a pointer to a THStorage object.

2. Tensor creation

2.1 torch.empty() creates an uninitialized tensor

Use ndimto check the dimensions of a tensor and use shapeto check the shape of a tensor.

# (4) 创建一个未初始化的张量
x = torch.empty(2, 3)  # x=torch.empty(size=(2, 3)
print(x)
print("x.ndim:", x.ndim)
print("x.shape:", x.shape)

The output is:

tensor([[6.8943e+34, 1.6212e-19, 1.4585e-19],
[7.7179e+28, 1.6217e-19, 1.4586e-19]])
x.ndim: 2
x.shape: torch.Size([2, 3])

2.2 torch.rand() creates a random number tensor

Machine learning models typically start with tensors of random numbers and learn from examples to adjust and update these random values.

• Use torch.rand()can create a random number tensor that obeys the uniform distribution, and the interval is [0,1)
• Use torch.randint()to create a random integer tensor that obeys the uniform distribution, and the interval is [low, high)
• Use torch.randn()to create a tensor of random numbers following a standard normal distribution with mean 0 and variance 1

torch.rand(*size, out=None, dtype=None, layout=torch.strided, device=None, requires_grad=False) → Tensor
torch.randint(low=0, high, size, generator=None, out=None, dtype=None, layout=torch.strided, device=None, requires_grad=False) → Tensor
torch.randn(*size, out=None, dtype=None, layout=torch.strided, device=None, requires_grad=False) → Tensor

Parameter Description:

• size: defines the shape of tensor, integer sequence list or tuple tuple
• dtype: optional, specifies the data type of the returned tensor
• layout: optional, an object representing the memory layout of torch.Tensor
• device: optional, the device stored by the tensor

# (5) 创建随机数张量
x1 = torch.rand(2, 3)  # 均匀分布随机数，[0,1) 区间
print(x1)
print("x1.shape:", x1.shape)
x2 = torch.randn(2, 3)  # 正态分布随机数，均值为 0 方差为1
print(x2)
print("x2.shape:", x2.shape)
x3 = torch.randint(low=0, high=10, size=(2,3))  # 均匀分布随机整数，[low,high) 区间
print(x3)
print("x3.shape:", x3.shape)

The output is:

tensor([[0.5007, 0.1634, 0.4525],
[0.1331, 0.5705, 0.0439]])
x1.shape: torch.Size([2, 3])
tensor([[ 0.5210, 1.8041, -0.5655],
[-2.0804, 0.3897, 0.4336]])
x2.shape: torch.Size([2, 3])
tensor([[5, 0, 6],
[1, 8, 2]])
x3.shape: torch.Size([2, 3])

2.3 Create special tensors such as all 0 matrix/all 1 matrix/identity matrix/diagonal matrix

Use torch.zeros()to create all 0 tensors, use torch.ones()to create all 1 tensors, and use torch.eye()to create identity matrix tensors.

torch.diag()Diagonal matrix tensors can be created using , but need to be created with one-dimensional tensors.

# (6) 创建全0矩阵/全1矩阵/单位矩阵/对角矩阵等特殊张量
xZeros = torch.zeros(3, 5)  # 全0矩阵
print(xZeros)
xOnes = torch.ones(3, 5)  # 全1矩阵
print(xOnes)
xEye = torch.eye(3)  # 单位矩阵
print(xEye)
x1 = torch.tensor([1.2, 2.5])  # 一维张量
xDiag = torch.diag(x1)  # 对角矩阵
print(xDiag)

The output is:

tensor([[0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0.],
[0., 0., 0., 0., 0.]])
tensor([[1., 1., 1., 1., 1.],
[1., 1., 1., 1., 1.],
[1., 1., 1., 1., 1.]])
tensor([[1., 0., 0.],
[0., 1., 0.],
[0., 0., 1.]])
tensor([[1.2000, 0.0000],
[0.0000, 2.5000]])

2.4 Create tensors directly from data

You can also directly create a tensor from your own data, and the data format can be a list, tuple or Numpy array.

# (7) 通过数据直接创建Tensor
# 通过列表创建
x1 = torch.tensor([[1, 2, 3],
               [4, 5, 6]])
print(x1)
# 通过元组创建
x2 = torch.tensor(((1, 2, 3), (4, 5, 6)))
print(x2)
# 通过Numpy创建
xnp = np.array([[1, 2, 3], [4, 5, 6]])
x3 = torch.tensor(xnp)
print(x3)

The output is:

tensor([[1, 2, 3],
[4, 5, 6]])
tensor([[1, 2, 3],
[4, 5, 6]])
tensor([[1, 2, 3],
[4, 5, 6]], dtype=torch.int32)

2.5 Create a new tensor from the shape of an existing tensor

It is also possible to create a new tensor from the shape of an existing tensor, and the created tensor has the same shape as the existing tensor.

To perform numerical filling according to the shape of the specified object, you only need _liketo , that is torch.*_like(). This method reuses some attributes of the input tensor by default, such as data type, but you can also customize the data type.

torch.zeros_like(input, *, dtype=None, layout=None, device=None) → Tensor
torch.ones_like(input, *, dtype=None, layout=None, device=None) → Tensor
torch.randn_like(input, *, dtype=None, layout=None, device=None) → Tensor

# (8) 通过已有张量的形状创建新的张量
x = torch.ones((2, 3), dtype=torch.float32)  # 均匀分布随机数
print(x)
print("x:", x.shape, x.type(), x.device)
# 返回的张量默认具有相同的 dtype
y1 = torch.rand_like(x)  # 默认具有相同的数据类型
y2 = torch.zeros_like(x, dtype=torch.int)  # 用户指定新的数据类型
print("y1:", y1.shape, y1.type(), y1.device)
print("y2:", y2.shape, y2.type(), y2.device)

The output is:

tensor([[1., 1., 1.],
[1., 1., 1.]])
x: torch.Size([2, 3]) torch.FloatTensor cpu
y1: torch.Size([2, 3]) torch.FloatTensor cpu
y2: torch.Size([2, 3]) torch.IntTensor cpu

2.6 Creating tensors from Numpy arrays

torch.from_numpy(ndarray)

torch.from_numpyFunction converts numpy array to tensor

# (9) 将 numpy 数组转换为张量
x_NP = np.random.rand(2, 3)
x = torch.from_numpy(x_NP)
print(x_NP)
print(x)

The output is:

[[0.02711595 0.35572182 0.67692876]
[0.73672641 0.01211524 0.22966701]]
tensor([[0.0271, 0.3557, 0.6769],
[0.7367, 0.0121, 0.2297]], dtype=torch.float64)

2.7 Creating sequential one-dimensional tensors

The function torch.arange() returns a one-dimensional tensor, the value is in the interval [start, end) (left closed and right open), and the step size is step.
The function torch.range() returns a one-dimensional tensor, the value is in the interval [start, end) (left closed and right closed), and the step size is step. This function was removed in a later version.
The function torch.linspace() returns a one-dimensional tensor, contained in the interval $[start,end]$ equidistant steps data points, the step size is calculated automatically.
The function torch.logspace() returns a one-dimensional tensor containing the base as the base and the index in the interval$[start,end]$ equidistant steps data points.

torch.arange(start=0, end, step=1, *, out=None, dtype=None)
torch.range(start=0, end, step=1, *, out=None, dtype=None) # 新版删除
torch.linspace(start, end, steps, *, out=None, dtype=None)
torch.logspace(start, end, steps, base=10.0, *, out=None, dtype=None)

Parameter Description:

• start: start value, the default value is 0
• end: end value
• step: step size, the default value is 1
• steps: number of data points

Note: The first element of the tensor is not necessarily start, and the last element is not necessarily end.

# (10) 创建顺序的一维张量
x1 = torch.arange(1, 3.6, 0.5)  # 6 个元素，(3.6-1.0)//0.5+1
print(x1)
print("x1:", x1.shape)
x2 = torch.linspace(3, 6, 10)  # 10 个元素，steps=10
print(x2)
print("x2:", x2.shape)
x3 = torch.logspace(3, 6, 10)  # # 10 个元素，steps=10
print(x3)
print("x3:", x3.shape)

The output is:

tensor([1.0000, 1.5000, 2.0000, 2.5000, 3.0000, 3.5000])
x1: torch.Size([6])
tensor([3.0000, 3.3333, 3.6667, 4.0000, 4.3333, 4.6667, 5.0000, 5.3333, 5.6667,
6.0000])
x2: torch.Size([10])
tensor([ 1000.0000, 2154.4346, 4641.5889, 10000.0000, 21544.3477,
46415.8867, 100000.0000, 215443.4688, 464158.8750, 1000000.0000])
x3: torch.Size([10])

For other methods of creating tensors, see [PyTorch Official Documentation] for details .

3. Data type of tensor

3.1 Properties of Tensors

Tensors are represented as torch.Tensor in PyTorch, and the main properties are as follows.

• data: Tensor data, which can be a value, a list of values, a Numpy array, etc.
• dtype: the data type of the tensor
• shape: the shape and dimension of the tensor
• device: The device type of tensor storage, supports CPU ('cpu') and GPU ('cuda')
• layout: layout mode of tensor in memory, supports torch.strided and torch.sparse_coo
• grad: gradient of data
• grad_fn: function to create tensors for automatic derivation
• requires_grad: whether to calculate the gradient, boolean value
• is_leaf: whether it is a leaf node, Boolean value

3.2 Tensor types defined by PyTorch

A tensor is a multidimensional matrix containing elements of a single data type.

The most common tensor type is torch.float32 or torch.float, known as "32-bit floating point". But there are also 16-bit floats (torch.float16 or torch.half) and 64-bit floats (torch.float64 or torch.double). In addition, there are 8-bit, 16-bit, 32-bit, and 64-bit integer types, Boolean types, and complex types.

Note that PyTorch does not have a string type, but can be represented by one-hot encoding or Word2vec models.

In particular, some tensor types in PyTorch are dedicated to CPUs, while others are more suitable for GPUs. For example, tensors with torch.cuda are all used for GPUs (Nvidia GPUs are called Compute Toolkit for CUDA).

Torch defines 9 different tensor types with CPU and GPU variants, the default data type is 32-bit float ( torch.float32 ).

Data type	dtype	CPU tensor	GPU tensor
32-bit float	torch.float32	torch.FloatTensor	torch.cuda.FloatTensor
64-bit float	torch.float64	torch.DoubleTensor	torch.cuda.DoubleTensor
16-bit float	torch.float16	torch.HalfTensor	torch.cuda.HalfTensor
8-bit integer (unsigned)	torch.uint8	torch.ByteTensor	torch.cuda.ByteTensor
8-bit integer (signed)	torch.int8	torch.CharTensor	torch.cuda.CharTensor
16-bit integer (signed)	torch.int16	torch.ShortTensor	torch.cuda.ShortTensor
32-bit integer (signed)	torch.int32	torch.IntTensor	torch.cuda.IntTensor
64-bit integer (signed)	torch.int64	torch.LongTensor	torch.cuda.LongTensor
Boolean	torch.bool	torch.BoolTensor	torch.cuda.BoolTensor

3.3 Check the data type of tensor

PyTorch provides several methods to check and judge data types:

• type(x): Returns the data type of the variable, applicable to types such as tensors, Numpy arrays, lists, and values.
• x.dtype: Returns the data type of the variable, applicable to tensors and Numpy arrays, not applicable to lists, values, etc.
• x.type(): Returns the data type of the variable, applicable to tensors, not applicable to Numpy arrays, lists, values, etc.
• isinstance(x, torch.FloatTensor): Compare data types, return True/False

# (11) 张量的数据类型
x_numpy = np.random.rand(2, 3)
x_list = x_numpy.tolist()
x_data = x_numpy[0,0].item()
x_tensor = torch.from_numpy(x_numpy)
print(type(x_numpy))  # <class 'numpy.ndarray'>
print(type(x_list))  # <class 'list'>
print(type(x_data))  # <class 'float'>
print(type(x_tensor))  # <class 'torch.Tensor'>
# 查看张量的数据类型
print(x_tensor.type())  # torch.DoubleTensor
print(x_tensor.dtype)  # torch.float64
# 判断张量的数据类型
print(isinstance(x_tensor, torch.FloatTensor))  # False

Notice:

(1) For tensor x, you can use x.type() or type(x) to view the data type, but the format of the return value is different. x.type() returns the concrete data type of the tensor, while type(x) returns the class name of the data type.
(2) For Numpy arrays, list lists and values, you can use type(x) to check the data type, but you cannot use the x.type() method.

3.3 Specify the data type of the tensor

When creating tensors, the default data type is 32-bit floating point ( torch.FloatTensor ).

Users can specify the data type of a tensor when creating a tensor, or change the data type of an existing tensor.

To change the data type of tensor x, you can use x.int() to change to torch.int32, or you can use x.type(torch.int32).

# (12) 指定张量的数据类型
# 在创建时指定张量的数据类型
x1 = torch.zeros(3, 3)
x2 = torch.zeros(3, 3, dtype=torch.float64)
x3 = torch.zeros(3, 3, dtype=torch.long)
print("x1.dtype: ", x1.dtype)
print("x2.dtype: ", x2.dtype)
print("x3.dtype: ", x3.dtype)
# 更改张量的数据类型
x1 = x1.long()  # x1=x1.type(torch.int64)
x2 = x2.int()  # x2=x2.type(torch.int32)
x3 = x3.type(torch.float64)
print("x1.long: ", x1.dtype)
print("x2.int: ", x2.dtype)
print("x3.float64: ", x3.dtype)

The output is:

x1.dtype: torch.float32
x2.dtype: torch.float64
x3.dtype: torch.int64

x1.long: torch.int64
x2.int: torch.int32
x3.float64: torch.float64

4. Tensor storage device

Tensors can be stored on CPU or GPU devices. For complex network models and large-scale data, GPUs are much faster than CPUs for certain types of operations, such as matrix multiplication.

Note that two tensors can only be operated on the same device (CPU or the same GPU). Data stored on the CPU cannot be operated on with data stored on the GPU, and data located on different GPUs cannot be directly operated on.

4.1 Check GPU device

In PyTorch, use torch.device('cpu')to represent a CPU device, and to torch.device('cuda')represent a GPU device.

The CPU device includes all physical CPUs and memory, i.e. PyTorch computations will try to use all CPU cores. The GPU device only represents a GPU graphics card and corresponding video memory. If there are multiple GPUs in the system, use to torch.device('cuda:{i}')represent the i-th GPU graphics card, and cuda:0 can also be abbreviated as cuda.

Use torch.cuda.is_available()to check if the system has a GPU device. If there is no GPU device, tensors can only be stored and operated on the CPU device. In general, user-facing APIs are device-independent, and developers do not need to pay attention to specific storage and computing devices when writing code, and the code will run on the CPU or GPU (if available) depending on the device.

# (13) 检查 GPU 设备
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print("device=", device)

print("CUDA版本: ", torch.version.cuda)
print("Pytorch版本: ", torch.__version__)
print("显卡是否可用: ", "GPU 可用" if(torch.cuda.is_available()) else " GPU 不可用")
print("显卡数量: ", torch.cuda.device_count())
print("是否支持BF16格式: ", "支持 BF16" if (torch.cuda.is_bf16_supported()) else "不支持 BF16")
print("当前显卡的型号: ", torch.cuda.get_device_name())
print("当前显卡的算力: ", torch.cuda.get_device_capability())
print("当前显卡的显存: ", torch.cuda.get_device_properties(0).total_memory/1024/1024/1024,'GB')
print("是否支持TensorCore: ", "支持" if (torch.cuda.get_device_properties(0).major >= 7) else "不支持")
print("当前显卡的使用率: ", torch.cuda.memory_allocated(0)/torch.cuda.get_device_properties(0).total_memory*100, "%")

Output result:

device= cuda
CUDA version: 11.7
Pytorch version: 1.13.1
Whether graphics card is available: GPU is available
Number of graphics cards: 1
Whether to support BF16 format: Support BF16
Current graphics card model: NVIDIA GeForce RTX 3060
Current graphics card computing power: (8, 6)
The video memory of the current graphics card: 11.99951171875 GB
Whether to support TensorCore: Yes
The utilization rate of the current graphics card: 0.0 %

4.2 Specify storage device for tensor

When creating tensors, the default tensor storage device.

Tensors and models can be placed on a specific device through the to(device) method.

Users can specify the tensor's storage device when creating a tensor, or change the storage device of an existing tensor. To change the storage device of tensor x, either use x.cuda(0) to change to GPU, or use the x.to(“cuda:0”) method.

Since NumPy cannot use the GPU, if you want to use NumPy to interact with tensors, you need to return the tensors to the CPU.

# (14) 指定张量的存储设备
# 在创建时指定张量的存储设备
x1 = torch.rand(3, 3, device="cpu")
x2 = torch.rand(3, 3, device="cuda")
x3 = torch.rand(3, 3, device="cuda:0")
# 获取张量的设备
print("x1: {}\nx2: {}\nx3: {}".format(x1.device, x2.device, x3.device))
# 修改张量的数据类型
x1 = x1.cuda(0)  # 将张量 x1 从 CPU 转移到 0#GPU
x2 = x1.to("cuda:0")  # # 将张量 x1 从 CPU 转移到 GPU
x3 = x3.cpu()  # # 将张量 x2 从 GPU 转移到 CPU
print("x1: {}\nx2: {}\nx3: {}".format(x1.device, x2.device, x3.device))

Output result:

x1: cpu
x2: cuda:0
x3: cuda:0

x1: cuda:0
x2: cuda:0
x3: cpu

4.3 GPU operation of the model

PyTorch models can also be stored on the CPU or GPU. By default, PyTorch creates data in memory and uses CPU for calculations.

Similarly, you can check devicethe properties to view the device where the model is stored, to()and transfer the model to the specified device through the method.

# (15) 模型的存储设备
model = torch.nn.Linear(3, 1)  # 线性模型
print("Linear model:\n", list(model.parameters()))
print("device of model: ", list(model.parameters())[0].device)

x = torch.rand(2, 3, device="cuda")  # 输入张量 x 在 GPU 设备
# print(model(x))  # 模型 model 与 输入张量 x 不在同一设备，程序报错

model.cuda()  # 将模型转移到 GPU
print("device of model: ", list(model.parameters())[0].device)
print(model(x))

Output result:

Linear model:
[Parameter containing:
tensor([[ 0.3656, -0.1920, 0.5609]], requires_grad=True), Parameter containing:
tensor([-0.0030], requires_grad=True)]
device of model: cpu
device of model: cuda:0
tensor([[0.4553],
[0.4906]], device=‘cuda:0’, grad_fn=)

Program description:

(1) The Pytorch model needs to be on the same device as the Tensor input to the model to perform calculations, otherwise an error will be reported:

RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cpu and cuda:0! (when checking argument for argument mat1 in method wrapper_addmm)

(2) Since the input parameters are stored in the GPU device, the calculation can only be performed after the model is transferred to the GPU.

Copyright Notice:
Welcome to the "Youcans' Deep Learning" series , please indicate the original text link when reposting:
[Youcans' Deep Learning 04] PyTorch Introductory Tutorial: Basic Tensor Operations (https://youcans.blog.csdn.net/article /details/130158748)
Copyright 2023 youcans, XUPT
Crated: 2023-04-12