Pytorch：Tensor和Autograd

From the interface point of view, the operation of the tensor can be divided into two categories:

torch.functionLike torch.saveand so on.
The other is tensor.functionsuch tensor.viewlike.

For ease of use, while most of the operations of tensor support these two types of interfaces, which is not specific distinction, such as torch.sum (torch.sum(a, b))the tensor.sum (a.sum(b))functional equivalent.

From the storage point of view, the operation of the tensor can be divided into two categories:

Does not modify its own data, such as a.add(b), it returns a result of the addition of new tensor.
Modify its own data, as a.add_(b)the result of the addition is stored in a still, a is is modified.

Function names _ending is inplace way that would modify the caller's own data, the need to distinguish in practice.

tensor.numel () # Tensor and the total number of elements, equivalent to tensor.nelement () 
tesor.size () == tensor.shape
 # when Note that, t.Tensor (* sizes) Tensor created, the system It does not allocate space immediately, but will calculate whether the remaining memory enough to use, will be allocated to the use of tensor, while other operations are carried out in the space allocated immediately after creating tensor. Other commonly used methods of creating tensor for example as follows.

Common Tensor operation

By tensor.viewbe adjusted shape tensor methods, but should be the same before and after the adjustment of the total number of elements. viewDoes not modify its own data, the new tensor returned to the source tensor shared memory, that is to change one of them, the other will also change. You may often need to add or reduce a dimension in practical applications, at this time squeeze, and unsqueezetwo functions come in handy.

resizeAnother is used to adjust the sizeprocess, but viewdifferent, it can change the size of the tensor. If the new size than the original size, automatically allocate new memory space, and if the new size is smaller than the original size, the data will still be saved before, look at an example.

Tensor support and numpy.ndarray similar indexing operations, and similar syntactically, by following a few examples to explain commonly used index operations. Unless otherwise specified, the index out of the results with the original tensor shared memory, that is a modification, other modifications will follow.

PyTorch improve the index operation in 0.2 version, now supports advanced indexing the vast majority of numpy 1 . Advanced indexing can be seen as an ordinary index operations expand, but the results are generally not advanced indexing operation and the original Tensor shared memory.

Tensor have different data types, as shown in Table 3-3, and each type corresponding to a CPU and GPU versions (except HalfTensor). The default tensor is FloatTensor, can t.set_default_tensor_type modify the default type tensor (if the default type GPU tensor, all operations are performed on the GPU). Tensor analysis of the type of memory footprint helpful. For example, a size of (1000, 1000, 1000) FloatTensor, which has 1000*1000*1000=10^9elements, each element representing 32bit / 8 = 4Byte memory, they accounted for about 4GB memory / memory. HalfTensor is designed specifically for GPU version, only half FloatTensor the same number of elements, memory usage, so it can greatly ease the GPU memory shortage problem, but because HalfTensor values can be represented by the limited size and precision 1 , it may appear overflow and other issues.

Element by element operation

Each element of this part of the operation would tensor (point-wise, also known element-wise) is operated, the operation of such input and output consistent shape. Common operations as shown in Table 3-4

Merge operations

Such operations cause the shape of the output is less than the input shape, and the operation can be specified along a certain dimension. As an addition sum, the whole may be calculated and tensor, tensor may be calculated for each row or each column and. Merging common operations as shown in Table 3-5.

Most of the above functions have a parameter dimto specify these operations are performed in which dimension. About dim (Numpy corresponding to the axis) of the divergent explanation, here is a simple memory mode:

Suppose that the shape of the input (m, n, k)

If you specify dim = 0, is the shape of the output (1, n, k) or (n, k)
If the specified dim = 1, is the shape of the output (m, 1, k) or (m, k)
If you specify dim = 2, it is the shape of the output (m, n, 1) or (m, n)

Is there a "1", depending on the size of the parameters keepdim, keepdim=Truewill remain dimension 1. Note that the above only lessons learned, not all functions are in line with changes in the way this shapes, such as cumsum.

Compare

There are some comparison function is element-wise comparison, the operation is similar to the operating element by element, and some are similar to merge operations. Comparison of common functions as shown in Table 3-6.

Comparison operation in the first row of the table has been achieved operator overloading, can be used a>=b, , a>b, a!=b, a==bwhich returns a result ByteTensor, can be used to select elements. max / min these two special operation to max, it has the following three usage:

t.max (tensor): Returns the largest of a number of tensor
t.max (tensor, dim): the maximum number of specified dimensions, and returns the index tensor
t.max (tensor1, tensor2): Comparison of two large compared tensor elements

As for comparing a tensor and a number, you can use the clamp function.

Linear Algebra

PyTorch linear function of the main package and Blas Lapack, interfaces and its usage is similar. Commonly used linear algebra functions shown in Table 3-7.

Specific instructions for use, see the official documentation 1 , note that the transposed matrix will result in storage space is not continuous, the need to call its .contiguousmethods to turn continuous.

Note: No matter what type of input is, t.tensor data will be copied, not shared memory

tensor internal structure

tensor分为头信息区(Tensor)和存储区(Storage)，信息区主要保存着tensor的形状（size）、步长（stride）、数据类型（type）等信息，而真正的数据则保存成连续数组。由于数据动辄成千上万，因此信息区元素占用内存较少，主要内存占用则取决于tensor中元素的数目，也即存储区的大小。一般来说一个tensor有着与之相对应的storage, storage是在data之上封装的接口，便于使用，而不同tensor的头信息一般不同，但却可能使用相同的数据。下面看两个例子。

可见绝大多数操作并不修改tensor的数据，而只是修改了tensor的头信息。这种做法更节省内存，同时提升了处理速度。在使用中需要注意。此外有些操作会导致tensor不连续，这时需调用tensor.contiguous方法将它们变成连续的数据，该方法会使数据复制一份，不再与原来的数据共享storage。另外读者可以思考一下，之前说过的高级索引一般不共享stroage，而普通索引共享storage，这是为什么？（提示：普通索引可以通过只修改tensor的offset，stride和size，而不修改storage来实现）。

尽量使用tensor.to(device), 将device设为一个可配置的参数，这样可以很轻松的使程序同时兼容GPU和CPU
数据在GPU之中传输的速度要远快于内存(CPU)到显存(GPU), 所以尽量避免频繁的在内存和显存中传输数据。

在平时写代码时，就应养成向量化的思维习惯，千万避免对较大的tensor进行逐元素遍历。

此外还有以下几点需要注意：

大多数t.function都有一个参数out，这时候产生的结果将保存在out指定tensor之中。
t.set_num_threads可以设置PyTorch进行CPU多线程并行计算时候所占用的线程数，这个可以用来限制PyTorch所占用的CPU数目。
t.set_printoptions可以用来设置打印tensor时的数值精度和格式。下面举例说明。