[Pytorch] AutoGrad personal understanding

Prerequisite knowledge: [Pytorch] Forward propagation and back propagation examples_Friend Xiao A's Blog-CSDN Blog

Table of contents

Introduction

leaf node

Tensor AutoGrad Functions

---

Introduction

torch.autograd is PyTorch's automatic differentiation engine (autogradation) that powers neural network training. torch.autograd requires minimal changes to existing code - the attribute requires_grad=True for declaring Tensors that need to compute gradients. As of now, PyTorch only supports autograd for FloatTensor types (half, float, double, and bfloat16) and ComplexTensor (cfloat, cdouble). 【Information from official website】

leaf node

Leaf nodes are concepts in discrete mathematics . A node that has no child nodes (that is, the degree is 0) in a tree is called a leaf node, or "leaf" for short. A leaf is a node with a degree of 0, also known as a terminal node.

In pytorch, what is a leaf node? According to the official definition, it is understood as follows.

• All tensors for which requires_grad is False are conventionally reduced to leaf tensors
• Tensors where requires_grad is True, if they are created by the user, they are leaf tensors (leaf Tensor), indicating that they are not the result of the operation, so grad_fn=None

Example 1

def test_training_pipeline2():
    input_data = [[4, 4, 4, 4],
                  [9, 9, 9, 9]]  # 2x4
    input = torch.tensor(input_data, dtype=torch.float32)  # requires_grad=False
    output = torch.sqrt(input)
    
    target_data = [1, 2, 3, 4]
    target = torch.tensor(target_data, dtype=torch.float32)  # requires_grad=False
    loss_fn = torch.nn.MSELoss()
    loss = loss_fn(input=output, target=target)
    print("\ninput.is_leaf:", input.is_leaf)
    print("output.requires_grad:", output.requires_grad)
    print("output.is_leaf:", output.is_leaf)
    print("target.is_leaf:", target.is_leaf)
    print("loss.requires_grad:", loss.requires_grad)
    print("loss.is_leaf:", loss.is_leaf)

sample 2

def test_training_pipeline2():
    input_data = [[4, 4, 4, 4],
                  [9, 9, 9, 9]]  # 2x4
    input = torch.tensor(input_data, dtype=torch.float32)  # requires_grad=False
    output = torch.sqrt(input)
    output.requires_grad_(True) # requires_grad=True
    
    target_data = [1, 2, 3, 4]
    target = torch.tensor(target_data, dtype=torch.float32)  # requires_grad=False
    loss_fn = torch.nn.MSELoss()
    loss = loss_fn(input=output, target=target)
    
    print("\ninput.is_leaf:", input.is_leaf)
    print("output.requires_grad:", output.requires_grad)
    print("output.is_leaf:", output.is_leaf)
    print("target.is_leaf:", target.is_leaf)
    print("loss.requires_grad:", loss.requires_grad)
    print("loss.is_leaf:", loss.is_leaf)

Example 3

 

def test_training_pipeline5():
    input = torch.rand(1, requires_grad=True)
    output = torch.unique(
        input=input, 
        sorted=True, 
        return_inverse=False, 
        return_counts=False, 
        dim=None
    )
    
    print("\ninput.is_leaf:", input.is_leaf)
    print("output.requires_grad:", output.requires_grad)
    print("output.is_leaf:", output.is_leaf)
    
    output.backward()

Example 4

def test_training_pipeline3():
    input_data = [[4, 4, 4, 4],
                  [9, 9, 9, 9]]  # 2x4
    input_a = torch.tensor(input_data, dtype=torch.float32, requires_grad=True)
    input_b = torch.tensor(input_data, dtype=torch.float32, requires_grad=True)
    output = torch.ne(input_a, input_b)

    print("\ninput_a.is_leaf:", input_a.is_leaf)
    print("input_b.is_leaf:", input_b.is_leaf)
    print("output.dtype:", output.dtype)
    print("output.requires_grad:", output.requires_grad)
    print("output.is_leaf:", output.is_leaf)

    output.backward()   # 报错

 

 

Example 5

def test_training_pipeline7():
    input_data = [[4, 4, 4, 4],
                  [9, 9, 9, 9]]  # 2x4
    input_a = torch.tensor(input_data, dtype=torch.float32, requires_grad=True)
    input_b = torch.tensor(input_data, dtype=torch.float32)    
    output = torch.add(input_a, input_b)
    print("\ninput_a.requires_grad:", input_a.requires_grad)
    print("input_b.requires_grad:", input_b.requires_grad)
    print("output.requires_grad:", output.requires_grad)
    print("output.is_leaf:", output.is_leaf)
    
    grad = torch.ones_like(output)
    
    input_b[0][0] = 10 
    input_a[0][0] = 10 
    output.backward(grad)

 Example 6

def test_training_pipeline9():
    x = torch.tensor([1.0], requires_grad=True)
    y = x + 2
    z = 2 * y		# <-- dz/dy=2
    y[0] = -2.0
    
    print("\nx.is_leaf:", x.is_leaf)
    print("y.is_leaf:", y.is_leaf)
    print("z.is_leaf:", z.is_leaf)
    print("\nx.requires_grad:", x.requires_grad)
    print("y.requires_grad:", y.requires_grad)
    print("z.requires_grad:", z.requires_grad)
    z.backward()


def test_training_pipeline9():
    x = torch.tensor([1.0], requires_grad=True)
    y = x + 2
   z = y * y  # <-- dz/dy= 2*y
    y[0] = -2.0
    
    print("\nx.is_leaf:", x.is_leaf)
    print("y.is_leaf:", y.is_leaf)
    print("z.is_leaf:", z.is_leaf)
    print("\nx.requires_grad:", x.requires_grad)
    print("y.requires_grad:", y.requires_grad)
    print("z.requires_grad:", z.requires_grad)
    z.backward()

 

Tensor AutoGrad Functions

1. Tensor.grad

2. Tensor.requires_grad

3. Tensor.is_leaf

4. Tensor.backward(gradient=None, reqain_graph=None, create_graph=False)

5. Tensor.detach()

6. Tensor.detach_()

7. Tensor.retain_grad()