torch.autogradThe module provides users with an interface for custom derivation. torch.autograd.gradIt is possible to manually derive the gradient of the loss for some parameters, which is especially suitable for debugging and observation.
The author often uses this interface to observe whether the gradient value is abnormal when there is a problem in model optimization ; and to replace visual tools such as tensorboard to quickly understand complex calculation graphs .
For example, consider the following code:
t2 = torch.autograd.grad(adv_loss,next(self.dis.parameters()),retain_graph=True)
t3 = torch.autograd.grad(adv_loss,next(self.gen.parameters()),retain_graph=True)
t1 = torch.autograd.grad(adv_loss,target,retain_graph=True)
For observation, whether the final loss is calculated by self.dis (model), self.gen (model), target (tensor), and then used to determine which models participate in the optimization of this loss. This function is similar to using tensorboard to visualize the calculation process of loss, but it is more convenient.
If the following error occurs during autograd:
“...appears to not have been used in the graph”it means that this part of the model parameters and variables have not participated in the calculation of the loss. Naturally, when loss.backward(), these parameters will not be optimized.
When using this technique, pay attention to two parameters:
allow_unused=False: The default is False, which means that if the parameter to be derived is not in the calculation graph, an error will be reported; if it is set to True, the gradient will be calculated even if the parameter is not in the loss calculation graph, and it must be 0. Here, it must be set to False , that is, if the parameter is not in the calculation graph, the above-mentioned `"...appears to not have been used in the graph" error will be prompted.
retain_graph=True: The default is False, that is, once the derivative is calculated, the calculation graph will be released; if you want to observe the results of autograd multiple times, you must set it to True to keep the calculation graph.
For more details about torch.autograd.grad, see:
torch official documentation