%matplotlib inlineAutograd (automatic derivation): Automatic Differentiation (automatic differentiation)
Central to all neural networks in PyTorch is the autograd package.
Let’s first briefly visit this, and we will then go to training our
first neural network.
The autograd package provides automatic differentiation for all operations
on Tensors. It is a define-by-run framework, which means that your backprop is
defined by how your code is run, and that every single iteration can be
different.
Let us see this in more simple terms with some examples.
All core PyTorch neural network is autograd package. Let's take a quick look at this, then we have to train our first neural network.
autograd package provides for all operations on automatic differentiation tensor. It is run by a defined framework, which means that your support is defined by the operating mode code, and each iteration can be different.
Let us use simpler terms and take a look at some examples.
Tensor
torch.Tensor is the central class of the package. If you set its attribute
.requires_grad as True, it starts to track all operations on it. When
you finish your computation you can call .backward() and have all the
gradients computed automatically. The gradient for this tensor will be
accumulated into .grad attribute.
torch.TensorIt is the central class package. If all of its operations .requires_grad property set to True, it will begin tracking on it. When you complete the calculation, you can call. Backer () and automatically calculate all gradients. This gradient tensor properties .grad cumulatively.
To stop a tensor from tracking history, you can call .detach() to detach
it from the computation history, and to prevent future computation from being
tracked.
To prevent tensor tracking history, you can call .detach () be separated from the history of computing out and prevent tracking future of computing.
To prevent tracking history (and using memory), you can also wrap the code block
in with torch.no_grad():. This can be particularly helpful when evaluating a
model because the model may have trainable parameters with requires_grad=True,
but for which we don't need the gradients.
To prevent the tracking history (and the use of memory), may also be used torch.no_grad (): encapsulate the code block. This is particularly useful in assessing the model, because the model may have requires_grad = True parameter can be trained, but we do not need gradient.
There’s one more class which is very important for autograd
implementation - a Function.
There is also a very important class of autograd realization - Function.
Tensor and Function are interconnected and build up an acyclic
graph, that encodes a complete history of computation. Each tensor has
a .grad_fn attribute that references a Function that has created
the Tensor (except for Tensors created by the user - their
grad_fn is None).
Tensor and functions are interrelated, and the establishment of a directed acyclic graph, which encodes a complete history of computing. Each tensor has a .grad_fn property, which references a tensor function creates (except tensor user-created - they grad_fn is None).
If you want to compute the derivatives, you can call .backward() on
a Tensor. If Tensor is a scalar (i.e. it holds a one element
data), you don’t need to specify any arguments to backward(),
however if it has more elements, you need to specify a gradient
argument that is a tensor of matching shape.
If you want to calculate the derivative, you can call on a tensor. Reverse (). If tensor is a scalar (i.e., it contains a data element), you do not specify any backward () parameter, but if it has more elements, you need to specify a parameter gradient, which is a matching shape Zhang the amount.
import torchCreate a tensor and set requires_grad=True to track computation with it
x = torch.ones(2, 2, requires_grad=True)
print(x)tensor([[1., 1.],
[1., 1.]], requires_grad=True)Do an operation of tensor:
y = x + 2
print(y)tensor([[3., 3.],
[3., 3.]], grad_fn=<AddBackward0>)y was created as a result of an operation, so it has a grad_fn.
print(y.grad_fn)<AddBackward0 object at 0x7f34eebf07b8>Do more operations on y
z = y * y * 3
out = z.mean()
print(z, out)tensor([[27., 27.],
[27., 27.]], grad_fn=<MulBackward0>) tensor(27., grad_fn=<MeanBackward0>).requires_grad_( ... ) changes an existing Tensor's requires_grad
flag in-place. The input flag defaults to False if not given.
a = torch.randn(2, 2)
a = ((a * 3) / (a - 1))
print(a.requires_grad)
a.requires_grad_(True)
print(a.requires_grad)
b = (a * a).sum()
print(b.grad_fn)False
True
<SumBackward0 object at 0x7f34933dacc0>Gradients
Let's backprop now
Because out contains a single scalar, out.backward() is
equivalent to out.backward(torch.tensor(1)).
out.backward()print gradients d(out)/dx
print(x.grad)tensor([[4.5000, 4.5000],
[4.5000, 4.5000]])You should have got a matrix of 4.5. Let’s call the out
Tensor “\(o\)”.
We have that \(o = \frac{1}{4}\sum_i z_i\),
\(z_i = 3(x_i+2)^2\) and \(z_i\bigr\rvert_{x_i=1} = 27\).
Therefore,
\(\frac{\partial o}{\partial x_i} = \frac{3}{2}(x_i+2)\), hence
\(\frac{\partial o}{\partial x_i}\bigr\rvert_{x_i=1} = \frac{9}{2} = 4.5\).
You can do many crazy things with autograd!
x = torch.randn(3, requires_grad=True)
print(x)
y = x * 2
print(y)
while y.data.norm() < 1000:
y = y * 2
print(y)tensor([ 0.0147, -1.4388, 1.3875], requires_grad=True)
tensor([ 0.0295, -2.8775, 2.7750], grad_fn=<MulBackward0>)
tensor([ 7.5405, -736.6459, 710.3948], grad_fn=<MulBackward0>)gradients = torch.tensor([0.1, 1.0, 0.0001], dtype=torch.float)
y.backward(gradients)
print(x.grad)tensor([5.1200e+01, 5.1200e+02, 5.1200e-02])You can also stop autograd from tracking history on Tensors
with .requires_grad=True by wrapping the code block in
with torch.no_grad():
print(x.requires_grad)
print((x ** 2).requires_grad)
with torch.no_grad():
print((x ** 2).requires_grad)True
True
FalseRead Later:
Documentation of autograd and Function is at
http://pytorch.org/docs/autograd