Add Training and Testing Accuracy to a Simple Neural Network in PyTorch

Goldman Clarck :

I know this is a primitive question but what should I add in my code for it to output the training accuracy of the Neural Network in addition to the loss, I checked PyTorch tutorials and they show how to add training/testing accuracy in image classification but I do not know how to do that in my simple XOR solving NN, below is the code:

# Step 1: importing our dependencies
import torch
from torch.autograd import Variable
import numpy as np

# Our data
x = Variable(torch.Tensor([[0, 0, 1], [0, 1, 1], [1, 0, 1], [0, 1, 0], [1, 0, 0], 
                           [1, 1, 1], [0, 0, 0]]))
y = Variable(torch.Tensor([[0], [1], [1], [1], [1], [0], [0]]))

# Step 2: building our class model             
class NeuralNetwork(torch.nn.Module):
def __init__(self):
    super(NeuralNetwork, self).__init__()
    self.linear_ij = torch.nn.Linear(3, 4)
    self.linear_jk = torch.nn.Linear(4, 1)

def forward(self, x):
    matmul = self.linear_ij(x)
    activation = torch.sigmoid(matmul)
    matmul = self.linear_jk(activation)
    prediction = torch.sigmoid(matmul)
    return prediction

# Our model
model = NeuralNetwork()

# Constructing the loss function and the optimization algorithm
criterion = torch.nn.BCELoss(reduction='mean')
optimizer = torch.optim.SGD(model.parameters(), lr=1)

# Step 3: the training process
for epoch in range(10000):

    prediction = model(x)
    loss = criterion(prediction, y)

    if epoch % 1000 == 0 or epoch == 10000 - 1:
        print("epoch ", epoch, ",", "loss: ", loss.item())

    # Backpropagation process
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()

and this is it what it gives as an output:

epoch  0 , loss:  0.6983293294906616
epoch  1000 , loss:  0.015215665102005005
epoch  2000 , loss:  0.0048239342868328094
epoch  3000 , loss:  0.00280318153090775
epoch  4000 , loss:  0.001963752554729581
epoch  5000 , loss:  0.0015071843517944217
epoch  6000 , loss:  0.0012211233843117952
epoch  7000 , loss:  0.0010254186345264316
epoch  8000 , loss:  0.000883264874573797
epoch  9000 , loss:  0.0007753585232421756
epoch  9999 , loss:  0.0006908221403136849

As for testing:

# Testing our model
model.eval()

x_test = Variable(torch.Tensor([[1, 1, 0], [0, 0, 1], [0, 1, 1]]))
y_test = Variable(torch.Tensor([[0], [0], [1]]))
y_pred = model(x_test)
print(model(x_test))   

with the output:

tensor([[0.0026],
        [0.0011],
        [0.9991]], grad_fn=<SigmoidBackward>)

Szymon Maszke :

To add accuracy you only need one line, namely:

print("Accuracy: ", ((prediction > 0.5) == y).float().mean().item())

When you use sigmoid anything greater than 0.5 is considered positive and anything below negative. (prediction > 0.5) creates a tensor of bool type and you check which of those are equal to y. float() is needed as you cannot calculate mean of bool tensors. item() isn't needed but returns python value from single valued tensor and IMO looks cleaner this way.

You can do the same for test, hence it would be:

model.eval()

x_test = Variable(torch.Tensor([[1, 1, 0], [0, 0, 1], [0, 1, 1]]))
y_test = Variable(torch.Tensor([[0], [0], [1]]))
with torch.no_grad():
    y_pred = model(x_test)
    print("Accuracy: ", ((y_pred > 0.5) == y_test).float().mean().item())

Please notice torch.no_grad(). This context manager disables autograph when you are within it's scope. As you are just passing inputs through your neural network and not training it using gradients there is no need for autograph to be a part of the equation.

Working with logits

It's usually a good habit not to use final activation in your neural networks (unless you really need it). Hence your forward would look like this:

def forward(self, x):
    matmul = self.linear_ij(x)
    activation = torch.sigmoid(matmul)
    # Notice no sigmoid
    return self.linear_jk(activation)

This outputs logits (let's say unnormalized probability ranging from [-inf, inf]) indicating how confident your neural network is it's positive (+inf) or negative class.

You have to change your loss function accordingly, e.g. torch.nn.BCEWithLogitsLoss (mean is deafult reduction, no need to make it explicit here):

criterion = torch.nn.BCEWithLogitsLoss()

Finally, accuracy changes slightly as well. Now anything greater than 0 is considered positive, hence you would do this:

print("Accuracy: ", ((prediction > 0) == y).float().mean().item())

If you need probability you can use torch.sigmoid on the output still, but you might not even need it (as it seems in this case).

EDIT

You should also specify your data as torch.Tensor, torch.Variable is deprecated (tensors already have requires_grad=True), e.g.:

x = torch.Tensor(
    [[0, 0, 1], [0, 1, 1], [1, 0, 1], [0, 1, 0], [1, 0, 0], [1, 1, 1], [0, 0, 0]]
)
y = torch.Tensor([[0], [1], [1], [1], [1], [0], [0]])

EDIT2:

It should be placed below (or above) your loss printing, e.g.:

for epoch in range(10000):

    prediction = model(x)
    loss = criterion(prediction, y)

    if epoch % 1000 == 0 or epoch == 10000 - 1:
        # Here is fine
        print("Accuracy: ", ((prediction > 0.5) == y).float().mean().item())
        print("epoch ", epoch, ",", "loss: ", loss.item())

    # Backpropagation process
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()