[Deep Learning_2.1.3] Gradient Test of Neural Networks

Gradient check is used to check that backprop is working properly

One-dimensional array model gradient test

Forward propagation, calculate the loss function:

J = theta * x

Backpropagation:

dtheta = x

Gradient test:

First compute gradapprox:

    thetaplus = theta + epsilon                               
    thetaminus = theta - epsilon                              
    J_plus = forward_propagation(x, thetaplus)                                  
    J_minus = forward_propagation(x, thetaminus)                                 
    gradapprox = (J_plus - J_minus)/(2*epsilon)

Then calculate grad :

   grad = backward_propagation(x, theta)     

Calculate the gradient difference:

    numerator = np.linalg.norm(grad - gradapprox)                               
    denominator = np.linalg.norm(grad) + np.linalg.norm(gradapprox)                    
    difference =  numerator/denominator

n-dimensional array model gradient test

calculation steps

Code:

thetaplus = np.copy(parameters_values)                                       
        thetaplus[i][0] = thetaplus[i][0] + epsilon                                
        J_plus[i], _ = forward_propagation_n(X, Y, vector_to_dictionary(thetaplus)) 

thetaminus = np.copy(parameters_values)                                     
        thetaminus[i][0] = thetaminus[i][0] - epsilon                                       
        J_minus[i], _ = forward_propagation_n(X, Y, vector_to_dictionary(thetaminus))

gradapprox[i] = (J_plus[i] - J_minus[i])/(2*epsilon)