使用单隐藏层神经网络对平面数据分类

引言

为了巩固下吴恩达深度学习——浅层神经网络中的理论知识，我们来实现一个使用单隐藏层神经网络对平面数据进行分类的例子。

关于本文代码中的公式推导可见吴恩达深度学习——浅层神经网络。

这是吴恩达深度学习第一节课的作业，如果你也在学习吴恩达深度学习教程，并且未做作业的话，建议你关闭该网页，根据作业指导自己实现代码先。

数据集

下面通过代码生成我们的数据集，文件名为planar_utils.py。

import matplotlib.pyplot as plt
import numpy as np
import sklearn
import sklearn.datasets
import sklearn.linear_model

# 绘制决策边界
def plot_decision_boundary(model, X, y):
    # Set min and max values and give it some padding
    x_min, x_max = X[0, :].min() - 1, X[0, :].max() + 1
    y_min, y_max = X[1, :].min() - 1, X[1, :].max() + 1
    h = 0.01
    # Generate a grid of points with distance h between them
    xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))
    # Predict the function value for the whole grid
    Z = model(np.c_[xx.ravel(), yy.ravel()])
    Z = Z.reshape(xx.shape)
    # Plot the contour and training examples
    plt.contourf(xx, yy, Z, cmap=plt.cm.Spectral)
    plt.ylabel('x2')
    plt.xlabel('x1')
    plt.scatter(X[0, :], X[1, :], c=np.squeeze(y), cmap=plt.cm.Spectral)
    

def sigmoid(x):
    """
    Compute the sigmoid of x

    Arguments:
    x -- A scalar or numpy array of any size.

    Return:
    s -- sigmoid(x)
    """
    s = 1/(1+np.exp(-x))
    return s

def load_planar_dataset():
    np.random.seed(1)
    m = 400 # number of examples
    N = int(m/2) # number of points per class
    D = 2 # dimensionality
    X = np.zeros((m,D)) # data matrix where each row is a single example
    Y = np.zeros((m,1), dtype='uint8') # labels vector (0 for red, 1 for blue)
    a = 4 # maximum ray of the flower

    for j in range(2):
        ix = range(N*j,N*(j+1))
        t = np.linspace(j*3.12,(j+1)*3.12,N) + np.random.randn(N)*0.2 # theta
        r = a*np.sin(4*t) + np.random.randn(N)*0.2 # radius
        X[ix] = np.c_[r*np.sin(t), r*np.cos(t)]
        Y[ix] = j
        
    X = X.T
    Y = Y.T

    return X, Y

可视化下我们的数据集：

# Package imports
import numpy as np
import matplotlib.pyplot as plt
import sklearn
import sklearn.datasets
import sklearn.linear_model
from planar_utils import plot_decision_boundary, sigmoid, load_planar_dataset

%matplotlib inline

np.random.seed(1) # set a seed so that the results are consistent

X, Y = load_planar_dataset()
# Visualize the data:
plt.scatter(X[0, :], X[1, :], c=np.squeeze(Y), s=40, cmap=plt.cm.Spectral)
shape_X = X.shape
shape_Y = Y.shape
m = X.shape[1]  # training set size

print ('The shape of X is: ' + str(shape_X))
print ('The shape of Y is: ' + str(shape_Y))
print ('I have m = %d training examples!' % (m))

The shape of X is: (2, 400)
The shape of Y is: (1, 400)
I have m = 400 training examples!

可以看到，我们的数据分布成一朵花瓣的形式， $X$有两个特征， $Y \in \{0,1\}$。这也是一个二分类问题。

我们要做的是实现一个单隐藏神经网络对这个平面上的数据点进行分类。

代码实现

逻辑回归

# Train the logistic regression classifier
clf = sklearn.linear_model.LogisticRegressionCV();
clf.fit(X.T, Y.T);

# Plot the decision boundary for logistic regression
plot_decision_boundary(lambda x: clf.predict(x), X, Y)
plt.title("Logistic Regression")

# Print accuracy
LR_predictions = clf.predict(X.T)
print ('Accuracy of logistic regression: %d ' % float((np.dot(Y,LR_predictions) + np.dot(1-Y,1-LR_predictions))/float(Y.size)*100) +
       '% ' + "(percentage of correctly labelled datapoints)")

先用sklearn中的逻辑回归来实现下，并画出决策边界。

Accuracy of logistic regression: 47 % (percentage of correctly labelled datapoints)

可以看到，逻辑回归尝试用一条直线来区分这两个类别，效果不好。

那如果改成多项式特征呢，会不会好一点。

from sklearn.preprocessing import PolynomialFeatures
from sklearn.preprocessing import StandardScaler
from sklearn.pipeline import Pipeline

def PolynomialLogisticRegression(degree):
    return Pipeline([
        ('poly', PolynomialFeatures(degree=degree)),
        ('std_scaler', StandardScaler()),
        ('log_reg', sklearn.linear_model.LogisticRegressionCV())
    ])



# Train the logistic regression classifier
poly_clf = PolynomialLogisticRegression(2).fit(X.T, Y.T);

# Plot the decision boundary for logistic regression
plot_decision_boundary(lambda x: poly_clf.predict(x), X, Y)
plt.title("Logistic Regression")

# Print accuracy
LR_predictions = poly_clf.predict(X.T)
print ('Accuracy of logistic regression: %d ' % float((np.dot(Y,LR_predictions) + np.dot(1-Y,1-LR_predictions))/float(Y.size)*100) +
       '% ' + "(percentage of correctly labelled datapoints)")

Accuracy of logistic regression: 48 % (percentage of correctly labelled datapoints)

这里用二项式的话准确率确实好一点，但是只好了一点点，如果看了生成数据的函数就知道，生成函数本身是sin函数，所以这里用多项式特征效果也不好。

下面是三项式的结果：

可以看到逻辑回归在这种类似花瓣分布的数据上表现不佳，我们接下来用单隐藏层神经网络试一下。

单隐藏层神经网路

上面是我们网络的结构，隐藏层有4个节点，隐藏层的激活函数使用tanh,由于是二分类问题，输出层的激活函数使用sigmoid。

用到的公式:

对于每个样本 $x^{(i)}$:
$z^{[1] (i)} = W^{[1]} x^{(i)} + b^{[1]}\tag{1}$
$a^{[1] (i)} = \tanh(z^{[1] (i)})\tag{2}$
$z^{[2] (i)} = W^{[2]} a^{[1] (i)} + b^{[2]}\tag{3}$
$\hat{y}^{(i)} = a^{[2] (i)} = \sigma(z^{ [2] (i)})\tag{4}$

$y^{(i)}_{prediction} = \begin{cases} 1 & \text{if } a^{[2](i)} > 0.5 \\ 0 & \text{otherwise } \end{cases} \tag{5}$

得到所有样本的预测值, 就可以用下面的公式计算成本值 $J$:

$J = - \frac{1}{m} \sum\limits_{i = 0}^{m} \large\left(\small y^{(i)}\log\left(a^{[2] (i)}\right) + (1-y^{(i)})\log\left(1- a^{[2] (i)}\right) \large \right) \small \tag{6}$

还是先实现一些基本函数：

# 得到各层大小
def layer_sizes(X, Y):
    """
    Arguments:
    X -- 输入数据X (输入特征数, 样本数)
    Y -- 标签 (输出大小, 样本数)
    
    Returns:
    n_x -- 输入层的大小
    n_h -- 隐藏层的大小
    n_y -- 输出层的大小
    """
    n_x = X.shape[0] 
    n_h = 4
    n_y = Y.shape[0]
    return (n_x, n_h, n_y)

# 初始化参数
def initialize_parameters(n_x, n_h, n_y):
    """
    Argument:
    n_x -- 输入层的大小
    n_h -- 隐藏层的大小
    n_y -- 输出层的大小
    
    Returns:
    params -- 字典:
                    W1 -- 形状(n_h, n_x)的权重矩阵
                    b1 -- 形状(n_h, 1)的偏置向量
                    W2 -- 形状(n_y, n_h)的权重矩阵
                    b2 -- 形状(n_y, 1)的偏置向量
    """
    
    np.random.seed(2) # we set up a seed so that your output matches ours although the initialization is random.
    
    W1 = np.random.randn(n_h,n_x) * 0.01
    b1 = np.zeros((n_h,1))
    W2 = np.random.randn(n_y,n_h) * 0.01
    b2 = np.zeros((n_y,1))
    
    parameters = {"W1": W1,
                  "b1": b1,
                  "W2": W2,
                  "b2": b2}
    
    return parameters

# 前向传播
def forward_propagation(X, parameters):
    """
    Argument:
    X --输入数据 (n_x, m)
    parameters -- initialize_parameters函数输出值，包含各参数的初始值
    
    Returns:
    A2 -- 第二层的激活值
    cache -- 包含 "Z1", "A1", "Z2" 和 "A2"的字典
    """
    W1 = parameters['W1'] #(n_h,n_x)
    b1 = parameters['b1'] #(n_h, 1)
    W2 = parameters['W2'] #(n_y, n_h)
    b2 = parameters['b2'] #(n_y, 1)
    
    # 实现前向传播来计算A2（就是概率）
    Z1 = np.dot(W1,X) + b1  #(n_h,1)
    A1 = np.tanh(Z1) # np提供了这个函数
    Z2 = np.dot(W2,A1) + b2 #(n_y,1)
    A2 = sigmoid(Z2)
        
    cache = {"Z1": Z1,
             "A1": A1,
             "Z2": Z2,
             "A2": A2}
    
    return A2, cache


# 计算损失值
def compute_cost(A2, Y):
    """
    通过公式 (6)来计算交叉熵损失值
    
    Arguments:
    A2 -- 第二层的激活值 (1, 样本数)
    Y -- 标签 (输出大小, 样本数)
   
    Returns:
    cost -- 损失值
    
    """
    
    m = Y.shape[1] # number of example


    logprobs = np.multiply(np.log(A2), Y) + np.multiply((1-Y),np.log(1-A2))
    cost = -np.sum(logprobs) / m
    
    cost = float(np.squeeze(cost))  # 确保损失值是个标量 
                                    # E.g., turns [[17]] into 17     
    return cost

接下来通过反向传播来实现梯度下降更新参数，用到的公式如下：

def backward_propagation(parameters, cache, X, Y):
    """
    反向传播
    
    Arguments:
    parameters -- 包含参数值的字典
    cache -- 包含 "Z1", "A1", "Z2" 和 "A2"的字典
    
    Returns:
    grads -- 包含dW1,db1,dW2,db2梯度的字典
    """
    m = X.shape[1]
    
    W1 = parameters['W1']
    W2 = parameters['W2']
    A1 = cache['A1']
    A2 = cache['A2']
    
    # 反向传播: 计算 dW1, db1, dW2, db2. 
    dZ2 = A2 - Y
    dW2 = 1.0 / m * np.dot(dZ2,A1.T)
    db2 = np.sum(dZ2,axis=1,keepdims=True) / m #keepdims=True 防止生成(n,)的数组
    dZ1 = np.dot(W2.T,dZ2) * (1-A1**2) # (1-A1**2)是tanh的导数
    dW1 = np.dot(dZ1,X.T) / m
    db1 = np.sum(dZ1,axis=1,keepdims=True) / m
    
    grads = {"dW1": dW1,
             "db1": db1,
             "dW2": dW2,
             "db2": db2}
    
    return grads

最后就是更新参数了：

def update_parameters(parameters, grads, learning_rate = 1.2):
    """
    使用梯度值更新参数
    
    Arguments:
    parameters -- 包含更新前的参数值 
    grads -- backward_propagation() 函数返回的梯度值
    
    Returns:
    parameters -- 更新后的参数值字典
    """
    W1 = parameters['W1']
    b1 = parameters['b1']
    W2 = parameters['W2']
    b2 = parameters['b2']
    
    dW1 = grads['dW1']
    db1 = grads['db1']
    dW2 = grads['dW2']
    db2 = grads['db2']
    
    # 更新每个参数
    W1 = W1 - learning_rate * dW1
    b1 = b1 - learning_rate * db1
    W2 = W2 - learning_rate * dW2
    b2 = b2 - learning_rate * db2
    
    parameters = {"W1": W1,
                  "b1": b1,
                  "W2": W2,
                  "b2": b2}
    
    return parameters

最后把所有函数整合到一个nn_model函数中：

def nn_model(X, Y, n_h, num_iterations = 10000, print_cost=False):
    """
    Arguments:
    X --输入数据 (2, 样本数)
    Y -- 标签 (1, 样本数)
    n_h -- 隐藏层单元数
    num_iterations -- 梯度下降迭代次数
    print_cost -- 是否打印损失值
    
    Returns:
    parameters -- 训练得到的参数值，可用于预测
    """
    
    np.random.seed(3)
    n_x = layer_sizes(X, Y)[0]
    n_y = layer_sizes(X, Y)[2]
    
    # 初始化参数
    parameters = initialize_parameters(n_x,n_h,n_y)
    
    # 梯度下降算法迭代
    for i in range(0, num_iterations):
         
        # 前向传播
        A2, cache = forward_propagation(X,parameters)
        
        # 计算损失
        cost = compute_cost(A2,Y,parameters)
 
        # 反向传播
        grads = backward_propagation(parameters,cache,X,Y)
         
        # 更新参数值    
        parameters = update_parameters(parameters,grads)
        
        
        # Print the cost every 1000 iterations
        if print_cost and i % 1000 == 0:
            print ("Cost after iteration %i: %f" %(i, cost))

    return parameters

在训练我们的模型之前还要先实现预测函数：

def predict(parameters, X):
    """
    预测每个X中样本的类别
    
    Arguments:
    parameters -- 
    X -- input data of size (n_x, m)
    
    Returns
    predictions -- 预测值向量 (红: 0 / 蓝: 1)
    """
    
    # 预测其实就是前向传播的过程
    A2, cache = forward_propagation(X,parameters)
    # 将概率值转换为1和0
    predictions = np.array(A2 > 0.5)
  
    return predictions

下面用我们上面实现的模型来分类：

parameters = nn_model(X, Y, n_h = 4, num_iterations = 10000, print_cost=True)

# Plot the decision boundary
plot_decision_boundary(lambda x: predict(parameters, x.T), X, Y)
plt.title("Decision Boundary for hidden layer size " + str(4))

Cost after iteration 0: 0.693048
Cost after iteration 1000: 0.288083
Cost after iteration 2000: 0.254385
Cost after iteration 3000: 0.233864
Cost after iteration 4000: 0.226792
Cost after iteration 5000: 0.222644
Cost after iteration 6000: 0.219731
Cost after iteration 7000: 0.217504
Cost after iteration 8000: 0.219437
Cost after iteration 9000: 0.218602

输出准确率:

# Print accuracy
predictions = predict(parameters, X)
print ('Accuracy: %d' % float((np.dot(Y,predictions.T) + np.dot(1-Y,1-predictions.T))/float(Y.size)*100) + '%')

Accuracy: 90%

可以看到准确率远高于逻辑回归，神经网络能学习高度线性不可分的决策边界。

上面的模型中隐藏层只有四个节点，我们可以尝试不同的节点数，看那个效果最好：

# This may take about 2 minutes to run

plt.figure(figsize=(16, 32))
hidden_layer_sizes = [1, 2, 3, 4, 5, 20, 50]
for i, n_h in enumerate(hidden_layer_sizes):
    plt.subplot(5, 2, i+1)
    plt.title('Hidden Layer of size %d' % n_h)
    parameters = nn_model(X, Y, n_h, num_iterations = 5000)
    plot_decision_boundary(lambda x: predict(parameters, x.T), X, Y)
    predictions = predict(parameters, X)
    accuracy = float((np.dot(Y,predictions.T) + np.dot(1-Y,1-predictions.T))/float(Y.size)*100)
    print ("Accuracy for {} hidden units: {} %".format(n_h, accuracy))

Accuracy for 1 hidden units: 67.5 %
Accuracy for 2 hidden units: 67.25 %
Accuracy for 3 hidden units: 90.75 %
Accuracy for 4 hidden units: 90.5 %
Accuracy for 5 hidden units: 91.25 %
Accuracy for 20 hidden units: 90.0 %
Accuracy for 50 hidden units: 90.75 %