Zhou Zhihua "machine learning" exercises solve: Ch5.7 - RBF Network Experiment

Programming single RBF neural network, and the experimental data set on an exclusive OR.

The relevant source code is hosted on Github: PnYuan / Machine-Learning_ZhouZhihua , welcome to visit.

topic

Note: This title program based on the Python implementation ( here for the complete code and data sets ).

RBF Network Fundamentals Review

RBF network using RBF (Radial Basis Function Function) hidden neurons as an activation function, a local approximation neural network to analyze the activation function RBF below, and then analyzing the structure of RBF neural network.

Radial basis function (RBF)

Radial basis function is a function of class values dependent on the sample to the center point distance, based on a common theme of the present Radial Basis conducted experiments (gaussian RBF). The following is a Radial Basis book form p108 formula (5.19):

Here scale coefficient for beta], C_i center point (determined by the input dimension), depends on the value of a function of the sample x the distance to the center point (the 2-norm), the function of the parameters (β, c_i).

Illustrated as follows Radial Basis schematic diagram ( drawing program ):

RBF network

RBF neural network refers generally to a single-feed forward neural network hidden layer, as it uses radial basis function neuron hidden layer activation function, and the output is a linear combination of the output of the hidden layer, a schematic network architecture as follows:

Reference p108 formula (5.18), the output of the neural network is:

Further analysis, the general functions can be expressed as a linear combination of a set of basis functions, the RBF network with hidden neuron corresponding to construct such a set of basis functions, linear combinations of the output layer, thereby realizing the function approximation function.

RBF networks

View the complete code

Here RBF neural network modeling process points about two steps:

Neuronal determined Gaussian radial basis function corresponding to the center C;
Using the BP algorithm to train the remaining parameters w, β;

Its implementation is discussed in turn below:

RBF center parameter acquisition

C parameters of RBF center of acquiring some of the following methods:

从输入数据样本中抽取，这就要求输入样本具有较好的代表性；
自组织生成，如聚类法生成，采用聚类中心来作为中心参数 c ，同时根据各中心的距离来初始化尺度系数 β ；

RBF-BP算法推导

参考神经网络基础 - 编程实现标准BP算法，这里隐层激活函数为RBF，输出层神经元数为 1 ，激活函数为 y=f(x)=x 。

BP算法采用梯度下降法进行参数迭代更新，参考书p102-103，进行RBF-BP算法中基于梯度下降的参数更新推导如下：

在完成基础推导之后，给出RBF网络的BP算法如下所示：

RBF-BP算法实现

样例代码如下：

def (self, x, y):
    '''
    the implementation of special BP algorithm on one slide of sample for RBF network
    @param x, y: array and float, input and 大专栏  周志华《机器学习》习题解答：Ch5.7 - RBF网络实验 output of the data sample
    '''
    
    
    import numpy as np 

    # get current network output
    self.y = self.Pred(x)
    
    # calculate the gradient for hidden layer 
    g = np.zeros(self.h_n)
    for h in range(self.h_n):
        g[h] = (self.y - y) * self.b[h]    
    
    # updating the parameter
    for h in range(self.h_n):
        self.beta[h] += self.lr * g[h] * self.w[h] * np.linalg.norm(x-self.c[h],2)
        self.w[h] -= self.lr * g[h]

异或问题实验

查看完整代码

准备数据

首先基于numpy.array生成异或数据，该数据为2输入，1输出，如下所示：

样例代码：

1
2
3

# train set
X_trn = np.random.randint(0,2,(100,2))
y_trn = np.logical_xor(X_trn[:,0],X_trn[:,1])

样例数据：

>>> X_trn
array([[0, 0],
       [1, 1],
       ...
>>> y_trn
array([False, False, ...

参数之-RBF中心点

这里由于采用异或数据，其中心点可以简单设置如下：

centers = np.array([[0,0],[0,1],[1,0],[1,1]])

同时取隐节点数目为4。

生成模型并训练

样例代码如下：

# construct the network
rbf_nn = RBP_network()  # initial a BP network class
rbf_nn.CreateNN(4, centers, learningrate=0.05)  # build the network structure

# parameter training（这里迭代10次）
for i in range(10): 
    rbf_nn.TrainRBF(X_trn, y_trn)

绘制出训练过程中的均方误差变化曲线如下图：

You can see from the chart, the curves converge very quickly, indicating that RBF network training data set here is exclusive or very easy, this is also our complete and correct related data generated.

Test Model

Generating a set of test data generation method in accordance with the training set data, predicted by the model, the results are as follows:

test error rate: 0.000

That test error rate is 0, we can see our model predicts very accurate, good generalization performance (mainly due to the XOR prediction model is too simple for RBF networks).

This question Summary

Recalling the RBF network works, such as reference books p108 formula (5.18) - (5.19). RBF basis function network model similar to modeling nonlinear models, and further, we may be linked to generalized additive model. Radial basis function RBF network as a single hidden layer activation function (i.e., kernel function), which is in turn linked to the SVM with RBF kernel.

Recalling the RBF network implementation process, we or the method may be considered as a semi-supervised learning method, specifically:

step 1: unsupervised learning acquisition center parameters, from the data clustering method used;
step 2: supervised learning, training parameters based on the data, the process is generally based on BP algorithm;

Related reference

Here are some references:

Basics: absolute classic RBF neural network - Baidu library
Advanced knowledge: RBF neural network and BP neural network What is the difference - know almost
Visualization: matplotlib Drawing FIG. 3D - matplotlib official website