Extreme Learning Machine (ELM) Regression Prediction Based on Butterfly Algorithm-with Code

Regression Prediction of Extreme Learning Machine (ELM) Based on Butterfly Algorithm

Abstract: This paper uses the butterfly algorithm to optimize the extreme learning machine and use it for regression prediction

1. Overview of the principle of extreme learning machine

A typical single-hidden layer feedforward neural network structure is shown in Figure 1. It consists of an input layer, a hidden layer, and an output layer. The neurons in the input layer and the hidden layer, and between the hidden layer and the output layer are fully connected. Among them, the input layer has n neurons, corresponding to n input variables, and the hidden layer has l neurons; the output layer has m neurons, corresponding to m output variables. Without loss of generality, let the connection weight w between the input layer and the hidden layer be:
$$w=\left[\begin{array}{cccc}{w}_{11}& w_{12}& ...& w_{1,n}\\ w_{21}& w_{22}& ...& w_2\\ ...& & & \\ w_l& w_l& ...& w_{ln}\end{array}\right]\text{\hspace{1em}( 1 )}$$
Among them,$w_n$Indicates the input layer $i$ neurons and hidden layer$Connection\; weights\; between\; j$ neurons.

Let the connection weight between hidden layer and output layer be $\beta$ :
$$b=\left[\begin{array}{cccc}{b}_{11}& b_{12}& ...& b_1\\ b_{21}& b_{22}& ...& b_2\\ ...& & & \\ b_l& b_l& ...& b_{lm}\end{array}\right]\text{\hspace{1em}( 2 )}$$
Among them, since$b_{jk}$Indicates the connection weight between the jth neuron in the hidden layer and the kth neuron in the output layer.

Let the threshold value b of hidden layer neurons be:
$$b=\left[\begin{array}{c}{b}_1\\ b_2\\ ...\\ b_l\end{array}\right]\text{\hspace{1em}( 3 )}$$
Let the input matrix X and output matrix Y of the training set with Q samples be
$$X=\left[\begin{array}{cccc}{x}_{11}& x_{12}& ...& x_{1Q}\\ x_{21}& x_{22}& ...& x_{2Q}\\ ...& & & \\ x_n& x_n& ...& x_{nQ}\end{array}\right]\text{\hspace{1em}(4)}$$

KaTeX parse error: Undefined control sequence: \matrix at position 11: Y =\left[\̲m̲a̲t̲r̲i̲x̲{y_{11},y_{12},…

Assuming that the activation function of neurons in the hidden layer is g(x), it can be obtained from Figure 1 that the output T of the network is:
$$T=[t_1,..,t_Q{]}_{m\ast Q},t_j=[t_1,...,t_{mj}{]}^T={\left[\begin{array}{c}{\sum }_{i=1}^t{b}_{i}g(w_i{x}_j+b_i)\\ {\sum }_{i=1}^t{b}_{i2}g(w_i{x}_j+b_i)\\ ...\\ {\sum }_{i=1}^t{b}_{im}g(w_i{x}_j+b_i)\end{array}\right]}_{m\ast 1},(j=1,2,...,Q)\text{\hspace{1em}( 6 )}$$
Formula (6) can be expressed as:
$$H\beta =T’\text{\hspace{1em}( 7 )}$$
Among them, T' is the transpose of the matrix T; H is called the hidden layer output matrix of the neural network, and the specific form is as follows:
$$H(w_1,...,w_i,b_1,...,b_l,x_1,...,x_Q)={\left[\begin{array}{cccc}g(w_1\ast x_1+b_1)& g(w_2\ast x_1+b_2)& ...& g(w_l\ast x_1+b_l)\\ g(w_1\ast x_2+b_1)& g(w_2\ast x_2+b_2)& ...& g(w_l\ast x_2+b_l)\\ ...& & & \\ g(w_1\ast x_Q+b_1)& g(w_2\ast x_Q+b_2)& ...& g(w_l\ast x_Q+b_l)\end{array}\right]}_{Q\ast l}$$

2. ELM learning algorithm

From the previous analysis, we can see that ELM can randomly generate w and b before training, and only need to determine the number of neurons in the hidden layer and the activation function (infinitely differentiable) of the hidden layer and neurons to calculate β \ $beta$ . Specifically, the learning algorithm of ELM mainly has the following steps:

(1) Determine the number of neurons in the hidden layer, and randomly set the connection weight w between the input layer and the hidden layer and the bias b of the hidden layer neurons;

(2) Select an infinitely differentiable function as the activation function of hidden layer neurons, and then calculate the hidden layer output matrix H;

(3) Calculate the weight of the output layer: $b=H^{+}{T}^{\prime }$

It is worth mentioning that related research results show that not only many nonlinear activation functions can be used in ELM (such as Sigmoid function, sinusoidal function and composite function, etc.), but also non-differentiable functions, and even discontinuous function as the activation function.

3. Regression problem data processing

The training set and test set are generated by random method, in which the training set contains 1900 samples, and the test set contains 100 samples. In order to reduce the influence of large variance of variables on the performance of the model, normalize the data before building the model.

4. ELM optimized based on butterfly algorithm

The specific principle of the butterfly algorithm refers to the blog: https://blog.csdn.net/u011835903/article/details/107855860

As can be seen from the foregoing, the initial weights and thresholds of ELM are randomly generated. The initial weights and thresholds generated each time are full-scale. In this paper, the butterfly algorithm is used to optimize the initial weight and threshold. The fitness function is designed as the MSE of the error of the training set:
$$fitness=argmin(MS{E}_{pridect})$$

The fitness function selects the MSE error after training. The smaller the MSE error, the higher the coincidence between the predicted data and the original data. The output of the final optimization is the best initial weight and threshold. Then the network trained with the optimal initial weight threshold is tested on the test data set.

5. Test results

The relevant parameters of the butterfly algorithm are as follows:

%训练数据相关尺寸
R = size(Pn_train,1);
S = size(Tn_train,1);
N = 20;%隐含层个数
%% 定义蝴蝶优化参数
pop=20; %种群数量
Max_iteration=50; %  设定最大迭代次数
dim = N*R + N*S;%维度，即权值与阈值的个数
lb = [-1.*ones(1,N*R),zeros(1,N*S)];%下边界
ub = [ones(1,N*R),ones(1,N*S)];%上边界

Compare the butterfly-optimized ELM with the base ELM.

The prediction results are as shown in the figure below

Base ELM MSE error: 0.00049385
BOA-ELM MSE error: 3.3277e-06

Butterfly-ELM is significantly better than base ELM from MSE

6. References

Book "MATLAB Neural Network 43 Case Analysis"

7. Matlab code