Data mining learning | task5 model fusion

1. The concept model of integration: first generate a set of individual learning, and then use a strategy combining them, to strengthen the model results. Zhou Zhihua and Li Hang teacher's book have proved that as the number of individual learner increases, integrated error rates show a downward exponentially, eventually tends to zero. Therefore, the integration model is widely used.

2. The model of integration strategies:
1. Simple Weighted Integration:
Regression (classification probability): arithmetic average convergence (Arithmetic mean), geometric mean fusion (Geometric mean);
Category: Voting (Voting)
Integrated: Sort fusion (Rank averaging), log fusion
2.stacking / blending:
Construction of a multilayer model, and then fit using the prediction result predicted.
3.boosting / bagging (already used in xgboost, Adaboost, GBDT in):
many trees upgrade method

Here regression \ classification probability - Fusion:
Fusion between 1) simple weighted average model

#回归/分类概率-融合
# 1.简单加权平均，结果直接融合
## 生成一些简单的样本数据，test_prei 代表第i个模型的预测值
test_pre1 = [1.2, 3.2, 2.1, 6.2]
test_pre2 = [0.9, 3.1, 2.0, 5.9]
test_pre3 = [1.1, 2.9, 2.2, 6.0]

# y_test_true 代表第模型的真实值
y_test_true = [1, 3, 2, 6] 

import numpy as np
import pandas as pd

## 定义结果的加权平均函数
def Weighted_method(test_pre1,test_pre2,test_pre3,w=[1/3,1/3,1/3]):
    Weighted_result = w[0]*pd.Series(test_pre1)+w[1]*pd.Series(test_pre2)+w[2]*pd.Series(test_pre3)
    return Weighted_result

from sklearn.metrics import mean_absolute_error as MAE
# 各模型的预测结果计算MAE
print('Pred1 MAE:',MAE(y_test_true, test_pre1))
print('Pred2 MAE:',MAE(y_test_true, test_pre2))
print('Pred3 MAE:',MAE(y_test_true, test_pre3))

Pred1 THERE: 0.1750000000000001
Pred2 THERE: .07499999999999993
Pred3 THERE: .10000000000000009

## 根据加权计算MAE
w = [0.3,0.4,0.3] # 定义比重权值
Weighted_pre = Weighted_method(test_pre1,test_pre2,test_pre3,w)
print('Weighted_pre MAE:',MAE(y_test_true, Weighted_pre))

Weighted_pre MAE: 0.05750000000000027
can be found in the results of the previous results with respect to the weight is lifted, which we call the simple weighted average. In addition, there are some special forms, such as mean average, median average.
2) Stacking fusion (return):

from sklearn import linear_model

def Stacking_method(train_reg1,train_reg2,train_reg3,y_train_true,test_pre1,test_pre2,test_pre3,model_L2= linear_model.LinearRegression()):
    model_L2.fit(pd.concat([pd.Series(train_reg1),pd.Series(train_reg2),pd.Series(train_reg3)],axis=1).values,y_train_true)
    Stacking_result = model_L2.predict(pd.concat([pd.Series(test_pre1),pd.Series(test_pre2),pd.Series(test_pre3)],axis=1).values)
    return Stacking_result

## 生成一些简单的样本数据，test_prei 代表第i个模型的预测值
train_reg1 = [3.2, 8.2, 9.1, 5.2]
train_reg2 = [2.9, 8.1, 9.0, 4.9]
train_reg3 = [3.1, 7.9, 9.2, 5.0]
# y_test_true 代表第模型的真实值
y_train_true = [3, 8, 9, 5] 

test_pre1 = [1.2, 3.2, 2.1, 6.2]
test_pre2 = [0.9, 3.1, 2.0, 5.9]
test_pre3 = [1.1, 2.9, 2.2, 6.0]

# y_test_true 代表第模型的真实值
y_test_true = [1, 3, 2, 6] 

model_L2= linear_model.LinearRegression()
Stacking_pre = Stacking_method(train_reg1,train_reg2,train_reg3,y_train_true,
                               test_pre1,test_pre2,test_pre3,model_L2)
print('Stacking_pre MAE:',metrics.mean_absolute_error(y_test_true, Stacking_pre))

Stacking_pre MAE: 0.0421348314607
can find further improved results relative to the prior model, which we need to note is that, for Stacking model selected second layer should not be too complicated, this will lead to over-fitting model on the training set, so that on the test set and can not achieve good results.

Finally, tell us about RMSE , MSE , MAE , SD concept.

1. The RMSE (the root Mean Square error) root mean square error
   measure between the observed value and the true value of the deviation, as a machine learning models used to predict the result of measure of the standard.
   
2. MSE (Mean Square error) the mean square error
   MSE is the square of the difference and the predicted value of the true value and then averaging, in the form of square to facilitate derivation, so often used as a linear regression function loss.
   
3. MAE (Mean Absolute error) mean absolute error
   is the mean absolute error, may better reflect the actual situation of the prediction error.
   
4. The SD (Standard Deviation) standard differential
   arithmetic average variance for a set of data to measure the degree of dispersion.