Example of XGBoost - Pima Indian Diabetes Prediction and Feature Screening

Use the Pima Indian diabetes data set to predict the diabetes of the Pima Indians. The following is the information of the data set:

• Pregnencies: Number of pregnancies
• Glucose: glucose
• BloodPressure: blood pressure (mm Hg)
• SkinThickness: cortical thickness (mm)
• Insulin: Insulin 2-hour serum insulin (mu U/ml)
• BMI: Body mass index (weight/height)^2
• DiabetesPedigreeFunction: Diabetes Pedigree Function
• Age: age (years)
• Outcome: target value (0 or 1)

import module

# 导入模块包
import pandas as pd
from sklearn.model_selection import train_test_split
import xgboost as xgb
import warnings
warnings.filterwarnings('ignore')
from sklearn.metrics import roc_auc_score, roc_curve, confusion_matrix, classification_report
import matplotlib.pyplot as plt
import seaborn as sns

read data

df = pd.read_csv('pima-indians-diabetes.csv')
print(df.info())
df.head()

Since the data is relatively complete and there is no problem of missing data, the data does not need to be processed.

predict directly

# 数据划分
feature_columns = ['Pregnancies', 'Glucose', 'BloodPressure', 'SkinThickness', 'Insulin', 'BMI', 'DiabetesPedigreeFunction', 'Age']
X = df[feature_columns]
y = df['Outcome']
train_X, test_X, train_y, test_y = train_test_split(X,y,random_state=7,test_size=0.2)
# 模型设置和训练
xgb_clf = xgb.XGBClassifier(n_estimators=20, max_depth=4,learning_rate=0.1,subsample=0.7,colsample_bytree=0.7)
xgb_clf.fit(train_X, train_y)
pred_y = xgb_clf.predict(test_X)
prob_y = xgb_clf.predict_proba(test_X)[:,1]
prob_train_y = xgb_clf.predict_proba(train_X)[:,1]

# 模型评估
auc_score = roc_auc_score(test_y, pred_y)
auc_score_train = roc_auc_score(train_y, prob_train_y)
fpr, tpr,_ = roc_curve(test_y, prob_y)
fpr_tr, tpr_tr,_ = roc_curve(train_y, prob_train_y)

# 绘制roc曲线
plt.plot(fpr,tpr,label = 'test xgb auc=%0.3f'%auc_score) #绘制训练集ROC 
plt.plot(fpr_tr,tpr_tr,label = 'train xgb auc=%0.3f'%auc_score_train) #绘制验证集ROC 
plt.plot([0,1],[0,1],'k--') 
plt.xlabel('False positive rate') 
plt.ylabel('True positive rate') 
plt.title('ROC Curve') 
plt.legend(loc = 'best') 
plt.show()
print(confusion_matrix(pred_y,test_y))
print((pred_y!=test_y).sum()/float(test_y.shape[0]))
print(classification_report(test_y,pred_y,  target_names=['Yes','No']))

Use xgboost to filter the features. Since the above has been directly used for classification, we can directly extract the indicators of the features.

# 使用xgboost进行特征筛选
temp=pd.DataFrame()
temp['feature_name'] = feature_columns
temp['feature_importance'] = xgb_clf.feature_importances_
temp.sort_values('feature_importance', ascending=False)

Use the filtered features for model training, and use the top four features for training.

# 使用大于0.1的特征进行训练
feature_lst = ['Glucose','BMI','Age','Insulin']
X = df[feature_lst]
y = df['Outcome']
train_X, test_X, train_y, test_y = train_test_split(X,y,random_state=7,test_size=0.2)
# 模型设置和训练
xgb_clf = xgb.XGBClassifier(n_estimators=20, max_depth=4,learning_rate=0.1,subsample=0.7,colsample_bytree=0.7)
xgb_clf.fit(train_X, train_y)
pred_y = xgb_clf.predict(test_X)
prob_y = xgb_clf.predict_proba(test_X)[:,1]
prob_train_y = xgb_clf.predict_proba(train_X)[:,1]

# 模型评估
auc_score = roc_auc_score(test_y, pred_y)
auc_score_train = roc_auc_score(train_y, prob_train_y)
fpr, tpr,_ = roc_curve(test_y, prob_y)
fpr_tr, tpr_tr,_ = roc_curve(train_y, prob_train_y)

# 绘制roc曲线
plt.plot(fpr,tpr,label = 'test xgb auc=%0.3f'%auc_score) #绘制训练集ROC 
plt.plot(fpr_tr,tpr_tr,label = 'train xgb auc=%0.3f'%auc_score_train) #绘制验证集ROC 
plt.plot([0,1],[0,1],'k--') 
plt.xlabel('False positive rate') 
plt.ylabel('True positive rate') 
plt.title('ROC Curve') 
plt.legend(loc = 'best') 
plt.show()
print(confusion_matrix(pred_y,test_y))
print((pred_y!=test_y).sum()/float(test_y.shape[0]))
print(classification_report(test_y,pred_y,  target_names=['Yes','No']))

Summarize

• The model after feature filtering is not enhanced
• The auc value of the training set and the test set varies greatly, and the generalization ability is weak, so K-fold verification of the data is required.