ML之PySpark：基于PySpark框架针对boston波士顿房价数据集利用lightgbm算法(评估)实现房价回归预测案例应用
目录
基于PySpark框架针对boston波士顿房价数据集利用lightgbm算法(评估)实现房价回归预测案例应用
# 3.1.2、数据类型转换：将spark_df类型转换为pandas_df类型
相关文章
ML之PySpark：基于PySpark框架针对boston波士顿房价数据集利用lightgbm算法(评估)实现房价回归预测案例应用
ML之PySpark：基于PySpark框架针对boston波士顿房价数据集利用lightgbm算法(评估)实现房价回归预测案例应用实现代码
基于PySpark框架针对boston波士顿房价数据集利用lightgbm算法(评估)实现房价回归预测案例应用
# 1、定义数据集
# 1.1、创建SparkSession连接
# 1.2、读取数据集
# 去除无意义的索引列
+---+-------+----+-----+----+-----+-----+----+------+---+---+-------+------+-----+----+
| ID| crim| zn|indus|chas| nox| rm| age| dis|rad|tax|ptratio| black|lstat|medv|
+---+-------+----+-----+----+-----+-----+----+------+---+---+-------+------+-----+----+
| 1|0.00632|18.0| 2.31| 0|0.538|6.575|65.2| 4.09| 1|296| 15.3| 396.9| 4.98|24.0|
| 2|0.02731| 0.0| 7.07| 0|0.469|6.421|78.9|4.9671| 2|242| 17.8| 396.9| 9.14|21.6|
| 4|0.03237| 0.0| 2.18| 0|0.458|6.998|45.8|6.0622| 3|222| 18.7|394.63| 2.94|33.4|
+---+-------+----+-----+----+-----+-----+----+------+---+---+-------+------+-----+----+
only showing top 3 rows
root
|-- ID: integer (nullable = true)
|-- crim: double (nullable = true)
|-- zn: double (nullable = true)
|-- indus: double (nullable = true)
|-- chas: integer (nullable = true)
|-- nox: double (nullable = true)
|-- rm: double (nullable = true)
|-- age: double (nullable = true)
|-- dis: double (nullable = true)
|-- rad: integer (nullable = true)
|-- tax: integer (nullable = true)
|-- ptratio: double (nullable = true)
|-- black: double (nullable = true)
|-- lstat: double (nullable = true)
|-- medv: double (nullable = true)
15 ['ID', 'crim', 'zn', 'indus', 'chas', 'nox', 'rm', 'age', 'dis', 'rad', 'tax', 'ptratio', 'black', 'lstat', 'medv']
+-------+----+-----+----+-----+-----+----+------+---+---+-------+------+-----+----+
| crim| zn|indus|chas| nox| rm| age| dis|rad|tax|ptratio| black|lstat|medv|
+-------+----+-----+----+-----+-----+----+------+---+---+-------+------+-----+----+
|0.00632|18.0| 2.31| 0|0.538|6.575|65.2| 4.09| 1|296| 15.3| 396.9| 4.98|24.0|
|0.02731| 0.0| 7.07| 0|0.469|6.421|78.9|4.9671| 2|242| 17.8| 396.9| 9.14|21.6|
|0.03237| 0.0| 2.18| 0|0.458|6.998|45.8|6.0622| 3|222| 18.7|394.63| 2.94|33.4|
+-------+----+-----+----+-----+-----+----+------+---+---+-------+------+-----+----+
only showing top 3 rows# 2、数据预处理/特征工程
# 2.1、定义特征向量化
# 2.2、合并特征与标签
23/04/12 18:05:04 WARN package: Truncated the string representation of a plan since it was too large. This behavior can be adjusted by setting 'spark.sql.debug.maxToStringFields'.
+----+--------------------+-------+----+-----+----+-----+-----+----+------+---+---+-------+------+-----+
|medv| features| crim| zn|indus|chas| nox| rm| age| dis|rad|tax|ptratio| black|lstat|
+----+--------------------+-------+----+-----+----+-----+-----+----+------+---+---+-------+------+-----+
|24.0|[0.00632,18.0,2.3...|0.00632|18.0| 2.31| 0|0.538|6.575|65.2| 4.09| 1|296| 15.3| 396.9| 4.98|
|21.6|[0.02731,0.0,7.07...|0.02731| 0.0| 7.07| 0|0.469|6.421|78.9|4.9671| 2|242| 17.8| 396.9| 9.14|
|33.4|[0.03237,0.0,2.18...|0.03237| 0.0| 2.18| 0|0.458|6.998|45.8|6.0622| 3|222| 18.7|394.63| 2.94|
+----+--------------------+-------+----+-----+----+-----+-----+----+------+---+---+-------+------+-----+
only showing top 3 rows# 3、模型训练与评估
# 3.1、切分数据集
# 3.1.1、切分为训练集、验证集
# 3.1.2、数据类型转换：将spark_df类型转换为pandas_df类型
# 3.1.3、分离特征和标签
[DenseVector([38.3518, 0.0, 18.1, 0.0, 0.693, 5.453, 100.0, 1.4896, 24.0, 666.0, 20.2, 396.9, 30.59])
DenseVector([25.0461, 0.0, 18.1, 0.0, 0.693, 5.987, 100.0, 1.5888, 24.0, 666.0, 20.2, 396.9, 26.77])
DenseVector([14.2362, 0.0, 18.1, 0.0, 0.693, 6.343, 100.0, 1.5741, 24.0, 666.0, 20.2, 396.9, 20.32])
DenseVector([18.0846, 0.0, 18.1, 0.0, 0.679, 6.434, 100.0, 1.8347, 24.0, 666.0, 20.2, 27.25, 29.05])
……
DenseVector([9.2323, 0.0, 18.1, 0.0, 0.631, 6.216, 100.0, 1.1691, 24.0, 666.0, 20.2, 366.15, 9.53])]
[[ 25.0461 0. 18.1 ... 20.2 396.9 26.77 ]
[ 45.7461 0. 18.1 ... 20.2 88.27 36.98 ]
[ 14.2362 0. 18.1 ... 20.2 396.9 20.32 ]
...
[ 4.89822 0. 18.1 ... 20.2 375.52 3.26 ]
[ 6.53876 0. 18.1 ... 20.2 392.05 2.96 ]
[ 9.2323 0. 18.1 ... 20.2 366.15 9.53 ]]
[ 5. 5.6 7.2 7.2 7.4 8.1 8.3 8.4 8.7 8.8 8.8 9.5 9.7 10.4
10.5 10.5 10.8 10.9 11. 11.3 11.7 12. 12.3 13.1 13.4 13.5 13.5 13.6
……
32.9 33. 33.1 33.4 33.4 33.8 34.9 34.9 35.1 35.2 36.1 36.2 36.2 36.4
37. 37.2 37.6 37.9 39.8 41.7 43.1 44.8 46. 48.3 48.5 48.8 50. 50.
50. 50. 50. 50. 50. 50. 50. ]# 3.1.4、构建LightGBM数据集
lgb_train
<lightgbm.basic.Dataset object at 0x000001E3B0A239D0>
lgb_eval
<lightgbm.basic.Dataset object at 0x000001E3B0A234C0># 3.2、模型训练
# 设置模型参数
# 训练模型
[LightGBM] [Warning] Auto-choosing col-wise multi-threading, the overhead of testing was 0.000313 seconds.
You can set `force_col_wise=true` to remove the overhead.
[LightGBM] [Info] Total Bins 638
[LightGBM] [Info] Number of data points in the train set: 251, number of used features: 12
[LightGBM] [Info] Start training from score 23.116335
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[1] valid_0's l1: 5.55296 valid_0's l2: 55.3567
Training until validation scores don't improve for 5 rounds
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[2] valid_0's l1: 5.37904 valid_0's l2: 52.1612
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[3] valid_0's l1: 5.18424 valid_0's l2: 48.5647
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[4] valid_0's l1: 4.98621 valid_0's l2: 45.1861
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[5] valid_0's l1: 4.81712 valid_0's l2: 42.6588
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[6] valid_0's l1: 4.64324 valid_0's l2: 39.9921
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[7] valid_0's l1: 4.48456 valid_0's l2: 37.6254
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[8] valid_0's l1: 4.32798 valid_0's l2: 35.562
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[9] valid_0's l1: 4.1635 valid_0's l2: 33.5291
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[10] valid_0's l1: 4.04524 valid_0's l2: 31.976
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[11] valid_0's l1: 3.92398 valid_0's l2: 30.43
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[12] valid_0's l1: 3.82096 valid_0's l2: 29.178
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[13] valid_0's l1: 3.70099 valid_0's l2: 27.894
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[14] valid_0's l1: 3.5979 valid_0's l2: 26.7405
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[15] valid_0's l1: 3.5056 valid_0's l2: 25.7414
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[16] valid_0's l1: 3.42553 valid_0's l2: 24.9116
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[17] valid_0's l1: 3.33382 valid_0's l2: 23.9615
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[18] valid_0's l1: 3.26277 valid_0's l2: 23.1291
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[19] valid_0's l1: 3.19639 valid_0's l2: 22.4056
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[20] valid_0's l1: 3.13146 valid_0's l2: 21.7271
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[21] valid_0's l1: 3.06704 valid_0's l2: 21.0404
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[22] valid_0's l1: 3.01769 valid_0's l2: 20.5252
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[23] valid_0's l1: 2.97406 valid_0's l2: 19.9501
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[24] valid_0's l1: 2.93286 valid_0's l2: 19.5415
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[25] valid_0's l1: 2.89854 valid_0's l2: 19.2612
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[26] valid_0's l1: 2.86595 valid_0's l2: 18.9108
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[27] valid_0's l1: 2.83455 valid_0's l2: 18.4983
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[28] valid_0's l1: 2.80282 valid_0's l2: 18.1043
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[29] valid_0's l1: 2.7676 valid_0's l2: 17.7102
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[30] valid_0's l1: 2.73643 valid_0's l2: 17.5716
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[31] valid_0's l1: 2.7189 valid_0's l2: 17.3091
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[32] valid_0's l1: 2.70055 valid_0's l2: 17.0558
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[33] valid_0's l1: 2.67765 valid_0's l2: 16.7472
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[34] valid_0's l1: 2.66355 valid_0's l2: 16.5579
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[35] valid_0's l1: 2.65571 valid_0's l2: 16.4488
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[36] valid_0's l1: 2.64186 valid_0's l2: 16.1984
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[37] valid_0's l1: 2.62485 valid_0's l2: 15.9867
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[38] valid_0's l1: 2.60517 valid_0's l2: 15.7785
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[39] valid_0's l1: 2.60108 valid_0's l2: 15.6885
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[40] valid_0's l1: 2.58566 valid_0's l2: 15.5137
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[41] valid_0's l1: 2.56902 valid_0's l2: 15.2941
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[42] valid_0's l1: 2.55925 valid_0's l2: 15.2012
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[43] valid_0's l1: 2.55608 valid_0's l2: 15.1871
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[44] valid_0's l1: 2.55078 valid_0's l2: 15.0495
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[45] valid_0's l1: 2.54551 valid_0's l2: 14.9857
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[46] valid_0's l1: 2.54433 valid_0's l2: 14.9713
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[47] valid_0's l1: 2.54001 valid_0's l2: 14.8974
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[48] valid_0's l1: 2.53317 valid_0's l2: 14.7989
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[49] valid_0's l1: 2.52767 valid_0's l2: 14.6993
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[50] valid_0's l1: 2.53182 valid_0's l2: 14.6695
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[51] valid_0's l1: 2.525 valid_0's l2: 14.5664
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[52] valid_0's l1: 2.52136 valid_0's l2: 14.5268
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[53] valid_0's l1: 2.51818 valid_0's l2: 14.4511
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[54] valid_0's l1: 2.51344 valid_0's l2: 14.3942
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[55] valid_0's l1: 2.51794 valid_0's l2: 14.3446
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[56] valid_0's l1: 2.51472 valid_0's l2: 14.3073
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[57] valid_0's l1: 2.51281 valid_0's l2: 14.2588
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[58] valid_0's l1: 2.51246 valid_0's l2: 14.1835
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[59] valid_0's l1: 2.51186 valid_0's l2: 14.1126
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[60] valid_0's l1: 2.50594 valid_0's l2: 14.0302
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[61] valid_0's l1: 2.50899 valid_0's l2: 14.0065
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[62] valid_0's l1: 2.51171 valid_0's l2: 13.9461
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[63] valid_0's l1: 2.51573 valid_0's l2: 13.95
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[64] valid_0's l1: 2.52288 valid_0's l2: 13.9602
[LightGBM] [Warning] No further splits with positive gain, best gain: -inf
[65] valid_0's l1: 2.52993 valid_0's l2: 13.9526
Early stopping, best iteration is:
[60] valid_0's l1: 2.50594 valid_0's l2: 14.0302
# 3.3、模型预测与评估
R2 = 0.757685
RMSE = 3.74569