Sklearn–(2)
首先介绍数据标准化工具onehot[独热编码]:
One-Hot编码，又称为一位有效编码，主要是采用N位状态寄存器来对N个状态进行编码，每个状态都由他独立的寄存器位，并且在任意时候只有一位有效。以手写数字识别为例，我们需要将0-9共十个数字标签转化成onehot标签。例如：数字标签“6”转化为onehot标签就是[0，0，0，0，0，0，1，0，0，0].
我第一次利用以下函数进行编码：
def convert2onehot(data):
# covert data to onehot representation
return pd.get_dummies(data, prefix=data.columns)结果报错：
很明显职业一项我们一共有15个而该函数只能提供10个以下的编码，只能另觅他法喽，
ValueError: Length of 'prefix' (15) did not match the length of the columns being encoded (9).虽然已经有很多人在 stackoverflow 和 sklearn 的 github issue 上讨论过这个问题，但目前为止的 sklearn 版本仍没有增加OneHotEncoder对字符串型类别变量的支持，所以一般都采用曲线救国的方式：
- 方法一 ：先用 LabelEncoder() 转换成连续的数值型变量，再用 OneHotEncoder() 二值化；
- 方法二 ：直接用 LabelBinarizer() 进行二值化；
import pandas as pd
#import Numpy as np
from sklearn.preprocessing import OneHotEncoder
from sklearn.preprocessing import LabelEncoder
from sklearn.preprocessing import LabelBinarizer
from sklearn.preprocessing import MultiLabelBinarizer
col_names = ["age", "workclass", "fnlwgt", "education", "education-num", "marital-status", "occupation",
"relationship", "race", "sex", "capital-gain", "capital-loss", "hours-per-week", "native-country", "result"]
data = pd.read_csv("D://ML//Data//adult.csv", names=col_names)
#利用函数将文字标准化为tensorflow可以处理的数字
#OneHotEncoder(sparse = False).fit_transform( data[['age','education-num']] )
data['age']=LabelBinarizer().fit_transform(data['age'])
data['workclass']=LabelBinarizer().fit_transform(data['workclass'])
data['fnlwgt']=data['fnlwgt']/100
data['fnlwgt']=OneHotEncoder(sparse = False).fit_transform( data[['fnlwgt']] )
data['education']=LabelBinarizer().fit_transform(data['education'])
data['education-num']=LabelBinarizer().fit_transform(data['education-num'])
data['marital-status']=LabelBinarizer().fit_transform(data['marital-status'])
data['occupation']=LabelBinarizer().fit_transform(data['occupation'])
data['relationship']=LabelBinarizer().fit_transform(data['relationship'])
data['race']=LabelBinarizer().fit_transform(data['race'])
data['sex']=LabelBinarizer().fit_transform(data['sex'])
data['capital-gain']=LabelBinarizer().fit_transform(data['capital-gain'])
data['capital-loss']=LabelBinarizer().fit_transform(data['capital-loss'])
data['hours-per-week']=LabelBinarizer().fit_transform(data['hours-per-week'])
data['native-country']=LabelBinarizer().fit_transform(data['native-country'])
data['result']=LabelBinarizer().fit_transform(data['result'])
print(data[:10])打印结果如下所示：
age workclass fnlwgt education education-num marital-status \
0 0 0 0.0 0 0 0
1 0 0 0.0 0 0 0
2 0 0 0.0 0 0 1
3 0 0 0.0 0 0 0
4 0 0 0.0 0 0 0
5 0 0 0.0 0 0 0
6 0 0 0.0 0 0 0
7 0 0 0.0 0 0 0
8 0 0 0.0 0 0 0
9 0 0 0.0 0 0 0
occupation relationship race sex capital-gain capital-loss \
0 0 0 0 1 0 1
1 0 1 0 1 1 1
2 0 0 0 1 1 1
3 0 1 0 1 1 1
4 0 0 0 0 1 1
5 0 0 0 0 1 1
6 0 0 0 0 1 1
7 0 1 0 1 1 1
8 0 0 0 0 0 1
9 0 1 0 1 0 1
hours-per-week native-country result
0 0 0 0
1 0 0 0
2 0 0 0
3 0 0 0
4 0 0 0
5 0 0 0
6 0 0 0
7 0 0 1
8 0 0 1
9 0 0 1