Keras(十七)关于feature_column的使用、keras模型转tf.estimator

本文将介绍：

• 加载Titanic数据集
• 使用feature_column做数据处理,并转化为tf.data.dataset类型数据
• keras_to_estimator

一，加载Titanic数据集

1，下载Titanic数据集,使用pandas读取并解析数据集

# 在如下的两个网址下载数据
# https://storage.googleapis.com/tf-datasets/titanic/train.csv
# https://storage.googleapis.com/tf-datasets/titanic/eval.csv

import matplotlib as mpl
import matplotlib.pyplot as plt
import numpy as np
import sklearn
import pandas as pd
import os
import sys
import time
import tensorflow as tf
from tensorflow import keras

# 打印使用的python库的版本信息
print(tf.__version__)
print(sys.version_info)
for module in mpl, np, pd, sklearn, tf, keras:
    print(module.__name__, module.__version__)

train_file = "./data/titanic/train.csv"
eval_file = "./data/titanic/eval.csv"

train_df = pd.read_csv(train_file)
eval_df = pd.read_csv(eval_file)

print(train_df.head()) # 默认取出前5条数据
print(eval_df.head())

2，分离出特征值和目标值

y_train = train_df.pop('survived')
y_eval = eval_df.pop('survived')

print(train_df.head())
print(eval_df.head())
print(y_train.head())
print(y_eval.head())

3，使用panda对数值型数据的字段进行统计

print(train_df.describe())

# ---output------
              age  n_siblings_spouses       parch        fare
count  627.000000          627.000000  627.000000  627.000000
mean    29.631308            0.545455    0.379585   34.385399
std     12.511818            1.151090    0.792999   54.597730
min      0.750000            0.000000    0.000000    0.000000
25%     23.000000            0.000000    0.000000    7.895800
50%     28.000000            0.000000    0.000000   15.045800
75%     35.000000            1.000000    0.000000   31.387500
max     80.000000            8.000000    5.000000  512.329200

4,查看数据集中的测试集,验证集的数据维度

print(train_df.shape, eval_df.shape)

# ---output------
(627, 9) (264, 9)

5,使用pands中的matplotlib绘制图表,更直观的了解数据

1）统计-年龄直观图

train_df.age.hist(bins = 50)# bins是将所有数据分为多少份

2）统计-性别直观图

# value_counts() --> 将value归类并按类计数
train_df.sex.value_counts().plot(kind = 'barh') # 横向的柱状图是"barh";纵向的柱状图"bar"

3）统计-不同仓位的乘客各有多少

train_df['class'].value_counts().plot(kind = 'barh')

4）统计-在Titanic中,男性有多少人获救了,女性有多少人获救了

pd.concat([train_df, y_train], axis = 1).groupby('sex').survived.mean()
pd.concat([train_df, y_train], axis = 1).groupby('sex').survived.mean().plot(kind='barh')

二，使用feature_column做数据处理,并转化为tf.data.dataset类型数据

1，将"离散特征"和"连续特征"整合为one-hot编码

1）将特征分为"离散特征"和"连续特征"两个列表

categorical_columns = ['sex', 'n_siblings_spouses', 'parch', 'class','deck', 'embark_town', 'alone']
numeric_columns = ['age', 'fare']

feature_columns = []

2）使用tf.feature_column对"离散特征"做处理

for categorical_column in categorical_columns:
    vocab = train_df[categorical_column].unique()
    print(categorical_column, vocab)
    feature_columns.append(
        tf.feature_column.indicator_column(
            tf.feature_column.categorical_column_with_vocabulary_list(
                categorical_column, vocab)))
                
# ---output------
sex ['male' 'female']
n_siblings_spouses [1 0 3 4 2 5 8]
parch [0 1 2 5 3 4]
class ['Third' 'First' 'Second']
deck ['unknown' 'C' 'G' 'A' 'B' 'D' 'F' 'E']
embark_town ['Southampton' 'Cherbourg' 'Queenstown' 'unknown']
alone ['n' 'y']

3）使用tf.feature_column对"连续特征"做处理

for categorical_column in numeric_columns:
    feature_columns.append(
        tf.feature_column.numeric_column(
            categorical_column, dtype=tf.float32))

2，将ndarray数据转化为tf.data.dataset中的BatchDataset类型数据

def make_dataset(data_df, label_df, epochs = 10, shuffle = True,batch_size = 32):
    dataset = tf.data.Dataset.from_tensor_slices((dict(data_df), label_df))
    if shuffle:
        dataset = dataset.shuffle(10000)
    dataset = dataset.repeat(epochs).batch(batch_size)
    return dataset

train_dataset = make_dataset(train_df, y_train, batch_size = 5)

# 查看转化后的tf.data.dataset中的一条数据的信息
for x, y in train_dataset.take(1):
    print(x, y)

# ---output---------
{
    
    'sex': <tf.Tensor: shape=(5,), dtype=string, numpy=array([b'female', b'male', b'male', b'male', b'male'], dtype=object)>, 'age': <tf.Tensor: shape=(5,), dtype=float64, numpy=array([32., 28., 44., 28., 28.])>, 'n_siblings_spouses': <tf.Tensor: shape=(5,), dtype=int32, numpy=array([1, 0, 1, 0, 0], dtype=int32)>, 'parch': <tf.Tensor: shape=(5,), dtype=int32, numpy=array([1, 0, 0, 0, 0], dtype=int32)>, 'fare': <tf.Tensor: shape=(5,), dtype=float64, numpy=array([15.5   ,  7.2292, 26.    ,  8.05  ,  7.8958])>, 'class': <tf.Tensor: shape=(5,), dtype=string, numpy=array([b'Third', b'Third', b'Second', b'Third', b'Third'], dtype=object)>, 'deck': <tf.Tensor: shape=(5,), dtype=string, numpy=
array([b'unknown', b'unknown', b'unknown', b'unknown', b'unknown'],
      dtype=object)>, 'embark_town': <tf.Tensor: shape=(5,), dtype=string, numpy=
array([b'Queenstown', b'Cherbourg', b'Southampton', b'Southampton',
       b'Southampton'], dtype=object)>, 'alone': <tf.Tensor: shape=(5,), dtype=string, numpy=array([b'n', b'y', b'n', b'y', b'y'], dtype=object)>} tf.Tensor([0 1 0 0 0], shape=(5,), dtype=int32)

3，使用keras.layers.DenseFeature将一条数据其中两个字段转化为one-hot处理后的数据

# keras.layers.DenseFeature
for x, y in train_dataset.take(1):
    age_column = feature_columns[7]
    gender_column = feature_columns[0]
    print(keras.layers.DenseFeatures(age_column)(x).numpy())
    print(keras.layers.DenseFeatures(gender_column)(x).numpy())

# ---output----------
[[28.]
 [50.]
 [27.]
 [28.]
 [32.]]

[[1. 0.]
 [0. 1.]
 [0. 1.]
 [0. 1.]
 [1. 0.]]

4，使用keras.layers.DenseFeature将一条数据中所有字段转化为one-hot处理后的数据

# keras.layers.DenseFeature
for x, y in train_dataset.take(1):
    print(keras.layers.DenseFeatures(feature_columns)(x).numpy())

三，keras_to_estimator

1，定义keras模型,输入层输入为转化为one-hot处理后的数据

model = keras.models.Sequential([
    keras.layers.DenseFeatures(feature_columns),
    keras.layers.Dense(100, activation='relu'),
    keras.layers.Dense(100, activation='relu'),
    keras.layers.Dense(2, activation='softmax'),
])
model.compile(loss='sparse_categorical_crossentropy',
              optimizer = keras.optimizers.SGD(lr=0.01),
              metrics = ['accuracy'])

2，训练模型

训练模型可以使用如下两种方法：

1）使用普通的model模型训练

train_dataset = make_dataset(train_df, y_train, epochs = 100)
eval_dataset = make_dataset(eval_df, y_eval, epochs = 1, shuffle = False)
history = model.fit(train_dataset,
                    validation_data = eval_dataset,
                    steps_per_epoch = 19,
                    validation_steps = 8,
                    epochs = 100)

2）使用转化为estimator后的model模型训练

注：在tensorflow2中该方法还存在bug，待解决。

estimator = keras.estimator.model_to_estimator(model)
# 1. function
# 2. return a. (features, labels) b. dataset -> (feature, label)
estimator.train(input_fn = lambda : make_dataset(
    train_df, y_train, epochs=100))