本文将会介绍如何使用keras-bert实现文本多标签分类任务，其中对BERT进行微调。
项目结构
本项目的项目结构如下：
其中依赖的Python第三方模块如下：
pandas==0.23.4
Keras==2.3.1
keras_bert==0.83.0
numpy==1.16.4
数据集介绍
本文采用的数据集与文章NLP（二十八）多标签文本分类中的一致，以事件抽取比赛的数据集为参考，形成文本与事件类型的多标签数据集，一共为65种事件类型。样例数据（csv格式）如下：
label,content
司法行为-起诉|组织关系-裁员,最近，一位前便利蜂员工就因公司违规裁员，将便利蜂所在的公司虫极科技（北京）有限公司告上法庭。
组织关系-裁员,思科上海大规模裁员人均可获赔100万官方澄清事实
组织关系-裁员,日本巨头面临危机，已裁员1000多人，苹果也救不了它！
组织关系-裁员|组织关系-解散,在硅谷镀金失败的造车新势力们：蔚来裁员、奇点被偷窃、拜腾解散
在label中，每个事件类型用|隔开。
在该数据集中，训练集一共11958个样本，测试集一共1498个样本。
模型训练
模型训练的脚本model_train.py的完整代码如下：
# -*- coding: utf-8 -*-
import json
import codecs
import pandas as pd
import numpy as np
from keras_bert import load_trained_model_from_checkpoint, Tokenizer
from keras.layers import *
from keras.models import Model
from keras.optimizers import Adam
# 建议长度<=510
maxlen = 256
BATCH_SIZE = 8
config_path = './chinese_L-12_H-768_A-12/bert_config.json'
checkpoint_path = './chinese_L-12_H-768_A-12/bert_model.ckpt'
dict_path = './chinese_L-12_H-768_A-12/vocab.txt'
token_dict = {
}
with codecs.open(dict_path, 'r', 'utf-8') as reader:
for line in reader:
token = line.strip()
token_dict[token] = len(token_dict)
class OurTokenizer(Tokenizer):
def _tokenize(self, text):
R = []
for c in text:
if c in self._token_dict:
R.append(c)
else:
R.append('[UNK]') # 剩余的字符是[UNK]
return R
tokenizer = OurTokenizer(token_dict)
def seq_padding(X, padding=0):
L = [len(x) for x in X]
ML = max(L)
return np.array([
np.concatenate([x, [padding] * (ML - len(x))]) if len(x) < ML else x for x in X
])
class DataGenerator:
def __init__(self, data, batch_size=BATCH_SIZE):
self.data = data
self.batch_size = batch_size
self.steps = len(self.data) // self.batch_size
if len(self.data) % self.batch_size != 0:
self.steps += 1
def __len__(self):
return self.steps
def __iter__(self):
while True:
idxs = list(range(len(self.data)))
np.random.shuffle(idxs)
X1, X2, Y = [], [], []
for i in idxs:
d = self.data[i]
text = d[0][:maxlen]
x1, x2 = tokenizer.encode(first=text)
y = d[1]
X1.append(x1)
X2.append(x2)
Y.append(y)
if len(X1) == self.batch_size or i == idxs[-1]:
X1 = seq_padding(X1)
X2 = seq_padding(X2)
Y = seq_padding(Y)
yield [X1, X2], Y
[X1, X2, Y] = [], [], []
# 构建模型
def create_cls_model(num_labels):
bert_model = load_trained_model_from_checkpoint(config_path, checkpoint_path, seq_len=None)
for layer in bert_model.layers:
layer.trainable = True
x1_in = Input(shape=(None,))
x2_in = Input(shape=(None,))
x = bert_model([x1_in, x2_in])
cls_layer = Lambda(lambda x: x[:, 0])(x) # 取出[CLS]对应的向量用来做分类
p = Dense(num_labels, activation='sigmoid')(cls_layer) # 多分类
model = Model([x1_in, x2_in], p)
model.compile(
loss='binary_crossentropy',
optimizer=Adam(1e-5), # 用足够小的学习率
metrics=['accuracy']
)
model.summary()
return model
if __name__ == '__main__':
# 数据处理, 读取训练集和测试集
print("begin data processing...")
train_df = pd.read_csv("data/train.csv").fillna(value="")
test_df = pd.read_csv("data/test.csv").fillna(value="")
select_labels = train_df["label"].unique()
labels = []
for label in select_labels:
if "|" not in label:
if label not in labels:
labels.append(label)
else:
for _ in label.split("|"):
if _ not in labels:
labels.append(_)
with open("label.json", "w", encoding="utf-8") as f:
f.write(json.dumps(dict(zip(range(len(labels)), labels)), ensure_ascii=False, indent=2))
train_data = []
test_data = []
for i in range(train_df.shape[0]):
label, content = train_df.iloc[i, :]
label_id = [0] * len(labels)
for j, _ in enumerate(labels):
for separate_label in label.split("|"):
if _ == separate_label:
label_id[j] = 1
train_data.append((content, label_id))
for i in range(test_df.shape[0]):
label, content = test_df.iloc[i, :]
label_id = [0] * len(labels)
for j, _ in enumerate(labels):
for separate_label in label.split("|"):
if _ == separate_label:
label_id[j] = 1
test_data.append((content, label_id))
# print(train_data[:10])
print("finish data processing!")
# 模型训练
model = create_cls_model(len(labels))
train_D = DataGenerator(train_data)
test_D = DataGenerator(test_data)
print("begin model training...")
model.fit_generator(
train_D.__iter__(),
steps_per_epoch=len(train_D),
epochs=10,
validation_data=test_D.__iter__(),
validation_steps=len(test_D)
)
print("finish model training!")
# 模型保存
model.save('multi-label-ee.h5')
print("Model saved!")
result = model.evaluate_generator(test_D.__iter__(), steps=len(test_D))
print("模型评估结果:", result)
模型结构如下：
__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
input_1 (InputLayer) (None, None) 0
__________________________________________________________________________________________________
input_2 (InputLayer) (None, None) 0
__________________________________________________________________________________________________
model_2 (Model) (None, None, 768) 101677056 input_1[0][0]
input_2[0][0]
__________________________________________________________________________________________________
lambda_1 (Lambda) (None, 768) 0 model_2[1][0]
__________________________________________________________________________________________________
dense_1 (Dense) (None, 65) 49985 lambda_1[0][0]
==================================================================================================
Total params: 101,727,041
Trainable params: 101,727,041
Non-trainable params: 0
__________________________________________________________________________________________________
从中我们可以发现，该模型结构与文章NLP（三十五）使用keras-bert实现文本多分类任务中给出的文本多分类模型结构大体一致，修改之处在于BERT后接的网络结构，所接的依然是dense层，但激活函数采用sigmoid函数，同时损失函数为binary_crossentropy。就其本质而言，该模型结构是对输出的65个结果采用0-1分类，故而激活函数采用sigmoid，这当然是文本多分类模型转化为多标签标签的最便捷方式，但不足之处在于，该模型并未考虑标签之间的依赖关系。
模型评估
模型评估脚本model_evaluate.py的完整代码如下：
# -*- coding: utf-8 -*-
# @Time : 2020/12/23 15:28
# @Author : Jclian91
# @File : model_evaluate.py
# @Place : Yangpu, Shanghai
# 模型评估脚本,利用hamming_loss作为多标签分类的评估指标，该值越小模型效果越好
import json
import numpy as np
import pandas as pd
from keras.models import load_model
from keras_bert import get_custom_objects
from sklearn.metrics import hamming_loss, classification_report
from model_train import token_dict, OurTokenizer
maxlen = 256
# 加载训练好的模型
model = load_model("multi-label-ee.h5", custom_objects=get_custom_objects())
tokenizer = OurTokenizer(token_dict)
with open("label.json", "r", encoding="utf-8") as f:
label_dict = json.loads(f.read())
# 对单句话进行预测
def predict_single_text(text):
# 利用BERT进行tokenize
text = text[:maxlen]
x1, x2 = tokenizer.encode(first=text)
X1 = x1 + [0] * (maxlen - len(x1)) if len(x1) < maxlen else x1
X2 = x2 + [0] * (maxlen - len(x2)) if len(x2) < maxlen else x2
# 模型预测并输出预测结果
prediction = model.predict([[X1], [X2]])
one_hot = np.where(prediction > 0.5, 1, 0)[0]
return one_hot, "|".join([label_dict[str(i)] for i in range(len(one_hot)) if one_hot[i]])
# 模型评估
def evaluate():
test_df = pd.read_csv("data/test.csv").fillna(value="")
true_y_list, pred_y_list = [], []
true_label_list, pred_label_list = [], []
common_cnt = 0
for i in range(test_df.shape[0]):
print("predict %d samples" % (i+1))
true_label, content = test_df.iloc[i, :]
true_y = [0] * len(label_dict.keys())
for key, value in label_dict.items():
if value in true_label:
true_y[int(key)] = 1
pred_y, pred_label = predict_single_text(content)
if true_label == pred_label:
common_cnt += 1
true_y_list.append(true_y)
pred_y_list.append(pred_y)
true_label_list.append(true_label)
pred_label_list.append(pred_label)
# F1值
print(classification_report(true_y_list, pred_y_list, digits=4))
return true_label_list, pred_label_list, hamming_loss(true_y_list, pred_y_list), common_cnt/len(true_y_list)
true_labels, pred_labels, h_loss, accuracy = evaluate()
df = pd.DataFrame({
"y_true": true_labels, "y_pred": pred_labels})
df.to_csv("pred_result.csv")
print("accuracy: ", accuracy)
print("hamming loss: ", h_loss)
Hamming Loss为多标签分类所特有的评估方式，其值越小代表多标签分类模型的效果越好。运行上述模型评估代码，输出结果如下：
micro avg 0.9341 0.9578 0.9458 1657
macro avg 0.9336 0.9462 0.9370 1657
weighted avg 0.9367 0.9578 0.9456 1657
samples avg 0.9520 0.9672 0.9531 1657
accuracy: 0.8985313751668892
hamming loss: 0.001869158878504673
在这里，笔者希望与之前的文章NLP（二十八）多标签文本分类中的模型对比一下。当时采用的模型为用ALBERT提取特征向量，再用Bi-GRU+Attention+FCN进行分类，模型结构如下：
对该模型同样采用上述评估办法，输出的结果如下：
micro avg 0.9424 0.8292 0.8822 1657
macro avg 0.8983 0.7218 0.7791 1657
weighted avg 0.9308 0.8292 0.8669 1657
samples avg 0.8675 0.8496 0.8517 1657
accuracy: 0.7983978638184246
hamming loss: 0.0037691280681934887
可以发现，采用BERT微调的模型，在accuracy方面高出了约10%，各种F1值高出约5%-10%，Hamming Loss也小了很多。因此，BERT微调的模型比之前的模型效果好很多。
总结
本项目已经开源，Github地址为：https://github.com/percent4/keras_bert_multi_label_cls 。
2020年12月27日于上海浦东
参考文章
- NLP（二十八）多标签文本分类：https://blog.csdn.net/jclian91/article/details/105386190
- NLP（三十五）使用keras-bert实现文本多分类任务：https://blog.csdn.net/jclian91/article/details/111742576