Move your little hand to make a fortune, give it a thumbs up!
In NLP, the Transformer model architecture is a revolution that greatly enhances the ability to understand and generate textual information.
In this tutorial [1] , we will delve into BERT, a well-known Transformer-based model, and provide a practical example of fine-tuning a basic BERT model for sentiment analysis.
Introduction to BERT
Launched by Google researchers in 2018, BERT is a powerful language model using the Transformer architecture. BERT breaks the unidirectional or sequential bidirectional boundaries of earlier model architectures such as LSTM and GRU, taking into account both past and future context. This is due to an innovative "attention mechanism" that allows the model to weigh the importance of words in a sentence when generating representations.
The BERT model is pretrained for the following two NLP tasks:
-
Masked Language Model (MLM)
-
Next Sentence Prediction (NSP)
It is often used as the base model for various downstream NLP tasks, such as the sentiment analysis we will cover in this tutorial.
Pre-training and fine-tuning
The power of BERT lies in its two-step process:
-
Pre-training is the stage where BERT is trained on a large amount of data. Thus, it learns to predict masked words in sentences (MLM task) and predict whether a sentence follows another sentence (NSP task). The output of this stage is a pretrained NLP model with a general "understanding" of the language -
Fine-tuning is to further train the pre-trained BERT model for a specific task. The model is initialized with pre-trained parameters, and the entire model is trained on downstream tasks, enabling BERT to fine-tune its understanding of language to the specifics of the task at hand.
Hands-On: Sentiment Analysis with BERT
The full code is available as a Jupyter Notebook on GitHub
In this hands-on exercise, we will train a sentiment analysis model on the IMDB movie review dataset (license: Apache 2.0), which
会标记评论是正面还是负面。我们还将使用 Hugging Face 的转换器库加载模型。
让我们加载所有库
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.metrics import confusion_matrix, roc_curve, auc
from datasets import load_dataset
from transformers import AutoTokenizer, AutoModelForSequenceClassification, TrainingArguments, Trainer
# Variables to set the number of epochs and samples
num_epochs = 10
num_samples = 100 # set this to -1 to use all data
首先，我们需要加载数据集和模型标记器。
# Step 1: Load dataset and model tokenizer
dataset = load_dataset('imdb')
tokenizer = AutoTokenizer.from_pretrained('bert-base-uncased')
接下来，我们将创建一个绘图来查看正类和负类的分布。
# Data Exploration
train_df = pd.DataFrame(dataset["train"])
sns.countplot(x='label', data=train_df)
plt.title('Class distribution')
plt.show()
接下来，我们通过标记文本来预处理数据集。我们使用 BERT 的标记器，它将文本转换为与 BERT 词汇相对应的标记。
# Step 2: Preprocess the dataset
def tokenize_function(examples):
return tokenizer(examples["text"], padding="max_length", truncation=True)
tokenized_datasets = dataset.map(tokenize_function, batched=True)
之后，我们准备训练和评估数据集。请记住，如果您想使用所有数据，可以将 num_samples 变量设置为 -1。
if num_samples == -1:
small_train_dataset = tokenized_datasets["train"].shuffle(seed=42)
small_eval_dataset = tokenized_datasets["test"].shuffle(seed=42)
else:
small_train_dataset = tokenized_datasets["train"].shuffle(seed=42).select(range(num_samples))
small_eval_dataset = tokenized_datasets["test"].shuffle(seed=42).select(range(num_samples))
然后，我们加载预训练的 BERT 模型。我们将使用 AutoModelForSequenceClassification 类，这是一个专为分类任务设计的 BERT 模型。
# Step 3: Load pre-trained model
model = AutoModelForSequenceClassification.from_pretrained('bert-base-uncased', num_labels=2)
现在，我们准备定义训练参数并创建一个 Trainer 实例来训练我们的模型。
# Step 4: Define training arguments
training_args = TrainingArguments("test_trainer", evaluation_strategy="epoch", no_cuda=True, num_train_epochs=num_epochs)
# Step 5: Create Trainer instance and train
trainer = Trainer(
model=model, args=training_args, train_dataset=small_train_dataset, eval_dataset=small_eval_dataset
)
trainer.train()
结果解释
训练完我们的模型后，让我们对其进行评估。我们将计算混淆矩阵和 ROC 曲线，以了解我们的模型的表现如何。
# Step 6: Evaluation
predictions = trainer.predict(small_eval_dataset)
# Confusion matrix
cm = confusion_matrix(small_eval_dataset['label'], predictions.predictions.argmax(-1))
sns.heatmap(cm, annot=True, fmt='d')
plt.title('Confusion Matrix')
plt.show()
# ROC Curve
fpr, tpr, _ = roc_curve(small_eval_dataset['label'], predictions.predictions[:, 1])
roc_auc = auc(fpr, tpr)
plt.figure(figsize=(1.618 * 5, 5))
plt.plot(fpr, tpr, color='darkorange', lw=2, label='ROC curve (area = %0.2f)' % roc_auc)
plt.plot([0, 1], [0, 1], color='navy', lw=2, linestyle='--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver operating characteristic')
plt.legend(loc="lower right")
plt.show()
混淆矩阵详细说明了我们的预测如何与实际标签相匹配，而 ROC 曲线则向我们展示了各种阈值设置下真阳性率（灵敏度）和假阳性率（1 - 特异性）之间的权衡。
最后，为了查看我们的模型的实际效果，让我们用它来推断示例文本的情绪。
# Step 7: Inference on a new sample
sample_text = "This is a fantastic movie. I really enjoyed it."
sample_inputs = tokenizer(sample_text, padding="max_length", truncation=True, max_length=512, return_tensors="pt")
# Move inputs to device (if GPU available)
sample_inputs.to(training_args.device)
# Make prediction
predictions = model(**sample_inputs)
predicted_class = predictions.logits.argmax(-1).item()
if predicted_class == 1:
print("Positive sentiment")
else:
print("Negative sentiment")
总结
通过浏览 IMDb 电影评论的情感分析示例，我希望您能够清楚地了解如何将 BERT 应用于现实世界的 NLP 问题。我在此处包含的 Python 代码可以进行调整和扩展，以处理不同的任务和数据集，为更复杂和更准确的语言模型铺平道路。
Reference
Source: https://towardsdatascience.com/practical-introduction-to-transformer-models-bert-4715ed0deede
本文由 mdnice 多平台发布