[Python] Transformers loading BERT model from_pretrained() problem solving

Build the development environment

Install Miniconda on the Ubuntu server, and connect to remote development through VSCode or PyCharm or Gateway.

Recommended reading: VSCode runs Python programs through a virtual environment

Install libraries such as PyTorch, Scikit-Learn, Transformers, etc.

Recommended reading: Conda installs GPU support packages for TensorFlow and PyTorch

Note: Do not install Scikit-Learn pip install sklearn, you should pip install scikit-learn.

OSError: Can‘t load config for ‘xxxxxx’. If you were trying

An error was reported:
OSError: Can't load config for 'xxxxxx'. If you were trying

According to this blog , I tried to download bert-base-uncasedthe relevant files manually, but it still failed.

UnicodeDecodeError: ‘utf-8’ codec can’t decode byte 0x80 in position 0: invalid start byte

Encountered an error:
UnicodeDecodeError: 'utf-8' codec can't decode byte 0x80 in position 0: invalid start byte

According to the articles on the Internet, the reason for this error is that the binary file is read in the wrong encoding format and reading method, which cannot be processed in this project.

Can’t load the configuration of ‘xxxxxx’.

Can’t load the configuration of ‘xxxxxx’. If you were trying to load it from ‘https://huggingface.co/models’, make sure you don’t have a local directory with the same name. Otherwise, make sure ‘xxxxxx’ is the correct path to a directory containing a config.json file

Introduce the following script, first bert-base-uncaseddownload the model from Huggingface's warehouse to the local:

from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
model = AutoModel.from_pretrained("bert-base-uncased")
tokenizer.save_pretrained('./bert/')
model.save_pretrained('./bert/')

Loading model from pytorch_pretrained_bert into transformers library

View the Huggingface official Discussion post Loading model from pytorch_pretrained_bert into transformers library , there is such a passage:

Hi. This is probably caused by the transformer verison. You might downgrade your transformer version from 4.4 to 2.8 with pip install transformers==2.8.0

So try downgrading the transformers version to 2.8.0.

First check the transformers version:
pip show transformers

输出信息显示版本为4.26.1：
Name: transformers
Version: 4.26.1
Summary: State-of-the-art Machine Learning for JAX, PyTorch and TensorFlow
Home-page: https://github.com/huggingface/transformers
Author: The Hugging Face team (past and future) with the help of all our contributors (https://github.com/huggingface/transformers/graphs/contributors)
Author-email: [email protected]
License: Apache
Location: xxxxxxxxxxxxxxxxxxxxxxx
Requires: filelock, huggingface-hub, importlib-metadata, numpy, packaging, pyyaml, regex, requests, tokenizers, tqdm
Required-by:

Install the 2.8.0 version of transformers directly:
pip install transformers==2.8.0

Encountered a string of errors, one of which was:
During handling of the above exception, another exception occurred:

Uninstall transformers:
pip uninstall transformers

Then install:
pip install transformers==2.8.0

遇到错误：
ERROR: Could not find a version that satisfies the requirement boto3 (from transformers) (from versions: none)
ERROR: No matching distribution found for boto3

ERROR: No matching distribution found for boto3

In order to solve the above problems, refer to StackOverflow and install the boto3 library.

First check if boto3 is installed:
pip show boto3

Output result:
WARNING: Package(s) not found: boto3

Apparently, it wasn't installed.

Then officially install boto3:
pip install boto3

Then install the 2.8.0 version of the transformers library:
pip install transformers==2.8.0

Missing key(s) in state_dict: “bert.embeddings.position_ids”.

After installing the transformers library of version 2.8.0, an error is reported when running the program:
Missing key(s) in state_dict: “bert.embeddings.position_ids”.

After referring to this blog and modifying it slightly, add the following code:

cudnn.benchmark = True

Still reporting an error:
TypeError: 'BertTokenizer' object is not callable

A related Issue on GitHub was retrieved: TypeError: 'BertTokenizer' object is not callable #53 , and the reply to this Issue pointed out:

Transformers fails “TypeError: ‘BertTokenizer’ object is not callable” if the installed version is <v3.0.0 . In the requirements file, transformers should be “transformers>=3.0.0”

So decided to upgrade the version to 3.0.0:
pip install transformers==3.0.0

成功，部分输出如下：
Installing collected packages: tokenizers, transformers
Attempting uninstall: tokenizers
Found existing installation: tokenizers 0.5.2
Uninstalling tokenizers-0.5.2:
Successfully uninstalled tokenizers-0.5.2
Attempting uninstall: transformers
Found existing installation: transformers 2.8.0
Uninstalling transformers-2.8.0:
Successfully uninstalled transformers-2.8.0
Successfully installed tokenizers-0.8.0rc4 transformers-3.0.0

运行程序，可以得到结果，伴随着输出如下内容：
Some weights of the model checkpoint at ./bert/ were not used when initializing BertModel: [‘embeddings.position_ids’]
- This IS expected if you are initializing BertModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPretraining model).
- This IS NOT expected if you are initializing BertModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).