[Use Anaconda to configure tensorflow+nlp environment]

foreword

Recently, I want to configure an NLP environment based on tensorflow. I mainly configure the two commonly used libraries of transformers and tf-models-official. The selection of some library versions is based on my previous experience in configuring the environment and referring to a large number of online resources. The hard-to-determined version at the end. Record the configuration process:

configuration process

1. conda creates a virtual environment

conda create -n tf23 python=3.8

Note: Based on the previous experience of configuring the environment, after repeated verifications, it is found that tensorflow 2.3 version has fewer problems, so the created environment is based on tensorflow 2.3, and the selected python version is 3.8 (the final downloaded python version is 3.8.13).

2. Download Tensorflow 2.3.0

pip install tensorflow==2.3.0 -i https://pypi.tuna.tsinghua.edu.cn/simple/

Note:

• It is recommended to add the source after the pip install command. This configuration environment uses the Tsinghua source. In order to simplify writing, the configuration of the pip download source is omitted when downloading the relevant library. Remember to add it yourself: https://pypi.tuna.tsinghua
  . edu.cn/simple/
  After adding it, experience the feeling of flying.
• Before configuring this environment, I have downloaded many libraries with the conda install command. During the process of creating the environment, I found that when downloading some libraries, they will always be pulled from Anaconda's local pkgs (cache) folder by default . Related libraries to be downloaded, which sometimes lead to incorrect versions of some associated libraries when downloading the tensorflow library, such as tensorflow-estimator, and I have never known how to solve this problem, so when configuring this environment, choose to use pip The install command is the main one.
• After executing the above command, it is found that tf23 (the virtual environment being configured) already contains numpy=1.18.5 .

3. Download NLP related libraries, here I mainly focus on transformers and tf-models-official

pip install tf-models-official==2.3.0
pip install transformers==4.16.2
pip install tensorflow-hub

Note:
(1) After testing, the selection of the tf-models-official version should generally be consistent with the tensorflow version in the virtual environment, otherwise an error may be reported. (2) The general official
website will describe each version of transformers and the corresponding python, tensorflow, pytorch version requirements, you can choose the corresponding version by yourself, address: transformers version selection

• Introduction to transformers: Transformers is developed by Hugging Face, a company focusing on NLP, which has a high number of stars on github.

• tf-models-official official introduction: tf-models-official is a model that uses TensorFlow's high-level API, providing TensorFlow users with many different state-of-the-art (SOTA) models and modeling solutions.

• TensorFlow Hub official introduction: TensorFlow Hub is a comprehensive code library containing a variety of pre-trained models that can be deployed to any device with minor adjustments. With the tensorflow_hub library, you can download the latest trained models and use them with only a small amount of coding.

4. other downloads

pip install matplotlib==3.3.4
pip install pandas==1.1.5
pip install scikit-learn==0.24.2	# sklearn版本貌似可以有更多的选择，大家可以试一下
# 其他所需......

Note:
The selection of the above library versions is the result of configuring several environments and experiencing a lot of version incompatibility issues. Please refer to it.

question

1. If you have this problem:
   the versions of numpy in the conda list and the pip list are inconsistent, numpy=1.23.1 under the pip list, and numpy=1.18.5 under the conda list, solve it by the following method: delete the tf23 virtual environment
   . The numpy-1.18.5.dist-info folder, the path is: the directory where all your virtual environments are located\tf23\Lib\site-packages\numpy-1.18.5.dist-info, mine is E:\zp\ appdata\annoconda\envs\tf23\Lib\site-packages
   reference: pip list and conda list are inconsistent and conflict

The reason for this problem may be that you downloaded the library in the wrong order. The download principle is:

• Download relatively large libraries first, such as tensorflow and tf-models-official, etc. Generally, if you download a relatively large library, you will download some compatible libraries accordingly, so that you can save a few hairs hahaha.

2. If the above related libraries are configured well, the following problems are found when importing related libraries in jupyter notebook:
   TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See Installation — Jupyter Widgets 7.6.5 documentation from .autonotebook import tqdm as notebook_tqdm

• It can be solved like this:

pip install ipywidgets widgetsnbextension pandas-profiling
jupyter nbextension enable --py widgetsnbextension

Reference: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. Error handling method

Next, let's have fun (hair loss)~