Summary of the column "Python Basics Tutorial"

  Hello everyone, I am Miaomiao who loves programming. Double 985 master's degree, now working as a full-stack engineer, keen to apply data thinking to work and life. Engaged in machine learning and related front-end and back-end development work. He has won many top rankings in competitions such as Alibaba Cloud, HKUST Xunfei, and CCF. Now he is a CSDN blog expert and a high-quality creator in the field of artificial intelligence. I like to summarize and summarize the knowledge I have learned through blog creation, which not only forms a deep and unique understanding, but also helps novices get started quickly.

The "Python Basics Course" column carefully created by me has more than 900   subscriptions and has helped many students solve bugs. Recently, a classmate who studies very seriously gave me feedback that the column blog helped him solve various problems such as large models, crawlers, Python basics, etc., but sometimes it is not convenient to find it. I hope that the column articles can be organized into categories. This can improve retrieval efficiency.

  Excellent suggestions must be accepted without hesitation. Although objectively speaking, organizing is a lot of work, but when you think about the painful struggle that silkworms must go through before they can emerge from the cocoon and become a butterfly. So as soon as I finished my work in the past two days, I continued to organize it non-stop. Existing categories include large models, machine learning, crawlers, data processing and visualization, anaconda & pip & jupyter notebook & ipython & pycharm, databases and middleware, commonly used Python libraries, Python basics, github, etc. You can click on the right side of the blog button to view the catalog, as shown below:

  This column updates at least 5 related articles every week, and the goal is at least 1,000+ articles, so it will continue to be expanded in the future, hoping that it will eventually be helpful to students who subscribe.

1. Large model

1.1 transformers & huggingface

• huggingface_hub.utils._validators.HFValidationError Repo id must be in the form repo_name or name解决方案
• TypeError: transformers.tokenization_utils_base.PreTrainedTokenizerBase._from_pretrained() got multiple values for keyword argument 'use_auth_token’解决方案
• ModuleNotFoundError: No module named 'transformers.modeling_bart'Solution
• Actual code for transformers to automatically select an idle single GPU when loading a model
• Transformers automatically select the actual combat code of multiple idle GPUs when loading the model
• How to modify the storage location of huggingface model
• Huggingface datasets solution for loading files offline
• Solution to KeyError: 'output' when huggingface datasets map
• RuntimeError: expected scalar type Half but found Float solution
• ValueError: paged_adamw_32bit is not a valid OptimizerNames, please select one of ['adamw_hf', 'adamw_torch', 'adamw_torch_fused', 'adamw_torch_xla', 'adamw_apex_fused', 'adafactor', 'adamw_bnb_8bit', 'adamw_anyprecision', 's gd', 'adagrad']Introduction
• RuntimeError:Expected all tensors to be on the same device, but found at least two devices解决方案
• tokenizers＞=0.13.3 is required for a normal functioning of this module, but found tokenizers解决方案
• ValueError: Connection error, and we cannot find the requested files in the cached path.解决方案

1.2 pytorch

• ModuleNotFoundError: No module named 'torch._six' solution

• Pytorch RuntimeError: No CUDA GPUs are available solution

• OSError Unable to load weights from pytorch checkpoint file for pytorch_model.bin solution

• Import Error: from torchtext.data import to_map_style_dataset solution

• A simple solution for Pytorch to load torchvision and download the pre-trained model locally

• RuntimeError: Detected that PyTorch and torchvision were compiled with different CUDA versions解决方案

• RuntimeError Cuda extensions are being compiled…Pytorch binaries were compiled with Cuda 11.7解决方案

• ImportError: libcupti.so.11.7: cannot open shared object file: No such file or directory解决方案

• RuntimeError: Boolean value of Tensor with more than one value is ambiguous解决方案

1.3 ChatGLM loading, training, prediction

• RuntimeError:Internal: src/sentencepiece_processor.cc(1101)model_proto-＞ParseFromArray(serialize.size())]解决方案
• ModuleNotFoundError: No module named 'transformers_modules.chatglm-6b_v1' solution
• ModuleNotFoundError: No module named 'transformers_modules.chatglm2-6b' solution
• OSError We couldn't connect to https://huggingface.co to load file, couldn't find chatglm2-6b solution
• Solution for downloading ChatGLM model files at high speed
• Solution for downloading ChatGLM2 model files at high speed
• Solution for downloading VisualGLM model files at high speed
• Solution to KeyError: 'context' when ChatGLM lora fine-tuning
• RuntimeError: Internal: [MASK] is already defined solution
• ChatGLM ptuning predict(prediction) is the solution to the null value
• AttributeError: 'ChatGLMConfig' object has no attribute 'quantization_bit' solution
• TypeError: quantize() got an unexpected keyword argument 'empty_init' solution
• AttributeError: ‘ChatGLMModel’ object has no attribute 'prefix_encoder’解决方案

1.4 bitsandbytes

• CUDA detection failed solution when using bitsandbytes
• ERROR : Your GPU does not support Int8 Matmul ! Solution
• cuda_setup.add_log_entry("WARNING: Compute capability < 7.5 detected solution
• cuda_setup.add_log_entry("WARNING: Compute capability ＜ 7.5 detected! Only slow 8-bit matmul is supported for your GPU 解决方案
• PackageNotFoundError: No package metadata was found for bits and bytes solution
• Bitsandbytes found a lower version of CUDA SETUP: Detected CUDA version 100 under the condition that a higher version of CUDA has been installed

1.5 langchain

• TypeError MyFAISS.similarity_search_with_score_by_vector got unexpected keyword argument filter解决方案
• ImportError: cannot import name 'SQLDatabaseChain' from 'langchain' solution
• ValueError: Could not import faiss python package. Please install it with pip install faiss solution
• OperationalError: (sqlite3.OperationalError) unable to open database file solution

1.6 fastchat

• Fastchat appears TypeError: unsupported operand type(s) for -: 'NoneType' and 'int' solution
• Fastchat UnicodeDecodeError utf-8 codec can‘t decode bytes in position invalid continuation byte解决方案

1.7 Installation of GPU driver, CUDA, CUDNN, NCCL, etc.

• A100 GPU server installation GPU driver tutorial
• A100 GPU server installation CUDA tutorial
• A100 GPU server installation CUDNN tutorial
• Practical tutorial on nvidia nccl installation and testing
• An NVIDIA kernel module ‘nvidia-uvm’ appears to already be loaded in your kernel解决方案
• /usr/src/kernels/el7.linux/Makefile:1006: Cannot generate ORC metadata for CONFIG_UNWINDER_ORC=y solution
• NVIDIA-SMI has failed because it couldn’t communicate with the NVIDIA driver.Make sure that解决方案
• Practical tutorial on uninstalling GPU driver and CUDA in Linux

1.8 Tongyi Qianwen

• cuda_setup/main.py: UserWarning: /home/anaconda3/envs/qwen-7b did not contain ['libcudart.so'…Solution
• ValueError: Unrecognized configuration class <class transformers_modules.Qwen-7B-Chat…Solution
• Solution to RuntimeError current installed version g++ (4.8.5) is less than mininum version when installing flash-attn

1.9 llama

• configuration_auto.py in getitem raise KeyError(key) KeyError: 'llama' solution
• AttributeError: module transformers has no attribute LLaMATokenizer solution

1.10 paddle

• ModuleNotFoundError: No module named 'paddle'Solution
• ImportError: libcudart.so.10.2: cannot open shared object file: No such file or directory解决方案

1.11 deepspeed

• error "You‘re trying to build PyTorch with a too old version of GCC. We need GCC 5 or later."解决方案
• RuntimeError: launcher 'pdsh' not installed solution

1.12 peft

• ImportError: cannot import name 'prepare_model_for_kbit_training' from 'peft' solution
• ValueError: Please specify `target_modules` in `peft_config` solution

1.13 cpm-bee & bmtrain

• pip install install CPM-Bee appears ModuleNotFoundError: No module named 'torch' solution
• RuntimeError: CUDA error: CUBLAS_STATUS_INVALID_VALUE when calling `cublasGemmEx Solution
• ImportError: /home/anaconda3/lib/libstdc++.so.6: version `GLIBCXX_3.4.29' not found solution
• subprocess.CalledProcessError: Command '['which', 'x86_64-conda_cos6-linux-gnu-c++']' solution
• RuntimeError: The current installed version of g++ (4.8.5) is less than the mininum version CUDA解决方案

1.14 apex

• ImportErrorImportError: cannot import name ‘UnencryptedCookieSessionFactoryConfig’ from pyramid解决方案

1.15 tensorflow

• TensorFlow 2.10 Unable to register cuBLAS factory Attempting to register factory for plugin cuBLAS方案

1.16 Others

• scripts/spider_qlora_finetune.sh: line 7: --source_max_len: command not found solution

2. Machine Learning

2.1 clear

• ModuleNotFoundError: No module named 'sklearn' solution
• ModuleNotFoundError: No module named 'sklearn.__check_build._check_build' solution
• ImportError: cannot import name '_ccallback_c' solution
• sklearn logistic regressionIncrease the number of iterations (max_iter) or scale the data as shown in solution
• Solution for TypeError: classification_report() takes 2 positional arguments but 3 were given
• Error loading sklearn California housing price data set housing = fetch_california_housing() HTTPError: HTTP Error 403: Forbidden solution
• Error loading sklearn face dataset fetch_olivetti_faces() HTTPError: HTTP Error 403: Forbidden solution
• Error loading sklearn news data set fetch_20newsgroups() HTTPError: HTTP Error 403: Forbidden solution
• Error loading sklearn covtype data set fetch_covtype() HTTPError: HTTP Error 403: Forbidden solution
• Detailed solution for reading the watermelon dataset
• Analysis of f1_score parameter in sklearn

2.2 ltp

• AttributeError: 'LTP' object has no attribute 'seg'Solution
• AttributeError: 'LTP' object has no attribute 'sent_split'Solution
• AttributeError: 'LTP' object has no attribute 'init_dict' solution

2.3 nltk

• [nltk_data] Error loading punkt: ＜urlopen error [Errno 111] Connection refused＞Solution
• Resource punkt not found. Please use the NLTK Downloader to obtain the resource 解决方案
• Solution for offline downloading of NLTK dependency packages ([WinError 10061] Unable to connect because the target computer actively refused)

2.4 spacy

• Spacy’s solution for installing the old version of en_core_web_sm
• OpenSSL SSl SysCallError (10054) solution when spacy.load('en') or spacy.load('de') occurs
• TypeError: Plain typing.NoReturn is not valid as type argument solution

2.5 joblib & flask

• Solution to TypeError: an integer is required (got type bytes)
• Python uses flask to encapsulate the actual code of restful API

2.6 Model reading and saving

• Detailed explanation of Python file reading and model saving
• Expansion of reading and saving model & data files

2.7 English word segmentation

• Dictionary file not found symspellpy.py symspellpy.py in word_segmentation(self, StopIteration solution

3. Crawler

3.1 selenium

• TypeError:__init__() got an unexpected keyword argunent 'executable_path' solution
• selenium.common.SessionNotCreatedException Message session not created.ChromeDriver support version解决方案
• When Python installs selenium, requests 2.19.1 has requirement urllib3＜1.24,＞=1.21.1, but urllib3 incompatible solution
• selenium 'WebDriver' object has no attribute 'find_element_by_link_text' solution
• selenium 'WebDriver' object has no attribute 'find_element_by_xpath' solution

3.2 requests

• ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:833) solution appears in requests
• Crawler encoding UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb1 in position 78: invalid start byte solution
• Python crawler requests return value is empty solution
• Practical code for python to save files requested by requests
• Solution to download multi-line json from the Internet and convert it into a dictionary
• Solution to download online compressed package (containing multi-line json) and convert it into a dictionary
• ImportError: cannot import name Template solution

3.3 Others

• Practical tutorial of python library for extracting web page title and text content

4. Data processing and visualization

4.1 pandas.DataFrame

• AttributeError: 'DataFrame' object has no attribute 'iteritems' solution
• ValueError: If using all scalar values, you must pass an index解决方案
• Python convert string to dataframe
• Solution to add a column to DataFrame and assign each value as a list (ValueError: Length of values ​​does not match length of index)
• pandas obtains the actual code for the row corresponding to the first or last day of every month (year) in the DataFrame
• Practical code to filter out all rows in a DataFrame that contain strings in a certain column
• Practical code to change the order of DataFrame columns

4.2 pandas

• Solution to ValueError:Trailing data in pd.read_json

• xlrd.biffh.XLRDError: Excel xlsx file; not supported solution appears in pd.read_excel

• PicklingError: Can't pickle ＜function ＜lambda＞＞: attribute lookup ＜lambda＞ on __main__ failed solution

• AttributeError: 'Timestamp' object has no attribute 'dt' solution

• UserWarning: Workbook contains no default style, apply openpyxl’s default warn("Workbook contains no default style, apply openpyxl’s default"解决方案

• _pickle.UnpicklingError: invalid load key, '\x00' solution when importing xlwings

• Solution to pandas to_excel generating null values

• df.rank & pd.pivot_table & pd.read_excel & df add rows & adjust df row and column order (reindex and then understand)

• Detailed explanation of pandas basics

4.3 numpy

• AttributeErrormodule 'numpy' has no attribute 'bool' solution
• ImportError: DLL load failed: The specified module cannot be found in import numpy.
• The difference between apply, applymap and map in numpy and pandas
• Split&concatenate&unstack&list nesting issues (loops, conversions) in np and pd
• Detailed explanation of numpy basics

4.4 matplotlib & seaborn

• Solution to RuntimeError: Invalid DISPLAY variable in matplotlib in Linux
• Seaborn load_dataset error solution

4.5 excel & csv

• Detailed explanation of python operation EXCEL (xlsxwriter/ExcelWriter)
• ValueError: Append mode is not supported with xlsxwriter solution
• pd.read_excel appears ValueError: Unknown engine: openpyxl solution
• PermissionError: [Errno 13] Permission denied: '.xlsx' solution
• Practical code for setting column names when pandas reads csv files
• Python practical code to convert dictionary to csv
• Solution to write extra commas in csv file
• _pickle.UnpicklingError: invalid load key, '\x00' solution when importing xlwings

4.6 json

• TypeError: Object of type 'datetime' is not JSON serializable solution occurs when converting dictionary to json
• JSONDecodeError: Expecting property name enclosed in double quotes: line 2 column 3 (char 4)解决方案
• ImportError: cannot import name 'json_util' solution
• Solution to TypeError: __init__() got an unexpected keyword argument 'encoding' when json.loads

4.7 yaml

• ModuleNotFoundError: No module named 'yaml' solution
• Pip install yaml appears ERROR: Could not find a version that satisfies the requirement yaml solution
• TypeError: load() missing 1 required positional argument: 'Loader' solution

4.8 pickle

• Pickle.load appears UnicodeDecodeError: 'ascii' codec can't decode byte 0x8b in position 6: ordinal not in range(128) solution

4.9 zip

• Solution to TypeError: pwd: expected bytes, got str when Python decompresses the zip file

5. anaconda & pip & jupyter notebook & ipython & pycharm

5.1 anaconda

• Win7 installs the new version of anaconda and appears Failed to extract packages solution
• Solution to missing api-mis-win-core-path-l1-1-0.dll when installing miniconda

5.2 cond

• Conda upgrade appears Solving environment: failed CondaValueError Malformed version string invalid chars solution

• CondaValueError: Malformed version string invalid character(s) solution appears in conda install

• The environment is inconsistent, please check the package plan carefully解决方案

• ERROR conda.core.link:_execute(502): error occurred while installing conda-forge::setuptools solution

• The solution to upgrade gcc in the conda environment

• SSLError(MaxRetryError('HTTPSConnectionPool(host='mirrors.tuna.tsinghua.edu.cn', port=443) solution

• Detailed explanation of creating a python virtual environment

• Install ipython in the conda virtual environment

• Solution to the inconsistent versions of ipython and conda python

• CommandNotFoundError : No command 'conda rename' solution

5.3 pip

• ERROR: Could not install packages due to an OSError: [WinError 5] Access Denied Solution
• pip._vendor.urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host=files.pythonhosted.org solution
• pip install timeout error solution
• pip installation ERROR: Could not install packages due to an OSError: [Errno 28] No space left on device solution
• Detailed explanation of offline installation of Python library
• The pip uninstall library corresponds to the solution of all dependent libraries
• pip install weditor error in setup command: Error parsing weditor\setup.cfg UnicodeDecodeError: solution
• Cannot uninstall … It is a distutils installed project and thus we cannot accurately determine 解决方案
• WARNING:Retrying Failed to establish a new connection: [Errno 101] Network is unreachable’)解决方案
• WARNING:Retrying(Retry(total=4,connect=None,read=None,redirect=None,status =None))connect broken解决方案
• Solution to modify pip cache address in Linux environment
• Solution to WARNING: Ignoring invalid distribution when pip install appears
• Solution to the problem of socket.timeout: The read operation timed out in pip install

5.4 jupyter notebook

• Solution to specify the pip installation library in the kernel in jupyter notebook
• Solution to install conda environment into jupyter notebook kernel
• jupyter notebook installation and removal kernel solution
• Solution to view python version in jupyter notebook
• Practical code for running html code in jupyter notebook
• jupyter '_xsrf' argument missing from post solution
• jupyter notebook crashes, command line execution error: ImportError: cannot import name 'soft_unicode' from 'markupsafe' solution
• ERR_SSL_VERSION_OR_CIPHER_MISMATCH solution appears in jupyter notebook
• Unable to load extension: pydevd_plugins.extensions.types.pydevd_plugin_pandas_types 解决方案
• The solution that jupyter notebook cannot be opened after entering the password
• Solution to AttributeError: module 'tornado.web' has no attribute 'asynchronous' when starting jupyter
• traitlets_version.py AssertionError ".".join(map(str, version_info)).replace(".b", "b") Solution
• attr problem in python (label endoer & jupyter notebook)

5.5 python

• Install ipython in the conda virtual environment
• Solution to the inconsistent versions of ipython and conda python
• The history saving thread hit an unexpected error OperationalError no such table history solution appears in ipython
• Solution to disabling IPython from generating history
• Solution to delete ipython history file
• ImportError: cannot import name ‘create_prompt_application’ from 'prompt_toolkit.shortcuts’解决方案

5.6 pycharm

• Practical solution for connecting to the server with pycharm/idea

6. Database & Middleware

6.1 mysql

• pip install mysql error: subprocess - exited-with-error solution
• Python practical code for converting CSV to MySQL
• OperationalError: foreign key mismatch - “RESTAURANT“ referencing “GEOGRAPHIC“ solution
• IntegrityError: FOREIGN KEY constraint failed solution
• ProgrammingError: 1064 (42000): You have an error in your SQL syntax;check the manual…解决方案
• Python practical code for converting CSV to MySQL

6.2 access(mdb)

• Practical code for reading and writing access database in Python
• Practical code for reading and writing mdb files in Python

6.3 clickhouse

• NoSuchModuleError: Can't load plugin: sqlalchemy.dialects:clickhouse solution

6.4 hdfs

• Practical solution for reading and writing hdfs files in python
• ImportError: Can not find the shared library: libhdfs3.so solution

6.5 kafka

• Explain kafka from a beginner's perspective and use java and python_API in practice

6.6 Others

• Pitfalls encountered by elasticsearch and kibana and their solutions

7. Commonly used Python libraries

• ERROR: Could not find a version that satisfies the requirement PIL (from versions: none)解决方案
• ModuleNotFoundError: No module named 'setuptools.command.build' solution
• AttributeError: 'HTMLParser' object has no attribute 'unescape' solution
• pynlpir update license Error: unable to fetch newest license solution
• ModuleNotFoundError: No module named 'pyLDAvis.gensim' solution
• pdfminer __init__() got an unexpected keyword argument 'codec' solution
• Solution for installing mpi4py in python
• twisted 18.7.0 requires PyHamcrest＞=1.9.0 solution
• AttributeError: Can't get attribute 'f' on <module '__main__'>Solution
• Python practical code to read the Windows registry
• Practical code for switching input methods in Python

8. Python basics

8.1 Encoding

• Solution to convert \uXXXX string to visible string in Python3
• SyntaxError: Non-UTF-8 code starting with '\xb3' in file solution
• UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc1 in position 41694: invalid start byte solution
• Decoding and encoding in python

8.2 built-in functions

• ImportError: cannot import name 'MutableMapping' from 'collections' solution
• Install python datetime&time&timeit
• Detailed explanation of python os module (os.walk&os.listdir)
• AttributeError: module ‘xxx’ has no attribute
• Detailed explanation of if __name__ == '__main__' and from __future__ import print_function

8.3 Basics

• Learn the basics of python in one article
• Python basic knowledge supplement
• Problem with quotes in python strings
• The solution to the error after the Python string split('\n')
• Detailed explanation of parameter parsing in python argparse&sys.argv
• Detailed explanation of python's global variables and local variables (global&nonlocal&locals)
• Comparison of java and python classes
• In-depth understanding of python decorators
• Detailed explanation of python exception handling try, except
• Explain python regular expressions in detail
• centos solution to decompile pyc files
• IndentationError: expected an indented block solution
• View solutions for Python library dependencies
• Python practical code to determine whether the specified program is running

8.4 Calculation

• Practical code for Python score calculation
• Python solution to display numbers using scientific notation
• Solve the Python practical code to solve the number of days in all months of the input year

8.5 windows installation error: Microsoft Visual C++ 14.0 solution

• talib installation error: Microsoft Visual C++ 14.0 or greater is required. Get it with Microsoft C++ Build solution
• pyflux installation error: Microsoft Visual C++ 14.0 or greater is required. Get it with Microsoft C++ Build solution
• ecos installation error: Microsoft Visual C++ 14.0 or greater is required. Get it with Microsoft C++ Build solution
• gensim installation error: Microsoft Visual C++ 14.0 or greater is required. Get it with Microsoft C++ Build solution
• wordcloud installation error:Microsoft Visual C++ 14.0 or greater is required.Get it with Microsoft C++ Build solution

9. Security and Network

• cpe (common platform enumeration) naming convention and python CPE library practice
• AttributeError: module 'netzob.Import.PCAPImporter.ImpactPacket' has no attribute 'ARP' solution
• Practical code for sending emails in python
• Python practical code to verify email
• Practical code to obtain website response time using Python code
• you-get [error] oops, something went wrong. Solution

10. github

• fatal: unable to access Failed to connect to github.com port 443: Timed out解决方案
• git clone fatal: unable to access Failed to connect to github.com port 443: Timed out solution
• git clone fatal: unable to access OpenSSL SSL_read: Connection was reset, errno 1054 solution
• error: external filter git-lfs smudge – %f failed solution