cuda, cudann, conda installation tutorial
Enter the following command to view the highest CUDA version supported by your GPU.
nvidia-smi
cuda installation (cudatoolkit)
Previously Nvidia CUDA official website:CUDA Toolkit Archive | NVIDIA Developer
CUDA Toolkit 11.8 Downloads | NVIDIA Developer
wget https://developer.download.nvidia.com/compute/cuda/11.8.0/local_installers/cuda_11.8.0_520.61.05_linux.run
sudo sh cuda_11.8.0_520.61.05_linux.runcudnn installation
linux
CUDA Toolkit Archive | NVIDIA Developer
Download it and throw it to the server:
cudnn-linux-x86_64-8.4.1.50_cuda10.2-archive.tar.xz
Note that it is xz not gz, so the decompression is:
tar -xf cudnn-linux-x86_64-8.4.1.50_cuda10.2-archive.tar.xzPreviously, it was OK to throw the cudnn files to the path of cuda. However, in fact, it does not necessarily need to be placed in the path of cuda. Just add it to LD_LIBRARY_PATH.
After decompression, there is the directory cudnn-linux-x86_64-8.4.1.50_cuda10.2-archive, under which there are include and lib
Add the path to LD_LIBRARY_PATH and give the path a name:
Put it in the configuration file:
echo "export CUDNN_DIR=$(pwd)" >> /etc/profile
echo "export LD_LIBRARY_PATH=$CUDNN_DIR/lib:$LD_LIBRARY_PATH" >> /etc/profile
Then refresh the environment variables:
source /etc/profileAfter refreshing, check if it was added correctly:
echo $CUDNN_DIR
/root/cudnn-linux-x86_64-8.4.1.50_cuda10.2-archive
It’s OK to return to this
Take another look at LD_LIBRARY_PATH, is it correct?
echo $LD_LIBRARY_PATHtorch installation
pip install torchaudio torch==2.0.0+cu118 torchvision ninja --force-reinstall --extra-index-url https://download.pytorch.org/whl/cu118
Choose the options on the webpage that suit your computer and system.
- The first line is to select whether the torch is a stable version. The default is the stable version.
- The second line is to choose your own system, including linux, mac, and windows.
- The third line indicates the installation environment: installation in anacoda, pip installation in cmd, installation in libtorch, or source code installation.
- The fourth line is language selection: python and c++/java are provided
- In the fifth line, do you choose to use the cuda gpu version? If so, you need to check your own cuda version (self-Google, Baidu). I choose none here, without gpu, so it is the cpu version of torch.
- The sixth line is to copy the installation command directly, and then select the corresponding environment for installation. Most of them use anaconda or pip.
There is another special note that you must use a 64-bit environment. Whether it is python IDE or Anaconda, you must install 64-bit. Otherwise, all methods you try online will be useless. (The blogger’s tearful experience is that he spent a lot of time and searched the method online, and found that it was a problem with the number of bits. He uninstalled the 32-bit anaconda, installed the 64-bit version and found that it was successful.) As for how to look at the number of bits, if it is installed The system python is used. Enter python directly in the cmd command. The premise is that the system environment variable is added to python. As shown below, I started the 32-bit installation. I tried many methods but failed. Ana has also been 32-bit since the beginning.
bitsandbytes installation
window
Magically modified the bitsandbytes library to support windows.
python -m pip install bitsandbytes --prefer-binary --extra-index-url=https://jllllll.github.io/bitsandbytes-windows-webuiInstall the window version of bitsandbytes
or
pip uninstall bitsandbytes
pip install bitsandbytes-windowsAnother option https://github.com/acpopescu/bitsandbytes/releases/tag/v0.38.0-win0
linux
GitHub - TimDettmers/bitsandbytes: 8-bit CUDA functions for PyTorch
1.pip installation
pip install bitsandbytes
2. Compile locally and choose your own cuda version
git clone https://github.com/timdettmers/bitsandbytes.git
cd bitsandbytes
# CUDA_VERSIONS in {110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 120}
# make argument in {cuda110, cuda11x, cuda12x}
# if you do not know what CUDA you have, try looking at the output of: python -m bitsandbytes
CUDA_VERSION=117 make cuda11x
python setup.py installUsing Int8 inference with the HuggingFace converter
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(
'decapoda-research/llama-7b-hf,
device_map='auto',
load_in_8bit=True,
max_memory=f'{int(torch.cuda.mem_get_info()[0]/1024**3)-2}GB')int4
from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers.generation import GenerationConfig
from transformers import AutoModelForCausalLM, BitsAndBytesConfig
# quantization configuration for NF4 (4 bits)
quantization_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type='nf4',
bnb_4bit_compute_dtype=torch.bfloat16
)
。。。。。。
model = AutoModelForCausalLM.from_pretrained(
"Qwen/Qwen-7B-Chat",
quantization_config=quantization_config,
device_map="auto",
trust_remote_code=True
).eval()int4 effect, 12G graphics card can drive
from transformers import AutoModelForCausalLM, BitsAndBytesConfig, AutoTokenizer
# quantization configuration for NF4 (4 bits)
quantization_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type='nf4',
bnb_4bit_compute_dtype=torch.bfloat16
)
# quantization configuration for Int8 (8 bits)
# quantization_config = BitsAndBytesConfig(load_in_8bit=True)
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen-7B-Chat", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained(
"Qwen/Qwen-7B-Chat",
device_map="cuda:0",
quantization_config=quantization_config,
trust_remote_code=True,
).eval()
response, history = model.chat(tokenizer, "你好", history=None)
print(response)
response, history = model.chat(tokenizer, "给我讲一个年轻人奋斗创业最终取得成功的故事。", history=history)
print(response)
你好！很高兴为你提供帮助。
这是一个关于一个年轻人奋斗创业最终取得成功的故事。
这个年轻人叫做李明，他来自一个普通的家庭。他从小就对商业感兴趣，梦想着有一天能够成为一名成功的企业家。然而，他的家庭条件并不富裕，他需要通过自己的努力来实现自己的梦想。
李明在大学期间就开始创业，他成立了一家电子商务公司。一开始，公司的发展并不顺利，面临着很多困难和挑战。但是，李明并没有放弃，他坚定地相信自己的梦想，并且不断地努力工作。
他不断地学习和探索，寻找新的商机和创意。他也不断地与人交流和合作，寻求帮助和支持。最终，他的公司开始逐渐发展壮大，获得了越来越多的客户和投资者。
经过几年的努力，李明的公司已经成为了一家成功的电子商务企业。他的创业经历也激励了很多人，让他们相信只要努力奋斗，就能够实现自己的梦想。
这个故事告诉我们，创业并不是一件容易的事情，需要付出大量的努力和时间。但是，只要我们相信自己，不断努力工作，就一定能够取得成功。pyopenjtalk installation
ModuleNotFoundError: No module named 'pyopenjtalk'
pip install pyopenjtalk --no-build-isolation
Other quantification methods
model = model.quantize(4).cuda()git-lfs installation
1. Introduction to git-lfs
git-lfs is a GIT extension, usually used for version control and management of large files. The full name of LFS is Large File Storage, which allows Git Users store and manage large binary files not directly in a Git repository, but instead in an external object store. By using git-lfs, users can avoid storing large binary files in Git repositories, thereby improving the performance and maintainability of Git repositories.
To use git-lfs, you need to install the extension first, and then configure the corresponding LFS tracking rules in the Git repository to specify which files should be managed using LFS. Once configured, Git users can manage large binary files just like other code files. When performing Git operations, git-lfs automatically handles large binary files tracked by LFS, ensuring that these files can be uploaded, downloaded, and versioned correctly.
2. Installation
apt-get update # 更新安装包列表
apt-get install git # 安装git
apt-get install git-lfs # 安装git-lfs
git lfs install # 初始化
References
GitHub - Keith-Hon/bitsandbytes-windows: 8-bit CUDA functions for PyTorch in Windows 10