Apex is a very useful library that can change the model from float32 to float16 for training and testing, which is very helpful for training large models on our own PC and saving video memory.
Brief description: Refer to the installation method of apex installation/usage records , and encounter problems after running the code. See the end for the installation method suitable for the moment (2021.5.29).
Question 1
ModuleNotFoundError: No module named 'fused_adam_cuda'
solve
The old version of apex is older, refer to https://github.com/NVIDIA/apex/issues/633 There is an answer in this question:
you can try FusedAdam in apex.optimizers instead of in apex.optimizers.contrib
Mainly due to the version change of apex, the package name introduced into FusedAdam has changed.
Refer to apex's official documentation on FusedAdam , put
from apex.contrib.optimizers import FusedAdam
changed to
from apex.optimizers import FusedAdam
Also, if used FP16_Optimizer, the imported package name has changed.
Refer to apex's official documentation on FP16_Optimizer , put
from apex.contrib.optimizers import FP16_Optimizer
changed to
from apex.fp16_utils import FP16_Optimizer
Question 2
ModuleNotFoundError: No module named 'fused_layer_norm_cuda'
solve
See ModuleNotFoundError: No module named 'fused_layer_norm_cuda' . There was a problem with the previous installation of apex. After uninstalling, reinstall the apex folder obtained after entering git.
pip uninstall apex
python setup.py install --cpp_ext --cuda_ext
Install
Refer to ModuleNotFoundError: No module named 'fused_layer_norm_cuda' , the installation method suitable for the moment (2021.5.29) is:
git clone https://github.com/NVIDIA/apex
cd apex
python setup.py install --cpp_ext --cuda_ext