遇到的问题:
THCudaCheck FAIL file=/pytorch/aten/src/THC/THCGeneral.cpp line=663 error=11 : invalid argument
Traceback (most recent call last):
File "../../tools/demo.py", line 53, in <module>
state = siamese_init(im, target_pos, target_sz, siammask, cfg['hp'], device=device) # init tracker
File "/home/pwh/SIAMMASK/SiamMask/tools/test.py", line 155, in siamese_init
net.template(z.to(device))
File "/home/pwh/SIAMMASK/SiamMask/experiments/siammask_sharp/custom.py", line 174, in template
self.zf = self.features(template)
File "/home/pwh/.conda/envs/siammask/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
result = self.forward(*input, **kwargs)
File "/home/pwh/SIAMMASK/SiamMask/experiments/siammask_sharp/custom.py", line 59, in forward
output = self.features(x)
File "/home/pwh/.conda/envs/siammask/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
result = self.forward(*input, **kwargs)
File "/home/pwh/SIAMMASK/SiamMask/experiments/siammask_sharp/resnet.py", line 218, in forward
x = self.conv1(x)
File "/home/pwh/.conda/envs/siammask/lib/python3.6/site-packages/torch/nn/modules/module.py", line 477, in __call__
result = self.forward(*input, **kwargs)
File "/home/pwh/.conda/envs/siammask/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 301, in forward
self.padding, self.dilation, self.groups)
RuntimeError: cuda runtime error (11) : invalid argument at /pytorch/aten/src/THC/THCGeneral.cpp:663
解决方法:
这个问题的主要原因是由于torch版本不对应,原先是通过pip install -r requirment.txt
下载,torch版本为0.4.0
。所以可以通过pip install -U torch
进行更新(但是最好指定自己系统cuda版本所对应的版本),这里需要注意的是,2021年torch版本已经更新到1.8,
如果直接输入上述命令,torch会更新到1.8版本,因此我又遇到了以下错误:
Traceback (most recent call last):
File "../../tools/demo.py", line 53, in <module>
state = siamese_init(im, target_pos, target_sz, siammask, cfg['hp'], device=device) # init tracker
File "/home/pwh/SIAMMASK/SiamMask/tools/test.py", line 155, in siamese_init
net.template(z.to(device))
File "/home/pwh/SIAMMASK/SiamMask/experiments/siammask_sharp/custom.py", line 174, in template
self.zf = self.features(template)
File "/home/pwh/.conda/envs/siammask/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/pwh/SIAMMASK/SiamMask/experiments/siammask_sharp/custom.py", line 59, in forward
output = self.features(x)
File "/home/pwh/.conda/envs/siammask/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/pwh/SIAMMASK/SiamMask/experiments/siammask_sharp/resnet.py", line 218, in forward
x = self.conv1(x)
File "/home/pwh/.conda/envs/siammask/lib/python3.6/site-packages/torch/nn/modules/module.py", line 889, in _call_impl
result = self.forward(*input, **kwargs)
File "/home/pwh/.conda/envs/siammask/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 399, in forward
return self._conv_forward(input, self.weight, self.bias)
File "/home/pwh/.conda/envs/siammask/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 396, in _conv_forward
self.padding, self.dilation, self.groups)
RuntimeError: cuDNN error: CUDNN_STATUS_NOT_INITIALIZED
这个就是由于cuda、cudnn、pytorch
的版本不对应导致的,所以我卸载了1.8版本的torch,选择了1.7版本的torch,命令如下:
pip install -i https://pypi.douban.com/simple/ torch==1.7
,更新完问题即成功解决。