这两天给服务器装了Anaconda，重装了cuda，都是为了GPU加速的问题
结果。。。发现并没有解决，每次跑代码都会卡住，然后出一大段报错
ssh://[email protected]:22/home/zhanglei/conda/envs/tensorflow-gpu/bin/python3.6 -u /mnt/ba3b04da-ce1b-4c21-ad1b-3aff7d337cdf/wangxing/WISDM/WISDM-master/WISDN_CSDN_change.py
WARNING:tensorflow:From /mnt/ba3b04da-ce1b-4c21-ad1b-3aff7d337cdf/wangxing/WISDM/WISDM-master/WISDN_CSDN_change.py:65: The name tf.placeholder is deprecated. Please use tf.compat.v1.placeholder instead.
WARNING:tensorflow:From /mnt/ba3b04da-ce1b-4c21-ad1b-3aff7d337cdf/wangxing/WISDM/WISDM-master/WISDN_CSDN_change.py:41: The name tf.truncated_normal is deprecated. Please use tf.random.truncated_normal instead.
WARNING:tensorflow:From /mnt/ba3b04da-ce1b-4c21-ad1b-3aff7d337cdf/wangxing/WISDM/WISDM-master/WISDN_CSDN_change.py:61: The name tf.nn.max_pool is deprecated. Please use tf.nn.max_pool2d instead.
WARNING:tensorflow:From /mnt/ba3b04da-ce1b-4c21-ad1b-3aff7d337cdf/wangxing/WISDM/WISDM-master/WISDN_CSDN_change.py:95: The name tf.log is deprecated. Please use tf.math.log instead.
WARNING:tensorflow:From /mnt/ba3b04da-ce1b-4c21-ad1b-3aff7d337cdf/wangxing/WISDM/WISDM-master/WISDN_CSDN_change.py:96: The name tf.train.GradientDescentOptimizer is deprecated. Please use tf.compat.v1.train.GradientDescentOptimizer instead.
WARNING:tensorflow:From /mnt/ba3b04da-ce1b-4c21-ad1b-3aff7d337cdf/wangxing/WISDM/WISDM-master/WISDN_CSDN_change.py:103: The name tf.Session is deprecated. Please use tf.compat.v1.Session instead.
2020-12-21 21:19:02.928666: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-12-21 21:19:02.969208: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: GeForce RTX 3090 major: 8 minor: 6 memoryClockRate(GHz): 1.71
pciBusID: 0000:03:00.0
2020-12-21 21:19:02.969407: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-12-21 21:19:02.970403: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-12-21 21:19:02.971277: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-12-21 21:19:02.971500: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-12-21 21:19:02.972680: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-12-21 21:19:02.973553: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-12-21 21:19:02.976372: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-12-21 21:19:02.978442: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2020-12-21 21:19:02.978726: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-12-21 21:19:02.983634: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 3597800000 Hz
2020-12-21 21:19:02.983988: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55e02caddc70 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-12-21 21:19:02.984003: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
2020-12-21 21:19:03.102321: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55e02c4eb3b0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-12-21 21:19:03.102370: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): GeForce RTX 3090, Compute Capability 8.6
2020-12-21 21:19:03.107308: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1618] Found device 0 with properties:
name: GeForce RTX 3090 major: 8 minor: 6 memoryClockRate(GHz): 1.71
pciBusID: 0000:03:00.0
2020-12-21 21:19:03.107391: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-12-21 21:19:03.107432: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-12-21 21:19:03.107468: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10.0
2020-12-21 21:19:03.107556: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10.0
2020-12-21 21:19:03.107617: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10.0
2020-12-21 21:19:03.107651: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10.0
2020-12-21 21:19:03.107684: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-12-21 21:19:03.114101: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Adding visible gpu devices: 0
2020-12-21 21:19:03.114148: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.0
2020-12-21 21:19:03.117098: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1159] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-12-21 21:19:03.117140: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1165] 0
2020-12-21 21:19:03.117158: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1178] 0: N
2020-12-21 21:19:03.135120: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1304] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 22797 MB memory) -> physical GPU (device: 0, name: GeForce RTX 3090, pci bus id: 0000:03:00.0, compute capability: 8.6)
WARNING:tensorflow:From /home/zhanglei/.local/lib/python3.6/site-packages/tensorflow_core/python/util/tf_should_use.py:198: initialize_all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2017-03-02.
Instructions for updating:
Use `tf.global_variables_initializer` instead.
2020-12-21 21:23:57.073416: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10.0
2020-12-21 21:25:21.265230: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-12-21 21:41:28.961104: E tensorflow/stream_executor/cuda/cuda_blas.cc:428] failed to run cuBLAS routine: CUBLAS_STATUS_EXECUTION_FAILED
Traceback (most recent call last):
File "/home/zhanglei/.local/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1365, in _do_call
return fn(*args)
File "/home/zhanglei/.local/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1350, in _run_fn
target_list, run_metadata)
File "/home/zhanglei/.local/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1443, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InternalError: Blas GEMM launch failed : a.shape=(150, 6), b.shape=(1000, 6), m=150, n=1000, k=6
[[{
{node gradients/MatMul_1_grad/MatMul}}]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/mnt/ba3b04da-ce1b-4c21-ad1b-3aff7d337cdf/wangxing/WISDM/WISDM-master/WISDN_CSDN_change.py", line 110, in <module>
_, c = session.run([optimizer, loss], feed_dict={X: batch_x, Y: batch_y})
File "/home/zhanglei/.local/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 956, in run
run_metadata_ptr)
File "/home/zhanglei/.local/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1180, in _run
feed_dict_tensor, options, run_metadata)
File "/home/zhanglei/.local/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1359, in _do_run
run_metadata)
File "/home/zhanglei/.local/lib/python3.6/site-packages/tensorflow_core/python/client/session.py", line 1384, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InternalError: Blas GEMM launch failed : a.shape=(150, 6), b.shape=(1000, 6), m=150, n=1000, k=6
[[node gradients/MatMul_1_grad/MatMul (defined at home/zhanglei/.local/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]
Original stack trace for 'gradients/MatMul_1_grad/MatMul':
File "mnt/ba3b04da-ce1b-4c21-ad1b-3aff7d337cdf/wangxing/WISDM/WISDM-master/WISDN_CSDN_change.py", line 96, in <module>
optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate).minimize(loss)
File "home/zhanglei/.local/lib/python3.6/site-packages/tensorflow_core/python/training/optimizer.py", line 403, in minimize
grad_loss=grad_loss)
File "home/zhanglei/.local/lib/python3.6/site-packages/tensorflow_core/python/training/optimizer.py", line 512, in compute_gradients
colocate_gradients_with_ops=colocate_gradients_with_ops)
File "home/zhanglei/.local/lib/python3.6/site-packages/tensorflow_core/python/ops/gradients_impl.py", line 158, in gradients
unconnected_gradients)
File "home/zhanglei/.local/lib/python3.6/site-packages/tensorflow_core/python/ops/gradients_util.py", line 679, in _GradientsHelper
lambda: grad_fn(op, *out_grads))
File "home/zhanglei/.local/lib/python3.6/site-packages/tensorflow_core/python/ops/gradients_util.py", line 350, in _MaybeCompile
return grad_fn() # Exit early
File "home/zhanglei/.local/lib/python3.6/site-packages/tensorflow_core/python/ops/gradients_util.py", line 679, in <lambda>
lambda: grad_fn(op, *out_grads))
File "home/zhanglei/.local/lib/python3.6/site-packages/tensorflow_core/python/ops/math_grad.py", line 1585, in _MatMulGrad
grad_a = gen_math_ops.mat_mul(grad, b, transpose_b=True)
File "home/zhanglei/.local/lib/python3.6/site-packages/tensorflow_core/python/ops/gen_math_ops.py", line 6136, in mat_mul
name=name)
File "home/zhanglei/.local/lib/python3.6/site-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper
op_def=op_def)
File "home/zhanglei/.local/lib/python3.6/site-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "home/zhanglei/.local/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op
attrs, op_def, compute_device)
File "home/zhanglei/.local/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal
op_def=op_def)
File "home/zhanglei/.local/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 1748, in __init__
self._traceback = tf_stack.extract_stack()
...which was originally created as op 'MatMul_1', defined at:
File "mnt/ba3b04da-ce1b-4c21-ad1b-3aff7d337cdf/wangxing/WISDM/WISDM-master/WISDN_CSDN_change.py", line 93, in <module>
y_ = tf.nn.softmax(tf.matmul(f, out_weights) + out_biases)
File "home/zhanglei/.local/lib/python3.6/site-packages/tensorflow_core/python/util/dispatch.py", line 180, in wrapper
return target(*args, **kwargs)
File "home/zhanglei/.local/lib/python3.6/site-packages/tensorflow_core/python/ops/math_ops.py", line 2754, in matmul
a, b, transpose_a=transpose_a, transpose_b=transpose_b, name=name)
File "home/zhanglei/.local/lib/python3.6/site-packages/tensorflow_core/python/ops/gen_math_ops.py", line 6136, in mat_mul
name=name)
File "home/zhanglei/.local/lib/python3.6/site-packages/tensorflow_core/python/framework/op_def_library.py", line 794, in _apply_op_helper
op_def=op_def)
File "home/zhanglei/.local/lib/python3.6/site-packages/tensorflow_core/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "home/zhanglei/.local/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 3357, in create_op
attrs, op_def, compute_device)
File "home/zhanglei/.local/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 3426, in _create_op_internal
op_def=op_def)
File "home/zhanglei/.local/lib/python3.6/site-packages/tensorflow_core/python/framework/ops.py", line 1748, in __init__
self._traceback = tf_stack.extract_stack()
Process finished with exit code 1
先介绍一下我的环境，tf1.15，cuda10，一张3090，一张2080ti
通过nvidia-smi一看，乖乖，显存快占满了，GPU利用率却是0%
本来想去细究为啥会有这一段报错。。。但是，随手一个举动，居然直接解决问题了。
我把
os.environ['CUDA_VISIBLE_DEVICES'] = '0'换成
os.environ['CUDA_VISIBLE_DEVICES'] = '1'就解决问题了，也就是把GPU由3090换成2080ti（我不知道，为啥顺序是反的），可能是cuda10无法支持3090的原因
导致了这一大段的报错