1. Before running the first check GPU usage:
Instruction: nvidia-smi Note: At this point of view the use of GPU
or
instructions: watch nvidia-smi Note: GPU usage in real-time return
2. Specify GPU Training:
method, the program is provided in the python:
Code: os.environ [ 'CUDA_VISIBLE_DEVICES'] = '0' Note: Use GPU 0
Code: os.environ [ 'CUDA_VISIBLE_DEVICES'] = '0,1' Note: use GPU 0,1
second method, when the program execution python:
instruction: CUDA_VISIBLE_DEVICES = 2 python yourcode.py
instructions: CUDA_VISIBLE_DEVICES = 0,1 python yourcode.py
Note: around '=' does not allow spaces
** Note: ** TensorFlow default fill our direct model deployment of storage resources GPU, allowing only a small memory of the program will take up all GPU resources. So sometimes we see the GPU state by nvidia-smi, you will find some GPU computing or computing utilization is low utilization rate is zero, but the store was already occupied, but this time the other person can not use this GPU . But the company's problems now are multi-model card is not enough, all only "civilized" Using the GPU, if set to allow dynamic growth, it has not been accounted for so that the GPU memory can also be used by other people.
3. The method defined GPU occupancy of two kinds: the
method set amount of memory usage GPU:
config = tf.ConfigProto ()
config.gpu_options.per_process_gpu_memory_fraction = 0.4 occupancy GPU40% # of memory
session = tf.Session (config = config)
Second method, provided the minimum amount of GPU memory, dynamic memory application :( recommendation)
config = tf.ConfigProto ()
config.gpu_options.allow_growth = True
the session = tf.Session (config = config)
Note: There is a problem of idle computing resources, computing found a very simple way is to use idle resources command nvidia-smi view the use of state of the GPU, as long as the GPU computing time utilization does not reach 80% -100% ,, are a waste of GPU.
4. The processing of data using multiple threads queue in the CPU, GPU at any time so that sufficient data to use and more focused on training, which can greatly enhance the training speed model.
tf.device with ( '/ CPU: 0'):
# function or to GET Images and Data Process.
distorted_inputs = load_and_distort_images ()
The program has nothing to do with the training try not sess.run () operation
6. Training code optimization and the like, refer to:
http://d0evi1.com/tensorflow/performance/
---------------------
Author: Peanut_ Fan
Source: CSDN
Original: https: //blog.csdn.net/u013841196/article/details/82468884