MonoGRNet Project - train.py
1. ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[3744,256,17,17]
/usr/bin/python2.7 /home/strong/git_workspace/MonoGRNet/train.py --gpus 0
2019-06-23 17:44:16,426 INFO GPU identifiers: 0
['/home/strong/git_workspace/MonoGRNet/train.py']
2019-06-23 17:44:16,426 INFO f: <open file 'hypes/kittiBox.json', mode 'r' at 0x7f02ca0eef60>
2019-06-23 17:44:16,426 INFO Initialize training folder.
2019-06-23 17:44:16,427 INFO Start training.
2019-06-23 17:44:16.563961: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA
2019-06-23 17:44:16.633622: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:892] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2019-06-23 17:44:16.633871: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1030] Found device 0 with properties:
name: GeForce GTX 1080 major: 6 minor: 1 memoryClockRate(GHz): 1.7335
pciBusID: 0000:01:00.0
totalMemory: 7.92GiB freeMemory: 7.49GiB
2019-06-23 17:44:16.633883: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1120] Creating TensorFlow device (/device:GPU:0) -> (device: 0, name: GeForce GTX 1080, pci bus id: 0000:01:00.0, compute capability: 6.1)
pkl file loaded
Layer name: conv1_1
Layer shape: (3, 3, 3, 64)
2019-06-23 17:44:17,747 INFO Creating Summary for: conv1_1/filter
2019-06-23 17:44:17,757 INFO Creating Summary for: conv1_1/biases
Layer name: conv1_2
Layer shape: (3, 3, 64, 64)
2019-06-23 17:44:17,771 INFO Creating Summary for: conv1_2/filter
2019-06-23 17:44:17,780 INFO Creating Summary for: conv1_2/biases
Layer name: conv2_1
Layer shape: (3, 3, 64, 128)
2019-06-23 17:44:17,793 INFO Creating Summary for: conv2_1/filter
2019-06-23 17:44:17,803 INFO Creating Summary for: conv2_1/biases
Layer name: conv2_2
Layer shape: (3, 3, 128, 128)
2019-06-23 17:44:17,816 INFO Creating Summary for: conv2_2/filter
2019-06-23 17:44:17,827 INFO Creating Summary for: conv2_2/biases
Layer name: conv3_1
Layer shape: (3, 3, 128, 256)
2019-06-23 17:44:17,841 INFO Creating Summary for: conv3_1/filter
2019-06-23 17:44:17,850 INFO Creating Summary for: conv3_1/biases
Layer name: conv3_2
Layer shape: (3, 3, 256, 256)
2019-06-23 17:44:17,864 INFO Creating Summary for: conv3_2/filter
2019-06-23 17:44:17,874 INFO Creating Summary for: conv3_2/biases
Layer name: conv3_3
Layer shape: (3, 3, 256, 256)
2019-06-23 17:44:17,888 INFO Creating Summary for: conv3_3/filter
2019-06-23 17:44:17,898 INFO Creating Summary for: conv3_3/biases
Layer name: conv4_1
Layer shape: (3, 3, 256, 512)
2019-06-23 17:44:17,913 INFO Creating Summary for: conv4_1/filter
2019-06-23 17:44:17,923 INFO Creating Summary for: conv4_1/biases
Layer name: conv4_2
Layer shape: (3, 3, 512, 512)
2019-06-23 17:44:17,938 INFO Creating Summary for: conv4_2/filter
2019-06-23 17:44:17,948 INFO Creating Summary for: conv4_2/biases
Layer name: conv4_3
Layer shape: (3, 3, 512, 512)
2019-06-23 17:44:17,965 INFO Creating Summary for: conv4_3/filter
2019-06-23 17:44:17,975 INFO Creating Summary for: conv4_3/biases
Layer name: conv4_encoder_full_neck
Layer shape: (3, 3, 512, 256)
2019-06-23 17:44:18,170 INFO Creating Summary for: conv4_encoder_full_neck/filter
2019-06-23 17:44:18,191 INFO Creating Summary for: conv4_encoder_full_neck/biases
Layer name: conv4_encoder_full
Layer shape: (3, 3, 256, 128)
2019-06-23 17:44:18,206 INFO Creating Summary for: conv4_encoder_full/filter
2019-06-23 17:44:18,225 INFO Creating Summary for: conv4_encoder_full/biases
Layer name: conv4_encoder_atrous_1
Layer shape: (3, 3, 256, 128)
2019-06-23 17:44:18,241 INFO Creating Summary for: conv4_encoder_atrous_1/filter
2019-06-23 17:44:18,263 INFO Creating Summary for: conv4_encoder_atrous_1/biases
Layer name: conv4_encoder_atrous_2
Layer shape: (3, 3, 256, 128)
2019-06-23 17:44:18,278 INFO Creating Summary for: conv4_encoder_atrous_2/filter
2019-06-23 17:44:18,301 INFO Creating Summary for: conv4_encoder_atrous_2/biases
Layer name: conv4_encoder_atrous_3
Layer shape: (3, 3, 256, 128)
2019-06-23 17:44:18,317 INFO Creating Summary for: conv4_encoder_atrous_3/filter
2019-06-23 17:44:18,338 INFO Creating Summary for: conv4_encoder_atrous_3/biases
Layer name: conv4_encoder_conv_neck
Layer shape: (1, 1, 512, 128)
2019-06-23 17:44:18,354 INFO Creating Summary for: conv4_encoder_conv_neck/filter
2019-06-23 17:44:18,374 INFO Creating Summary for: conv4_encoder_conv_neck/biases
Layer name: conv4_encoder_location_neck
Layer shape: (3, 3, 512, 256)
2019-06-23 17:44:18,390 INFO Creating Summary for: conv4_encoder_location_neck/filter
2019-06-23 17:44:18,410 INFO Creating Summary for: conv4_encoder_location_neck/biases
Layer name: conv4_encoder_location_head
Layer shape: (3, 3, 256, 128)
2019-06-23 17:44:18,426 INFO Creating Summary for: conv4_encoder_location_head/filter
2019-06-23 17:44:18,446 INFO Creating Summary for: conv4_encoder_location_head/biases
Layer name: conv4_encoder_corners_neck
Layer shape: (3, 3, 512, 256)
2019-06-23 17:44:18,462 INFO Creating Summary for: conv4_encoder_corners_neck/filter
2019-06-23 17:44:18,482 INFO Creating Summary for: conv4_encoder_corners_neck/biases
Layer name: conv4_encoder_corners_head
Layer shape: (3, 3, 256, 128)
2019-06-23 17:44:18,497 INFO Creating Summary for: conv4_encoder_corners_head/filter
2019-06-23 17:44:18,516 INFO Creating Summary for: conv4_encoder_corners_head/biases
Layer name: conv5_1
Layer shape: (3, 3, 512, 512)
2019-06-23 17:44:18,537 INFO Creating Summary for: conv5_1/filter
2019-06-23 17:44:18,594 INFO Creating Summary for: conv5_1/biases
Layer name: conv5_2
Layer shape: (3, 3, 512, 512)
2019-06-23 17:44:18,615 INFO Creating Summary for: conv5_2/filter
2019-06-23 17:44:18,625 INFO Creating Summary for: conv5_2/biases
Layer name: conv5_3
Layer shape: (3, 3, 512, 512)
2019-06-23 17:44:18,644 INFO Creating Summary for: conv5_3/filter
2019-06-23 17:44:18,655 INFO Creating Summary for: conv5_3/biases
Layer name: pool5_encoder_full_neck
Layer shape: (3, 3, 512, 256)
2019-06-23 17:44:18,673 INFO Creating Summary for: pool5_encoder_full_neck/filter
2019-06-23 17:44:18,691 INFO Creating Summary for: pool5_encoder_full_neck/biases
Layer name: pool5_encoder_full
Layer shape: (3, 3, 256, 128)
2019-06-23 17:44:18,706 INFO Creating Summary for: pool5_encoder_full/filter
2019-06-23 17:44:18,725 INFO Creating Summary for: pool5_encoder_full/biases
Layer name: pool5_encoder_atrous_1
Layer shape: (3, 3, 256, 128)
2019-06-23 17:44:18,740 INFO Creating Summary for: pool5_encoder_atrous_1/filter
2019-06-23 17:44:18,762 INFO Creating Summary for: pool5_encoder_atrous_1/biases
Layer name: pool5_encoder_atrous_2
Layer shape: (3, 3, 256, 128)
2019-06-23 17:44:18,777 INFO Creating Summary for: pool5_encoder_atrous_2/filter
2019-06-23 17:44:18,799 INFO Creating Summary for: pool5_encoder_atrous_2/biases
Layer name: pool5_encoder_atrous_3
Layer shape: (3, 3, 256, 128)
2019-06-23 17:44:18,817 INFO Creating Summary for: pool5_encoder_atrous_3/filter
2019-06-23 17:44:18,841 INFO Creating Summary for: pool5_encoder_atrous_3/biases
Layer name: pool5_encoder_conv_neck
Layer shape: (1, 1, 512, 128)
2019-06-23 17:44:18,857 INFO Creating Summary for: pool5_encoder_conv_neck/filter
2019-06-23 17:44:18,878 INFO Creating Summary for: pool5_encoder_conv_neck/biases
Layer name: pool5_encoder_location_neck
Layer shape: (3, 3, 512, 256)
2019-06-23 17:44:18,894 INFO Creating Summary for: pool5_encoder_location_neck/filter
2019-06-23 17:44:18,914 INFO Creating Summary for: pool5_encoder_location_neck/biases
Layer name: pool5_encoder_location_head
Layer shape: (3, 3, 256, 128)
2019-06-23 17:44:18,929 INFO Creating Summary for: pool5_encoder_location_head/filter
2019-06-23 17:44:18,947 INFO Creating Summary for: pool5_encoder_location_head/biases
Layer name: pool5_encoder_corners_neck
Layer shape: (3, 3, 512, 256)
2019-06-23 17:44:18,964 INFO Creating Summary for: pool5_encoder_corners_neck/filter
2019-06-23 17:44:18,982 INFO Creating Summary for: pool5_encoder_corners_neck/biases
Layer name: pool5_encoder_corners_head
Layer shape: (3, 3, 256, 128)
2019-06-23 17:44:18,997 INFO Creating Summary for: pool5_encoder_corners_head/filter
2019-06-23 17:44:19,017 INFO Creating Summary for: pool5_encoder_corners_head/biases
pkl file loaded
Layer name: conv1_1
Layer shape: (3, 3, 3, 64)
Layer name: conv1_2
Layer shape: (3, 3, 64, 64)
Layer name: conv2_1
Layer shape: (3, 3, 64, 128)
Layer name: conv2_2
Layer shape: (3, 3, 128, 128)
Layer name: conv3_1
Layer shape: (3, 3, 128, 256)
Layer name: conv3_2
Layer shape: (3, 3, 256, 256)
Layer name: conv3_3
Layer shape: (3, 3, 256, 256)
Layer name: conv4_1
Layer shape: (3, 3, 256, 512)
Layer name: conv4_2
Layer shape: (3, 3, 512, 512)
Layer name: conv4_3
Layer shape: (3, 3, 512, 512)
Layer name: conv4_encoder_full_neck
Layer shape: (3, 3, 512, 256)
Layer name: conv4_encoder_full
Layer shape: (3, 3, 256, 128)
Layer name: conv4_encoder_atrous_1
Layer shape: (3, 3, 256, 128)
Layer name: conv4_encoder_atrous_2
Layer shape: (3, 3, 256, 128)
Layer name: conv4_encoder_atrous_3
Layer shape: (3, 3, 256, 128)
Layer name: conv4_encoder_conv_neck
Layer shape: (1, 1, 512, 128)
Layer name: conv4_encoder_location_neck
Layer shape: (3, 3, 512, 256)
Layer name: conv4_encoder_location_head
Layer shape: (3, 3, 256, 128)
Layer name: conv4_encoder_corners_neck
Layer shape: (3, 3, 512, 256)
Layer name: conv4_encoder_corners_head
Layer shape: (3, 3, 256, 128)
Layer name: conv5_1
Layer shape: (3, 3, 512, 512)
Layer name: conv5_2
Layer shape: (3, 3, 512, 512)
Layer name: conv5_3
Layer shape: (3, 3, 512, 512)
Layer name: pool5_encoder_full_neck
Layer shape: (3, 3, 512, 256)
Layer name: pool5_encoder_full
Layer shape: (3, 3, 256, 128)
Layer name: pool5_encoder_atrous_1
Layer shape: (3, 3, 256, 128)
Layer name: pool5_encoder_atrous_2
Layer shape: (3, 3, 256, 128)
Layer name: pool5_encoder_atrous_3
Layer shape: (3, 3, 256, 128)
Layer name: pool5_encoder_conv_neck
Layer shape: (1, 1, 512, 128)
Layer name: pool5_encoder_location_neck
Layer shape: (3, 3, 512, 256)
Layer name: pool5_encoder_location_head
Layer shape: (3, 3, 256, 128)
Layer name: pool5_encoder_corners_neck
Layer shape: (3, 3, 512, 256)
Layer name: pool5_encoder_corners_head
Layer shape: (3, 3, 256, 128)
INFO:tensorflow:Restoring parameters from data/model_3D
2019-06-23 17:44:29,333 INFO Restoring parameters from data/model_3D
2019-06-23 17:44:45.157298: W tensorflow/core/common_runtime/bfc_allocator.cc:273] Allocator (GPU_0_bfc) ran out of memory trying to allocate 1.03GiB. Current allocation summary follows.
2019-06-23 17:44:45.157336: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (256): Total Chunks: 181, Chunks in use: 181. 45.2KiB allocated for chunks. 45.2KiB in use in bin. 2.2KiB client-requested in use in bin.
2019-06-23 17:44:45.157343: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (512): Total Chunks: 63, Chunks in use: 63. 31.5KiB allocated for chunks. 31.5KiB in use in bin. 31.5KiB client-requested in use in bin.
2019-06-23 17:44:45.157347: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (1024): Total Chunks: 50, Chunks in use: 50. 51.0KiB allocated for chunks. 51.0KiB in use in bin. 50.0KiB client-requested in use in bin.
2019-06-23 17:44:45.157351: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (2048): Total Chunks: 21, Chunks in use: 21. 42.0KiB allocated for chunks. 42.0KiB in use in bin. 42.0KiB client-requested in use in bin.
2019-06-23 17:44:45.157355: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (4096): Total Chunks: 6, Chunks in use: 6. 32.2KiB allocated for chunks. 32.2KiB in use in bin. 32.2KiB client-requested in use in bin.
2019-06-23 17:44:45.157359: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (8192): Total Chunks: 12, Chunks in use: 12. 156.8KiB allocated for chunks. 156.8KiB in use in bin. 155.6KiB client-requested in use in bin.
2019-06-23 17:44:45.157363: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (16384): Total Chunks: 2, Chunks in use: 1. 51.0KiB allocated for chunks. 29.2KiB in use in bin. 29.2KiB client-requested in use in bin.
2019-06-23 17:44:45.157367: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (32768): Total Chunks: 2, Chunks in use: 2. 76.0KiB allocated for chunks. 76.0KiB in use in bin. 75.9KiB client-requested in use in bin.
2019-06-23 17:44:45.157371: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (65536): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-06-23 17:44:45.157375: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (131072): Total Chunks: 22, Chunks in use: 22. 3.10MiB allocated for chunks. 3.10MiB in use in bin. 2.79MiB client-requested in use in bin.
2019-06-23 17:44:45.157378: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (262144): Total Chunks: 13, Chunks in use: 13. 4.34MiB allocated for chunks. 4.34MiB in use in bin. 4.34MiB client-requested in use in bin.
2019-06-23 17:44:45.157382: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (524288): Total Chunks: 5, Chunks in use: 4. 3.12MiB allocated for chunks. 2.19MiB in use in bin. 1.97MiB client-requested in use in bin.
2019-06-23 17:44:45.157392: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (1048576): Total Chunks: 21, Chunks in use: 21. 24.50MiB allocated for chunks. 24.50MiB in use in bin. 24.00MiB client-requested in use in bin.
2019-06-23 17:44:45.157397: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (2097152): Total Chunks: 19, Chunks in use: 19. 43.00MiB allocated for chunks. 43.00MiB in use in bin. 42.00MiB client-requested in use in bin.
2019-06-23 17:44:45.157401: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (4194304): Total Chunks: 10, Chunks in use: 9. 47.14MiB allocated for chunks. 40.50MiB in use in bin. 40.50MiB client-requested in use in bin.
2019-06-23 17:44:45.157405: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (8388608): Total Chunks: 15, Chunks in use: 15. 135.00MiB allocated for chunks. 135.00MiB in use in bin. 135.00MiB client-requested in use in bin.
2019-06-23 17:44:45.157409: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (16777216): Total Chunks: 1, Chunks in use: 0. 29.64MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-06-23 17:44:45.157413: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (33554432): Total Chunks: 2, Chunks in use: 0. 79.09MiB allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-06-23 17:44:45.157417: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (67108864): Total Chunks: 0, Chunks in use: 0. 0B allocated for chunks. 0B in use in bin. 0B client-requested in use in bin.
2019-06-23 17:44:45.157421: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (134217728): Total Chunks: 2, Chunks in use: 1. 482.64MiB allocated for chunks. 234.00MiB in use in bin. 234.00MiB client-requested in use in bin.
2019-06-23 17:44:45.157424: I tensorflow/core/common_runtime/bfc_allocator.cc:627] Bin (268435456): Total Chunks: 8, Chunks in use: 8. 6.28GiB allocated for chunks. 6.28GiB in use in bin. 6.06GiB client-requested in use in bin.
2019-06-23 17:44:45.157428: I tensorflow/core/common_runtime/bfc_allocator.cc:643] Bin for 1.03GiB was 256.00MiB, Chunk State:
2019-06-23 17:44:45.157433: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020b400000 of size 1280
2019-06-23 17:44:45.157436: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020b400500 of size 6912
2019-06-23 17:44:45.157438: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020b402000 of size 256
2019-06-23 17:44:45.157441: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020b402100 of size 147456
2019-06-23 17:44:45.157444: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020b426100 of size 256
2019-06-23 17:44:45.157447: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020b426200 of size 294912
2019-06-23 17:44:45.157450: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020b46e200 of size 512
2019-06-23 17:44:45.157453: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020b46e400 of size 589824
2019-06-23 17:44:45.157455: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020b4fe400 of size 512
2019-06-23 17:44:45.157458: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020b4fe600 of size 1179648
2019-06-23 17:44:45.157461: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020b61e600 of size 1024
2019-06-23 17:44:45.157463: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020b61ea00 of size 2359296
2019-06-23 17:44:45.157466: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020b85ea00 of size 1024
2019-06-23 17:44:45.157469: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020b85ee00 of size 2359296
2019-06-23 17:44:45.157471: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020ba9ee00 of size 1024
2019-06-23 17:44:45.157474: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020ba9f200 of size 4718592
2019-06-23 17:44:45.157477: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020bf1f200 of size 2048
2019-06-23 17:44:45.157479: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020bf1fa00 of size 9437184
2019-06-23 17:44:45.157482: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020c81fa00 of size 2048
2019-06-23 17:44:45.157485: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020c820200 of size 9437184
2019-06-23 17:44:45.157487: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d120200 of size 2048
2019-06-23 17:44:45.157490: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d120a00 of size 256
2019-06-23 17:44:45.157492: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d120b00 of size 256
2019-06-23 17:44:45.157495: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d120c00 of size 1024
2019-06-23 17:44:45.157498: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d121000 of size 256
2019-06-23 17:44:45.157500: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d121100 of size 256
2019-06-23 17:44:45.157503: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d121200 of size 256
2019-06-23 17:44:45.157506: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d121300 of size 256
2019-06-23 17:44:45.157508: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d121400 of size 512
2019-06-23 17:44:45.157511: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d121600 of size 256
2019-06-23 17:44:45.157513: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d121700 of size 256
2019-06-23 17:44:45.157516: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d121800 of size 256
2019-06-23 17:44:45.157518: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d121900 of size 256
2019-06-23 17:44:45.157521: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d121a00 of size 512
2019-06-23 17:44:45.157523: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d121c00 of size 256
2019-06-23 17:44:45.157526: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d121d00 of size 256
2019-06-23 17:44:45.157529: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d121e00 of size 512
2019-06-23 17:44:45.157531: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d122000 of size 256
2019-06-23 17:44:45.157534: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d122100 of size 256
2019-06-23 17:44:45.157536: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d122200 of size 256
2019-06-23 17:44:45.157539: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d122300 of size 512
2019-06-23 17:44:45.157541: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d122500 of size 256
2019-06-23 17:44:45.157544: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d122600 of size 256
2019-06-23 17:44:45.157546: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d122700 of size 256
2019-06-23 17:44:45.157549: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d122800 of size 512
2019-06-23 17:44:45.157552: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d122a00 of size 131072
2019-06-23 17:44:45.157556: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d142a00 of size 256
2019-06-23 17:44:45.157560: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d142b00 of size 256
2019-06-23 17:44:45.157564: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d142c00 of size 1024
2019-06-23 17:44:45.157568: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d143000 of size 256
2019-06-23 17:44:45.157571: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d143100 of size 256
2019-06-23 17:44:45.157575: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d143200 of size 512
2019-06-23 17:44:45.157579: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d143400 of size 256
2019-06-23 17:44:45.157583: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d143500 of size 256
2019-06-23 17:44:45.157587: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d143600 of size 1024
2019-06-23 17:44:45.157591: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d143a00 of size 131072
2019-06-23 17:44:45.157595: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d163a00 of size 256
2019-06-23 17:44:45.157599: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d163b00 of size 256
2019-06-23 17:44:45.157618: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d163c00 of size 512
2019-06-23 17:44:45.157621: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d163e00 of size 256
2019-06-23 17:44:45.157625: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020d163f00 of size 9437184
2019-06-23 17:44:45.157627: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020da63f00 of size 2048
2019-06-23 17:44:45.157630: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020da64700 of size 9437184
2019-06-23 17:44:45.157633: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020e364700 of size 2048
2019-06-23 17:44:45.157635: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020e364f00 of size 9437184
2019-06-23 17:44:45.157638: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020ec64f00 of size 2048
2019-06-23 17:44:45.157641: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020ec65700 of size 256
2019-06-23 17:44:45.157643: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020ec65800 of size 1024
2019-06-23 17:44:45.157646: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020ec65c00 of size 256
2019-06-23 17:44:45.157649: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020ec65d00 of size 256
2019-06-23 17:44:45.157651: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020ec65e00 of size 512
2019-06-23 17:44:45.157654: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020ec66000 of size 131072
2019-06-23 17:44:45.157657: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020ec86000 of size 256
2019-06-23 17:44:45.157659: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020ec86100 of size 256
2019-06-23 17:44:45.157662: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020ec86200 of size 512
2019-06-23 17:44:45.157664: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020ec86400 of size 256
2019-06-23 17:44:45.157668: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020ec86500 of size 393216
2019-06-23 17:44:45.157672: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020ece6500 of size 256
2019-06-23 17:44:45.157676: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020ece6600 of size 512
2019-06-23 17:44:45.157679: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020ece6800 of size 256
2019-06-23 17:44:45.157682: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020ece6900 of size 256
2019-06-23 17:44:45.157684: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020ece6a00 of size 512
2019-06-23 17:44:45.157687: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020ece6c00 of size 2359296
2019-06-23 17:44:45.157702: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020ef26c00 of size 256
2019-06-23 17:44:45.157705: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020ef26d00 of size 256
2019-06-23 17:44:45.157707: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020ef26e00 of size 512
2019-06-23 17:44:45.157710: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020ef27000 of size 256
2019-06-23 17:44:45.157712: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020ef27100 of size 256
2019-06-23 17:44:45.157715: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020ef27200 of size 1024
2019-06-23 17:44:45.157718: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020ef27600 of size 256
2019-06-23 17:44:45.157720: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020ef27700 of size 256
2019-06-23 17:44:45.157723: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020ef27800 of size 512
2019-06-23 17:44:45.157725: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020ef27a00 of size 1024
2019-06-23 17:44:45.157728: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020ef27e00 of size 256
2019-06-23 17:44:45.157731: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020ef27f00 of size 256
2019-06-23 17:44:45.157733: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020ef28000 of size 1024
2019-06-23 17:44:45.157736: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020ef28400 of size 256
2019-06-23 17:44:45.157738: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020ef28500 of size 256
2019-06-23 17:44:45.157741: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020ef28600 of size 512
2019-06-23 17:44:45.157744: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020ef28800 of size 256
2019-06-23 17:44:45.157746: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020ef28900 of size 1048576
2019-06-23 17:44:45.157749: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020f028900 of size 8192
2019-06-23 17:44:45.157752: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020f02a900 of size 4096
2019-06-23 17:44:45.157754: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020f02b900 of size 256
2019-06-23 17:44:45.157757: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020f02ba00 of size 256
2019-06-23 17:44:45.157760: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020f02bb00 of size 256
2019-06-23 17:44:45.157762: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020f02bc00 of size 1441792
2019-06-23 17:44:45.157765: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020f18bc00 of size 1024
2019-06-23 17:44:45.157767: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020f18c000 of size 2048
2019-06-23 17:44:45.157770: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020f18c800 of size 2359296
2019-06-23 17:44:45.157773: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020f3cc800 of size 256
2019-06-23 17:44:45.157775: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020f3cc900 of size 256
2019-06-23 17:44:45.157778: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020f3cca00 of size 256
2019-06-23 17:44:45.157780: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020f3ccb00 of size 256
2019-06-23 17:44:45.157783: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020f3ccc00 of size 256
2019-06-23 17:44:45.157786: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020f3ccd00 of size 256
2019-06-23 17:44:45.157788: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020f3cce00 of size 1024
2019-06-23 17:44:45.157791: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020f3cd200 of size 256
2019-06-23 17:44:45.157793: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020f3cd300 of size 256
2019-06-23 17:44:45.157796: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020f3cd400 of size 256
2019-06-23 17:44:45.157798: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020f3cd500 of size 256
2019-06-23 17:44:45.157801: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020f3cd600 of size 512
2019-06-23 17:44:45.157804: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020f3cd800 of size 1024
2019-06-23 17:44:45.157806: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020f3cdc00 of size 1024
2019-06-23 17:44:45.157809: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020f3ce000 of size 1024
2019-06-23 17:44:45.157811: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020f3ce400 of size 1024
2019-06-23 17:44:45.157814: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020f3ce800 of size 32768
2019-06-23 17:44:45.157817: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020f3d6800 of size 225280
2019-06-23 17:44:45.157820: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020f40d800 of size 512
2019-06-23 17:44:45.157822: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020f40da00 of size 512
2019-06-23 17:44:45.157825: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020f40dc00 of size 131072
2019-06-23 17:44:45.157828: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020f42dc00 of size 1179648
2019-06-23 17:44:45.157830: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020f54dc00 of size 1179648
2019-06-23 17:44:45.157833: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020f66dc00 of size 2359296
2019-06-23 17:44:45.157835: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020f8adc00 of size 1024
2019-06-23 17:44:45.157838: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020f8ae000 of size 1024
2019-06-23 17:44:45.157841: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020f8ae400 of size 131072
2019-06-23 17:44:45.157843: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020f8ce400 of size 1179648
2019-06-23 17:44:45.157846: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020f9ee400 of size 512
2019-06-23 17:44:45.157848: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020f9ee600 of size 4719104
2019-06-23 17:44:45.157851: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020fe6e800 of size 1024
2019-06-23 17:44:45.157854: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020fe6ec00 of size 512
2019-06-23 17:44:45.157856: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020fe6ee00 of size 512
2019-06-23 17:44:45.157859: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020fe6f000 of size 131072
2019-06-23 17:44:45.157862: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020fe8f000 of size 1179648
2019-06-23 17:44:45.157864: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020ffaf000 of size 512
2019-06-23 17:44:45.157867: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020ffaf200 of size 512
2019-06-23 17:44:45.157869: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020ffaf400 of size 131072
2019-06-23 17:44:45.157872: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1020ffcf400 of size 9437184
2019-06-23 17:44:45.157875: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102108cf400 of size 2048
2019-06-23 17:44:45.157877: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102108cfc00 of size 9437184
2019-06-23 17:44:45.157880: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102111cfc00 of size 2048
2019-06-23 17:44:45.157882: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102111d0400 of size 9437184
2019-06-23 17:44:45.157885: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10211ad0400 of size 2048
2019-06-23 17:44:45.157887: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10211ad0c00 of size 4718592
2019-06-23 17:44:45.157890: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10211f50c00 of size 1024
2019-06-23 17:44:45.157893: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10211f51000 of size 1180672
2019-06-23 17:44:45.157895: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10212071400 of size 512
2019-06-23 17:44:45.157898: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10212071600 of size 512
2019-06-23 17:44:45.157900: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10212071800 of size 131072
2019-06-23 17:44:45.157903: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10212091800 of size 1179648
2019-06-23 17:44:45.157906: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102121b1800 of size 512
2019-06-23 17:44:45.157908: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102121b1a00 of size 512
2019-06-23 17:44:45.157911: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102121b1c00 of size 131072
2019-06-23 17:44:45.157914: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102121d1c00 of size 1179648
2019-06-23 17:44:45.157916: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102122f1c00 of size 393216
2019-06-23 17:44:45.157919: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10212351c00 of size 512
2019-06-23 17:44:45.157922: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10212351e00 of size 262144
2019-06-23 17:44:45.157924: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10212391e00 of size 393216
2019-06-23 17:44:45.157927: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102123f1e00 of size 524800
2019-06-23 17:44:45.157930: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10212472000 of size 512
2019-06-23 17:44:45.157932: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10212472200 of size 512
2019-06-23 17:44:45.157935: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10212472400 of size 2359296
2019-06-23 17:44:45.157937: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102126b2400 of size 131072
2019-06-23 17:44:45.157940: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102126d2400 of size 131072
2019-06-23 17:44:45.157942: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102126f2400 of size 512
2019-06-23 17:44:45.157945: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102126f2600 of size 512
2019-06-23 17:44:45.157948: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102126f2800 of size 2359296
2019-06-23 17:44:45.157950: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10212932800 of size 4718592
2019-06-23 17:44:45.157953: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10212db2800 of size 1024
2019-06-23 17:44:45.157955: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10212db2c00 of size 1180672
2019-06-23 17:44:45.157958: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10212ed3000 of size 512
2019-06-23 17:44:45.157961: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10212ed3200 of size 512
2019-06-23 17:44:45.157963: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10212ed3400 of size 1024
2019-06-23 17:44:45.157966: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10212ed3800 of size 2359296
2019-06-23 17:44:45.157968: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10213113800 of size 2359296
2019-06-23 17:44:45.157971: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10213353800 of size 1024
2019-06-23 17:44:45.157973: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10213353c00 of size 1024
2019-06-23 17:44:45.157976: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10213354000 of size 1024
2019-06-23 17:44:45.157979: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10213354400 of size 1179648
2019-06-23 17:44:45.157981: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10213474400 of size 393216
2019-06-23 17:44:45.157984: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102134d4400 of size 512
2019-06-23 17:44:45.157986: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102134d4600 of size 512
2019-06-23 17:44:45.157989: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102134d4800 of size 1048576
2019-06-23 17:44:45.157992: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102135d4800 of size 8192
2019-06-23 17:44:45.157994: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102135d6800 of size 4096
2019-06-23 17:44:45.157997: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102135d7800 of size 1536
2019-06-23 17:44:45.158000: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102135d7e00 of size 1441792
2019-06-23 17:44:45.158003: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10213737e00 of size 1024
2019-06-23 17:44:45.158005: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10213738200 of size 2048
2019-06-23 17:44:45.158008: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10213738a00 of size 1024
2019-06-23 17:44:45.158010: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10213738e00 of size 1024
2019-06-23 17:44:45.158013: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10213739200 of size 512
2019-06-23 17:44:45.158016: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10213739400 of size 512
2019-06-23 17:44:45.158018: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10213739600 of size 512
2019-06-23 17:44:45.158021: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10213739800 of size 6912
2019-06-23 17:44:45.158023: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021373b300 of size 512
2019-06-23 17:44:45.158026: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021373b500 of size 512
2019-06-23 17:44:45.158029: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021373b700 of size 1024
2019-06-23 17:44:45.158031: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021373bb00 of size 1024
2019-06-23 17:44:45.158034: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021373bf00 of size 1024
2019-06-23 17:44:45.158036: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021373c300 of size 2048
2019-06-23 17:44:45.158039: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021373cb00 of size 2048
2019-06-23 17:44:45.158041: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021373d300 of size 2048
2019-06-23 17:44:45.158044: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021373db00 of size 256
2019-06-23 17:44:45.158047: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021373dc00 of size 256
2019-06-23 17:44:45.158049: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021373dd00 of size 1024
2019-06-23 17:44:45.158052: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021373e100 of size 256
2019-06-23 17:44:45.158054: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021373e200 of size 256
2019-06-23 17:44:45.158057: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021373e300 of size 256
2019-06-23 17:44:45.158059: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021373e400 of size 256
2019-06-23 17:44:45.158062: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021373e500 of size 512
2019-06-23 17:44:45.158065: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021373e700 of size 256
2019-06-23 17:44:45.158067: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021373e800 of size 256
2019-06-23 17:44:45.158070: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021373e900 of size 256
2019-06-23 17:44:45.158072: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021373ea00 of size 256
2019-06-23 17:44:45.158075: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021373eb00 of size 512
2019-06-23 17:44:45.158077: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021373ed00 of size 256
2019-06-23 17:44:45.158080: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021373ee00 of size 256
2019-06-23 17:44:45.158083: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021373ef00 of size 512
2019-06-23 17:44:45.158085: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021373f100 of size 256
2019-06-23 17:44:45.158088: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021373f200 of size 256
2019-06-23 17:44:45.158091: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021373f300 of size 256
2019-06-23 17:44:45.158093: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021373f400 of size 512
2019-06-23 17:44:45.158096: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021373f600 of size 256
2019-06-23 17:44:45.158098: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021373f700 of size 256
2019-06-23 17:44:45.158101: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021373f800 of size 256
2019-06-23 17:44:45.158103: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021373f900 of size 512
2019-06-23 17:44:45.158106: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021373fb00 of size 256
2019-06-23 17:44:45.158108: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021373fc00 of size 256
2019-06-23 17:44:45.158111: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021373fd00 of size 1024
2019-06-23 17:44:45.158114: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10213740100 of size 256
2019-06-23 17:44:45.158116: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10213740200 of size 256
2019-06-23 17:44:45.158119: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10213740300 of size 512
2019-06-23 17:44:45.158121: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10213740500 of size 256
2019-06-23 17:44:45.158124: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10213740600 of size 256
2019-06-23 17:44:45.158126: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10213740700 of size 256
2019-06-23 17:44:45.158129: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10213740800 of size 256
2019-06-23 17:44:45.158132: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10213740900 of size 256
2019-06-23 17:44:45.158134: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10213740a00 of size 2359296
2019-06-23 17:44:45.158137: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10213980a00 of size 1703936
2019-06-23 17:44:45.158140: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10213b20a00 of size 2359296
2019-06-23 17:44:45.158142: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10213d60a00 of size 2622464
2019-06-23 17:44:45.158145: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10213fe0e00 of size 1024
2019-06-23 17:44:45.158148: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10213fe1200 of size 2360320
2019-06-23 17:44:45.158150: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10214221600 of size 1024
2019-06-23 17:44:45.158153: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10214221a00 of size 147456
2019-06-23 17:44:45.158156: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10214245a00 of size 245760
2019-06-23 17:44:45.158159: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10214281a00 of size 6912
2019-06-23 17:44:45.158161: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10214283500 of size 256
2019-06-23 17:44:45.158164: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10214283600 of size 147456
2019-06-23 17:44:45.158166: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102142a7600 of size 256
2019-06-23 17:44:45.158169: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102142a7700 of size 294912
2019-06-23 17:44:45.158172: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102142ef700 of size 512
2019-06-23 17:44:45.158175: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102142ef900 of size 589824
2019-06-23 17:44:45.158177: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021437f900 of size 512
2019-06-23 17:44:45.158180: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021437fb00 of size 1179648
2019-06-23 17:44:45.158183: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021449fb00 of size 1024
2019-06-23 17:44:45.158185: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021449ff00 of size 2359296
2019-06-23 17:44:45.158188: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102146dff00 of size 1024
2019-06-23 17:44:45.158190: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102146e0300 of size 2359296
2019-06-23 17:44:45.158193: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10214920300 of size 1024
2019-06-23 17:44:45.158196: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10214920700 of size 4718592
2019-06-23 17:44:45.158198: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10214da0700 of size 2048
2019-06-23 17:44:45.158201: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10214da0f00 of size 9437184
2019-06-23 17:44:45.158203: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102156a0f00 of size 2048
2019-06-23 17:44:45.158206: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102156a1700 of size 9437184
2019-06-23 17:44:45.158209: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10215fa1700 of size 2048
2019-06-23 17:44:45.158211: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10215fa1f00 of size 4718592
2019-06-23 17:44:45.158214: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216421f00 of size 1024
2019-06-23 17:44:45.158216: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216422300 of size 1180672
2019-06-23 17:44:45.158219: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216542700 of size 512
2019-06-23 17:44:45.158222: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216542900 of size 1180160
2019-06-23 17:44:45.158224: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216662b00 of size 256
2019-06-23 17:44:45.158227: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216662c00 of size 512
2019-06-23 17:44:45.158229: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216662e00 of size 1180160
2019-06-23 17:44:45.158232: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216783000 of size 512
2019-06-23 17:44:45.158235: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216783200 of size 256
2019-06-23 17:44:45.158237: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216783300 of size 256
2019-06-23 17:44:45.158240: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216783400 of size 256
2019-06-23 17:44:45.158242: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216783500 of size 589824
2019-06-23 17:44:45.158245: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216813500 of size 1024
2019-06-23 17:44:45.158247: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216813900 of size 131072
2019-06-23 17:44:45.158250: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216833900 of size 512
2019-06-23 17:44:45.158253: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216833b00 of size 2048
2019-06-23 17:44:45.158255: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216834300 of size 2048
2019-06-23 17:44:45.158258: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216834b00 of size 2048
2019-06-23 17:44:45.158260: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216835300 of size 256
2019-06-23 17:44:45.158263: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216835400 of size 1024
2019-06-23 17:44:45.158266: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216835800 of size 256
2019-06-23 17:44:45.158268: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216835900 of size 256
2019-06-23 17:44:45.158271: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216835a00 of size 512
2019-06-23 17:44:45.158274: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216835c00 of size 131072
2019-06-23 17:44:45.158276: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216855c00 of size 256
2019-06-23 17:44:45.158279: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216855d00 of size 256
2019-06-23 17:44:45.158281: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216855e00 of size 512
2019-06-23 17:44:45.158284: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216856000 of size 256
2019-06-23 17:44:45.158286: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216856100 of size 256
2019-06-23 17:44:45.158289: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216856200 of size 512
2019-06-23 17:44:45.158291: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216856400 of size 256
2019-06-23 17:44:45.158294: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216856500 of size 256
2019-06-23 17:44:45.158296: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216856600 of size 512
2019-06-23 17:44:45.158299: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216856800 of size 256
2019-06-23 17:44:45.158301: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216856900 of size 256
2019-06-23 17:44:45.158304: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216856a00 of size 512
2019-06-23 17:44:45.158307: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216856c00 of size 256
2019-06-23 17:44:45.158309: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216856d00 of size 256
2019-06-23 17:44:45.158312: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216856e00 of size 1024
2019-06-23 17:44:45.158314: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216857200 of size 256
2019-06-23 17:44:45.158317: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216857300 of size 256
2019-06-23 17:44:45.158319: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216857400 of size 512
2019-06-23 17:44:45.158322: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216857600 of size 1024
2019-06-23 17:44:45.158324: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216857a00 of size 256
2019-06-23 17:44:45.158327: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216857b00 of size 256
2019-06-23 17:44:45.158329: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216857c00 of size 1024
2019-06-23 17:44:45.158332: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216858000 of size 256
2019-06-23 17:44:45.158334: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216858100 of size 256
2019-06-23 17:44:45.158337: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216858200 of size 512
2019-06-23 17:44:45.158340: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216858400 of size 256
2019-06-23 17:44:45.158342: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216858500 of size 8192
2019-06-23 17:44:45.158345: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021685a500 of size 4096
2019-06-23 17:44:45.158347: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021685b500 of size 256
2019-06-23 17:44:45.158350: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021685b600 of size 256
2019-06-23 17:44:45.158353: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021685b700 of size 256
2019-06-23 17:44:45.158355: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021685b800 of size 1024
2019-06-23 17:44:45.158358: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021685bc00 of size 2048
2019-06-23 17:44:45.158361: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021685c400 of size 256
2019-06-23 17:44:45.158363: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021685c500 of size 256
2019-06-23 17:44:45.158366: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021685c600 of size 256
2019-06-23 17:44:45.158368: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021685c700 of size 256
2019-06-23 17:44:45.158371: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021685c800 of size 256
2019-06-23 17:44:45.158373: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021685c900 of size 256
2019-06-23 17:44:45.158376: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021685ca00 of size 1024
2019-06-23 17:44:45.158378: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021685ce00 of size 256
2019-06-23 17:44:45.158381: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021685cf00 of size 256
2019-06-23 17:44:45.158384: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021685d000 of size 256
2019-06-23 17:44:45.158386: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021685d100 of size 256
2019-06-23 17:44:45.158389: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021685d200 of size 256
2019-06-23 17:44:45.158391: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021685d300 of size 256
2019-06-23 17:44:45.158394: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021685d400 of size 256
2019-06-23 17:44:45.158397: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021685d500 of size 256
2019-06-23 17:44:45.158399: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021685d600 of size 15104
2019-06-23 17:44:45.158402: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216861100 of size 256
2019-06-23 17:44:45.158404: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216861200 of size 256
2019-06-23 17:44:45.158407: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216861300 of size 256
2019-06-23 17:44:45.158410: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216861400 of size 256
2019-06-23 17:44:45.158412: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216861500 of size 256
2019-06-23 17:44:45.158415: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216861600 of size 256
2019-06-23 17:44:45.158417: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216861700 of size 256
2019-06-23 17:44:45.158420: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216861800 of size 256
2019-06-23 17:44:45.158422: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216861900 of size 256
2019-06-23 17:44:45.158425: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216861a00 of size 256
2019-06-23 17:44:45.158428: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216861b00 of size 256
2019-06-23 17:44:45.158430: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216861c00 of size 256
2019-06-23 17:44:45.158433: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216861d00 of size 256
2019-06-23 17:44:45.158435: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216861e00 of size 256
2019-06-23 17:44:45.158438: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216861f00 of size 256
2019-06-23 17:44:45.158441: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216862000 of size 256
2019-06-23 17:44:45.158443: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216862100 of size 15104
2019-06-23 17:44:45.158446: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216865c00 of size 15104
2019-06-23 17:44:45.158448: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216869700 of size 256
2019-06-23 17:44:45.158451: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216869800 of size 256
2019-06-23 17:44:45.158453: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216869900 of size 256
2019-06-23 17:44:45.158456: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216869a00 of size 256
2019-06-23 17:44:45.158459: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216869b00 of size 236032
2019-06-23 17:44:45.158462: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102168a3500 of size 512
2019-06-23 17:44:45.158464: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102168a3700 of size 512
2019-06-23 17:44:45.158467: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102168a3900 of size 262144
2019-06-23 17:44:45.158470: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102168e3900 of size 4718592
2019-06-23 17:44:45.158472: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10216d63900 of size 4718592
2019-06-23 17:44:45.158475: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102171e3900 of size 4718592
2019-06-23 17:44:45.158477: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10217663900 of size 9437184
2019-06-23 17:44:45.158480: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10217f63900 of size 9437184
2019-06-23 17:44:45.158483: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10218863900 of size 9437184
2019-06-23 17:44:45.158485: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10219163900 of size 9437184
2019-06-23 17:44:45.158488: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10219a63900 of size 9437184
2019-06-23 17:44:45.158491: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021a363900 of size 393216
2019-06-23 17:44:45.158493: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021a3c3900 of size 2359296
2019-06-23 17:44:45.158496: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021a603900 of size 1048576
2019-06-23 17:44:45.158498: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021a703900 of size 1441792
2019-06-23 17:44:45.158501: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021a863900 of size 2359296
2019-06-23 17:44:45.158504: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021aaa3900 of size 256
2019-06-23 17:44:45.158506: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021aaa3a00 of size 256
2019-06-23 17:44:45.158509: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021aaa3b00 of size 15104
2019-06-23 17:44:45.158511: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021aaa7600 of size 256
2019-06-23 17:44:45.158514: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021aaa7700 of size 256
2019-06-23 17:44:45.158517: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021aaa7800 of size 256
2019-06-23 17:44:45.158519: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021aaa7900 of size 256
2019-06-23 17:44:45.158522: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021aaa7a00 of size 256
2019-06-23 17:44:45.158524: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021aaa7b00 of size 256
2019-06-23 17:44:45.158527: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021aaa7c00 of size 256
2019-06-23 17:44:45.158530: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021aaa7d00 of size 256
2019-06-23 17:44:45.158532: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021aaa7e00 of size 256
2019-06-23 17:44:45.158535: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021aaa7f00 of size 256
2019-06-23 17:44:45.158538: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021aaa8000 of size 256
2019-06-23 17:44:45.158540: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021aaa8100 of size 45056
2019-06-23 17:44:45.158543: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021aab3100 of size 256
2019-06-23 17:44:45.158545: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021aab3200 of size 256
2019-06-23 17:44:45.158548: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021aab3300 of size 256
2019-06-23 17:44:45.158551: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021aab3400 of size 29952
2019-06-23 17:44:45.158553: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021aaba900 of size 256
2019-06-23 17:44:45.158556: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021aabaa00 of size 15104
2019-06-23 17:44:45.158558: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021aabe500 of size 256
2019-06-23 17:44:45.158561: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021aabe600 of size 256
2019-06-23 17:44:45.158564: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021aabe700 of size 15104
2019-06-23 17:44:45.158566: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021aac2200 of size 256
2019-06-23 17:44:45.158569: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021aac2300 of size 256
2019-06-23 17:44:45.158572: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021aac2400 of size 15104
2019-06-23 17:44:45.158574: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021aac5f00 of size 256
2019-06-23 17:44:45.158577: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021aac6000 of size 256
2019-06-23 17:44:45.158579: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021aac6100 of size 256
2019-06-23 17:44:45.158582: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021aac6200 of size 256
2019-06-23 17:44:45.158584: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021aac6300 of size 256
2019-06-23 17:44:45.158587: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021aac6400 of size 359424
2019-06-23 17:44:45.158590: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021ab1e000 of size 256
2019-06-23 17:44:45.158592: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021ce31d00 of size 359424
2019-06-23 17:44:45.158595: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021d52dc00 of size 1024
2019-06-23 17:44:45.158598: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021d52e000 of size 1280
2019-06-23 17:44:45.158600: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021d52e500 of size 256
2019-06-23 17:44:45.158603: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021d52e600 of size 393216
2019-06-23 17:44:45.158605: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021d58e600 of size 2359296
2019-06-23 17:44:45.158608: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021d7ce600 of size 2359296
2019-06-23 17:44:45.158611: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021da0e600 of size 131072
2019-06-23 17:44:45.158613: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021da2e600 of size 131072
2019-06-23 17:44:45.158616: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021da4e600 of size 131072
2019-06-23 17:44:45.158619: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1021da73d00 of size 256
2019-06-23 17:44:45.158621: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10220676a00 of size 15104
2019-06-23 17:44:45.158624: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1022241d300 of size 490733568
2019-06-23 17:44:45.158627: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1023f81d300 of size 490733568
2019-06-23 17:44:45.158630: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1025cc1d300 of size 490733568
2019-06-23 17:44:45.158632: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1027a01d300 of size 245366784
2019-06-23 17:44:45.158635: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102982c0e00 of size 15104
2019-06-23 17:44:45.158638: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102983b2500 of size 359424
2019-06-23 17:44:45.158641: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1029840a100 of size 981467136
2019-06-23 17:44:45.158643: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x102d2c0a100 of size 981467136
2019-06-23 17:44:45.158646: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1030d40a100 of size 981467136
2019-06-23 17:44:45.158649: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x10347c0a100 of size 981467136
2019-06-23 17:44:45.158651: I tensorflow/core/common_runtime/bfc_allocator.cc:661] Chunk at 0x1038240a100 of size 1348640512
2019-06-23 17:44:45.158657: I tensorflow/core/common_runtime/bfc_allocator.cc:670] Free at 0x1021ab1e100 of size 36781056
2019-06-23 17:44:45.158660: I tensorflow/core/common_runtime/bfc_allocator.cc:670] Free at 0x1021ce89900 of size 6963968
2019-06-23 17:44:45.158662: I tensorflow/core/common_runtime/bfc_allocator.cc:670] Free at 0x1021da6e600 of size 22272
2019-06-23 17:44:45.158665: I tensorflow/core/common_runtime/bfc_allocator.cc:670] Free at 0x1021da73e00 of size 46148608
2019-06-23 17:44:45.158668: I tensorflow/core/common_runtime/bfc_allocator.cc:670] Free at 0x1022067a500 of size 31075840
2019-06-23 17:44:45.158670: I tensorflow/core/common_runtime/bfc_allocator.cc:670] Free at 0x10288a1d300 of size 260717312
2019-06-23 17:44:45.158673: I tensorflow/core/common_runtime/bfc_allocator.cc:670] Free at 0x102982c4900 of size 973824
2019-06-23 17:44:45.158675: I tensorflow/core/common_runtime/bfc_allocator.cc:676] Summary of in-use Chunks by size:
2019-06-23 17:44:45.158680: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 181 Chunks of size 256 totalling 45.2KiB
2019-06-23 17:44:45.158684: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 63 Chunks of size 512 totalling 31.5KiB
2019-06-23 17:44:45.158687: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 47 Chunks of size 1024 totalling 47.0KiB
2019-06-23 17:44:45.158690: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 2 Chunks of size 1280 totalling 2.5KiB
2019-06-23 17:44:45.158693: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 1536 totalling 1.5KiB
2019-06-23 17:44:45.158696: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 21 Chunks of size 2048 totalling 42.0KiB
2019-06-23 17:44:45.158700: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 3 Chunks of size 4096 totalling 12.0KiB
2019-06-23 17:44:45.158703: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 3 Chunks of size 6912 totalling 20.2KiB
2019-06-23 17:44:45.158706: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 3 Chunks of size 8192 totalling 24.0KiB
2019-06-23 17:44:45.158710: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 9 Chunks of size 15104 totalling 132.8KiB
2019-06-23 17:44:45.158713: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 29952 totalling 29.2KiB
2019-06-23 17:44:45.158716: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 32768 totalling 32.0KiB
2019-06-23 17:44:45.158719: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 45056 totalling 44.0KiB
2019-06-23 17:44:45.158723: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 16 Chunks of size 131072 totalling 2.00MiB
2019-06-23 17:44:45.158726: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 3 Chunks of size 147456 totalling 432.0KiB
2019-06-23 17:44:45.158729: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 225280 totalling 220.0KiB
2019-06-23 17:44:45.158732: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 236032 totalling 230.5KiB
2019-06-23 17:44:45.158750: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 245760 totalling 240.0KiB
2019-06-23 17:44:45.158753: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 2 Chunks of size 262144 totalling 512.0KiB
2019-06-23 17:44:45.158756: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 2 Chunks of size 294912 totalling 576.0KiB
2019-06-23 17:44:45.158759: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 3 Chunks of size 359424 totalling 1.03MiB
2019-06-23 17:44:45.158763: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 6 Chunks of size 393216 totalling 2.25MiB
2019-06-23 17:44:45.158766: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 524800 totalling 512.5KiB
2019-06-23 17:44:45.158769: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 3 Chunks of size 589824 totalling 1.69MiB
2019-06-23 17:44:45.158773: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 3 Chunks of size 1048576 totalling 3.00MiB
2019-06-23 17:44:45.158776: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 9 Chunks of size 1179648 totalling 10.12MiB
2019-06-23 17:44:45.158779: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 2 Chunks of size 1180160 totalling 2.25MiB
2019-06-23 17:44:45.158782: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 3 Chunks of size 1180672 totalling 3.38MiB
2019-06-23 17:44:45.158786: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 3 Chunks of size 1441792 totalling 4.12MiB
2019-06-23 17:44:45.158789: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 1703936 totalling 1.62MiB
2019-06-23 17:44:45.158793: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 17 Chunks of size 2359296 totalling 38.25MiB
2019-06-23 17:44:45.158796: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 2360320 totalling 2.25MiB
2019-06-23 17:44:45.158799: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 2622464 totalling 2.50MiB
2019-06-23 17:44:45.158802: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 8 Chunks of size 4718592 totalling 36.00MiB
2019-06-23 17:44:45.158805: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 4719104 totalling 4.50MiB
2019-06-23 17:44:45.158809: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 15 Chunks of size 9437184 totalling 135.00MiB
2019-06-23 17:44:45.158812: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 245366784 totalling 234.00MiB
2019-06-23 17:44:45.158815: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 3 Chunks of size 490733568 totalling 1.37GiB
2019-06-23 17:44:45.158819: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 4 Chunks of size 981467136 totalling 3.66GiB
2019-06-23 17:44:45.158822: I tensorflow/core/common_runtime/bfc_allocator.cc:679] 1 Chunks of size 1348640512 totalling 1.26GiB
2019-06-23 17:44:45.158825: I tensorflow/core/common_runtime/bfc_allocator.cc:683] Sum Total of in-use chunks: 6.76GiB
2019-06-23 17:44:45.158831: I tensorflow/core/common_runtime/bfc_allocator.cc:685] Stats:
Limit: 7640137728
InUse: 7257454848
MaxInUse: 7257454848
NumAllocs: 1136
MaxAllocSize: 5274509056
2019-06-23 17:44:45.158849: W tensorflow/core/common_runtime/bfc_allocator.cc:277] ****************************__*******************************************************************xxx
2019-06-23 17:44:45.158862: W tensorflow/core/framework/op_kernel.cc:1192] Resource exhausted: OOM when allocating tensor with shape[3744,256,17,17]
2019-06-23 17:44:45.159916: W tensorflow/core/kernels/queue_base.cc:295] _0_Queues/fifo_queue: Skipping cancelled enqueue attempt with queue not closed
Traceback (most recent call last):
File "/home/strong/git_workspace/MonoGRNet/train.py", line 79, in <module>
tf.app.run()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "/home/strong/git_workspace/MonoGRNet/train.py", line 75, in main
train.do_training(hypes)
File "/home/strong/git_workspace/MonoGRNet/include/tensorvision/train.py", line 398, in do_training
run_training(hypes, modules, tv_graph, tv_sess)
File "/home/strong/git_workspace/MonoGRNet/include/tensorvision/train.py", line 236, in run_training
feed_dict=feed_dict)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 889, in run
run_metadata_ptr)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1120, in _run
feed_dict_tensor, options, run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1317, in _do_run
options, run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1336, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.ResourceExhaustedErrorException in thread Thread-2:
Traceback (most recent call last):
File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
self.run()
File "/usr/lib/python2.7/threading.py", line 754, in run
self.__target(*self.__args, **self.__kwargs)
File "/home/strong/git_workspace/MonoGRNet/hypes/../inputs/kitti_input.py", line 363, in thread_loop
sess.run(enqueue_op, feed_dict=make_feed(d))
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 889, in run
run_metadata_ptr)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1120, in _run
feed_dict_tensor, options, run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1317, in _do_run
options, run_metadata)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1336, in _do_call
raise type(e)(node_def, op, message)
CancelledError: Enqueue operation was cancelled
[[Node: fifo_queue_enqueue = QueueEnqueueV2[Tcomponents=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](Queues/fifo_queue, _arg_Placeholder_2_0_0, _arg_Placeholder_3_0_1, _arg_Placeholder_4_0_2, _arg_Placeholder_5_0_3, _arg_Placeholder_6_0_4, _arg_Placeholder_7_0_5, _arg_Placeholder_8_0_6)]]
Caused by op u'fifo_queue_enqueue', defined at:
File "/home/strong/git_workspace/MonoGRNet/train.py", line 79, in <module>
tf.app.run()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "/home/strong/git_workspace/MonoGRNet/train.py", line 75, in main
train.do_training(hypes)
File "/home/strong/git_workspace/MonoGRNet/include/tensorvision/train.py", line 395, in do_training
modules['input'].start_enqueuing_threads(hypes, queue, 'train', sess)
File "/home/strong/git_workspace/MonoGRNet/hypes/../inputs/kitti_input.py", line 350, in start_enqueuing_threads
enqueue_op = q.enqueue((x_in, confs_in, boxes_in, mask_in, calib_in, calib_pinv_in, xy_scale_in))
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/data_flow_ops.py", line 327, in enqueue
self._queue_ref, vals, name=scope)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_data_flow_ops.py", line 2777, in _queue_enqueue_v2
timeout_ms=timeout_ms, name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2956, in create_op
op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1470, in __init__
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access
CancelledError (see above for traceback): Enqueue operation was cancelled
[[Node: fifo_queue_enqueue = QueueEnqueueV2[Tcomponents=[DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT, DT_FLOAT], timeout_ms=-1, _device="/job:localhost/replica:0/task:0/device:CPU:0"](Queues/fifo_queue, _arg_Placeholder_2_0_0, _arg_Placeholder_3_0_1, _arg_Placeholder_4_0_2, _arg_Placeholder_5_0_3, _arg_Placeholder_6_0_4, _arg_Placeholder_7_0_5, _arg_Placeholder_8_0_6)]]
: OOM when allocating tensor with shape[3744,256,17,17]
[[Node: decoder/Conv2D = Conv2D[T=DT_FLOAT, data_format="NHWC", padding="SAME", strides=[1, 2, 2, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](decoder/add_9, decoder/refine_filt_1/read)]]
[[Node: Loss/mul_11/_267 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_2260_Loss/mul_11", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Caused by op u'decoder/Conv2D', defined at:
File "/home/strong/git_workspace/MonoGRNet/train.py", line 79, in <module>
tf.app.run()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
File "/home/strong/git_workspace/MonoGRNet/train.py", line 75, in main
train.do_training(hypes)
File "/home/strong/git_workspace/MonoGRNet/include/tensorvision/train.py", line 368, in do_training
tv_graph = core.build_training_graph(hypes, queue, modules)
File "/home/strong/git_workspace/MonoGRNet/include/tensorvision/core.py", line 88, in build_training_graph
decoded_logits = objective.decoder(hypes, logits, labels, train=True)
File "/home/strong/git_workspace/MonoGRNet/hypes/../decoder/fastBox.py", line 609, in decoder
dlogits['delta_global_corners'] = _build_refine_layer(hyp, logits, dlogits['pred_bbox_proj'])
File "/home/strong/git_workspace/MonoGRNet/hypes/../decoder/fastBox.py", line 513, in _build_refine_layer
conv_1 = tf.nn.conv2d(refine_feat, filt_1, [1, 2, 2, 1], padding='SAME')
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_nn_ops.py", line 631, in conv2d
data_format=data_format, name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2956, in create_op
op_def=op_def)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1470, in __init__
self._traceback = self._graph._extract_stack() # pylint: disable=protected-access
ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[3744,256,17,17]
[[Node: decoder/Conv2D = Conv2D[T=DT_FLOAT, data_format="NHWC", padding="SAME", strides=[1, 2, 2, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](decoder/add_9, decoder/refine_filt_1/read)]]
[[Node: Loss/mul_11/_267 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_2260_Loss/mul_11", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Process finished with exit code 1
2. /home/strong/git_workspace/MonoGRNet/hypes/kittiBox.json
“batch_size”: 8 ===>>> “batch_size”: 4 ===>>> “batch_size”: 2
{
"model": {
"input_file": "../inputs/kitti_input.py",
"architecture_file" : "../encoder/vgg.py",
"objective_file" : "../decoder/fastBox.py",
"optimizer_file" : "../optimizer/generic_optimizer.py",
"evaluator_file" : "../evals/kitti_eval.py"
},
"data": {
"train_file": "KittiBox/train.txt",
"val_file": "KittiBox/val.txt",
"truncate_data": false,
"eval_cmd": "../submodules/KittiEvaluation/./evaluate_object",
"label_dir": "KittiBox/training/label_2"
},
"logging": {
"display_iter": 500,
"eval_iter": 4000,
"write_iter": 800,
"save_iter": 4000,
"max_to_keep": 10,
"image_iter": 150000
},
"arch": {
"deep_feat": "pool5"
},
"path": ["../include"],
"solver": {
"opt": "Adam",
"use_jitter": true,
"rnd_seed": 1,
"epsilon": 0.00001,
"learning_rate": 1e-4,
"learning_rate_step": null,
"hungarian_iou": 0.35,
"weights": "",
"head_weights": [1.0, 0.1],
"max_steps": 96000
},
"pretrained": "data/model_3D",
"use_rezoom": true,
"augment_level": -1,
"use_mask": true,
"biggest_box_px": 10000,
"early_feat_channels": 256,
"rezoom_change_loss": "center",
"rezoom_w_coords": [-0.25, 0, 0.25],
"rezoom_h_coords": [-0.25, 0, 0.25],
"image_height": 384,
"image_width": 1248,
"grid_width": 39,
"grid_height": 12,
"batch_size": 8,
"tau": 0.35,
"scale_down": 0.1,
"focus_size": 1.8,
"region_size": 32,
"clip_norm": 1.0,
"num_inner_channel": 512,
"num_classes": 2,
"wd": 5e-4,
"detect_truck": false
}
3.