2018-06-23 21:08:51.461822: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fdfb3d6fe00 of size 201326592
2018-06-23 21:08:51.461825: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fdfbfd6fe00 of size 268435456
2018-06-23 21:08:51.461829: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fdfcfd6fe00 of size 134217728
2018-06-23 21:08:51.461832: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fdfd7d6fe00 of size 134217728
2018-06-23 21:08:51.461835: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fdfdfd6fe00 of size 134217728
2018-06-23 21:08:51.461839: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fdfe7d6fe00 of size 134217728
2018-06-23 21:08:51.461842: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fdfefd6fe00 of size 268435456
2018-06-23 21:08:51.461845: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fdfffd6fe00 of size 268435456
2018-06-23 21:08:51.461849: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Free at 0x7fe00fd6fe00 of size 134217728
2018-06-23 21:08:51.461852: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe017d6fe00 of size 536870912
2018-06-23 21:08:51.461855: I tensorflow/core/common_runtime/bfc_allocator.cc:665] Chunk at 0x7fe037d6fe00 of size 337003008
2018-06-23 21:08:51.461859: I tensorflow/core/common_runtime/bfc_allocator.cc:671] Summary of in-use Chunks by size:
2018-06-23 21:08:51.461863: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 426 Chunks of size 256 totalling 106.5KiB
2018-06-23 21:08:51.461868: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 70 Chunks of size 512 totalling 35.0KiB
2018-06-23 21:08:51.461871: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 104 Chunks of size 1024 totalling 104.0KiB
2018-06-23 21:08:51.461875: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 5 Chunks of size 1280 totalling 6.2KiB
2018-06-23 21:08:51.461879: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 206 Chunks of size 2048 totalling 412.0KiB
2018-06-23 21:08:51.461883: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 3 Chunks of size 6912 totalling 20.2KiB
2018-06-23 21:08:51.461887: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 8448 totalling 8.2KiB
2018-06-23 21:08:51.461891: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 6 Chunks of size 147456 totalling 864.0KiB
2018-06-23 21:08:51.461895: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 294144 totalling 287.2KiB
2018-06-23 21:08:51.461898: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 7 Chunks of size 294912 totalling 1.97MiB
2018-06-23 21:08:51.461902: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 589056 totalling 575.2KiB
2018-06-23 21:08:51.461906: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 6 Chunks of size 589824 totalling 3.38MiB
2018-06-23 21:08:51.461910: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 1178112 totalling 1.12MiB
2018-06-23 21:08:51.461913: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 6 Chunks of size 1179648 totalling 6.75MiB
2018-06-23 21:08:51.461917: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 1180928 totalling 1.13MiB
2018-06-23 21:08:51.461921: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 2097152 totalling 2.00MiB
2018-06-23 21:08:51.461925: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 2357760 totalling 2.25MiB
2018-06-23 21:08:51.461929: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 12 Chunks of size 2359296 totalling 27.00MiB
2018-06-23 21:08:51.461932: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 4715520 totalling 4.50MiB
2018-06-23 21:08:51.461936: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 7 Chunks of size 4718592 totalling 31.50MiB
2018-06-23 21:08:51.461940: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 5425152 totalling 5.17MiB
2018-06-23 21:08:51.461944: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 11 Chunks of size 8388608 totalling 88.00MiB
2018-06-23 21:08:51.461948: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 41 Chunks of size 9437184 totalling 369.00MiB
2018-06-23 21:08:51.461951: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 12582912 totalling 12.00MiB
2018-06-23 21:08:51.461955: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 2 Chunks of size 15728640 totalling 30.00MiB
2018-06-23 21:08:51.461959: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 16777216 totalling 16.00MiB
2018-06-23 21:08:51.461963: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 20971520 totalling 20.00MiB
2018-06-23 21:08:51.461967: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 17 Chunks of size 33554432 totalling 544.00MiB
2018-06-23 21:08:51.461971: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 40226560 totalling 38.36MiB
2018-06-23 21:08:51.461975: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 62390272 totalling 59.50MiB
2018-06-23 21:08:51.461978: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 17 Chunks of size 67108864 totalling 1.06GiB
2018-06-23 21:08:51.461982: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 100663296 totalling 96.00MiB
2018-06-23 21:08:51.461986: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 11 Chunks of size 134217728 totalling 1.38GiB
2018-06-23 21:08:51.461990: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 201326592 totalling 192.00MiB
2018-06-23 21:08:51.461993: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 9 Chunks of size 268435456 totalling 2.25GiB
2018-06-23 21:08:51.461997: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 337003008 totalling 321.39MiB
2018-06-23 21:08:51.462001: I tensorflow/core/common_runtime/bfc_allocator.cc:674] 1 Chunks of size 536870912 totalling 512.00MiB
2018-06-23 21:08:51.462005: I tensorflow/core/common_runtime/bfc_allocator.cc:678] Sum Total of in-use chunks: 7.02GiB
2018-06-23 21:08:51.462011: I tensorflow/core/common_runtime/bfc_allocator.cc:680] Stats:
Limit: 7816953856
InUse: 7536513280
MaxInUse: 7536514048
NumAllocs: 1515
MaxAllocSize: 2415919104
2018-06-23 21:08:51.462043: W tensorflow/core/common_runtime/bfc_allocator.cc:279] ****************************************************************************************************
2018-06-23 21:08:51.462057: W tensorflow/core/framework/op_kernel.cc:1273]
OP_REQUIRES failed at transpose_op.cc:199 :
Resource exhausted: OOM when allocating tensor with shape[16,256,256,64] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
通常是GPU显存不够了，改小一点batch_size吧