版权声明：本文为博主原创文章，未经博主允许不得转载。 https://blog.csdn.net/u012435142/article/details/81811487
Faster-RCNN_TF实验及错误解决
1 说明
这是我前一段时间学习Faster-RCNN_TF的一些笔记，包括程序运行出错的解决过程，和自己训练测试过程。
Faster-RCNN_TF github连接：
https://github.com/smallcorgi/Faster-RCNN_TF
Faster-RCNN_TF代码作者已经2年没有维护，并不完全兼容现在的新tensorflow版本，所以需要做一定的修改工作，具体见下面的错误解决内容。
我的环境配置：
cuda 9.0
tensorflow 1.8
显卡nvidia 1060 6G
python 2.7
该代码包含一部分c语言文件，需要编译后才能正常使用；
在VGG预训练模型基础上，大概6个小时可以完成VOC2007数据集的训练，一共70000次迭代，测试一张图像0.15~0.2s。
2 make及运行demo出错
按照github上作者写的说明，在Build the Cython modules步骤，执行以下内容时：
cd $FRCN_ROOT/lib
make我在执行完make后，有一堆warning，然后运行demo:python ./tools/demo.py --model pretrain_model/VGGnet_fast_rcnn_iter_70000.ckpt
出错:
tensorflow.python.framework.errors_impl.NotFoundError: /media/han/E/mWork/mCode/Faster-RCNN_TF/tools/../lib/roi_pooling_layer/roi_pooling.so: undefined symbol: _ZTIN10tensorflow8OpKernelE
以下是make和运行demo执行结果是：
(py27) han@MS:/media/han/E/mWork/mCode/Faster-RCNN_TF/lib$ make
python setup.py build_ext --inplace
running build_ext
skipping 'utils/bbox.c' Cython extension (up-to-date)
skipping 'utils/nms.c' Cython extension (up-to-date)
skipping 'nms/cpu_nms.c' Cython extension (up-to-date)
skipping 'nms/gpu_nms.cpp' Cython extension (up-to-date)
rm -rf build
bash make.sh
/home/han/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/include/unsupported/Eigen/CXX11/../../../Eigen/src/Core/MathFunctions.h(1265): warning: calling a constexpr __host__ function("real") from a __host__ __device__ function("abs") is not allowed. The experimental flag '--expt-relaxed-constexpr' can be used to allow this.
/home/han/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/include/unsupported/Eigen/CXX11/../../../Eigen/src/Core/MathFunctions.h(1265): warning: calling a constexpr __host__ function("imag") from a __host__ __device__ function("abs") is not allowed. The experimental flag '--expt-relaxed-constexpr' can be used to allow this.
/home/han/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/include/unsupported/Eigen/CXX11/../../../Eigen/src/Core/MathFunctions.h(1265): warning: calling a constexpr __host__ function from a __host__ __device__ function is not allowed. The experimental flag '--expt-relaxed-constexpr' can be used to allow this.
/home/han/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/include/unsupported/Eigen/CXX11/../../../Eigen/src/Core/MathFunctions.h(1265): warning: calling a constexpr __host__ function from a __host__ __device__ function is not allowed. The experimental flag '--expt-relaxed-constexpr' can be used to allow this.
/home/han/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/include/unsupported/Eigen/CXX11/../../../Eigen/src/Core/MathFunctions.h(1270): warning: calling a constexpr __host__ function("real") from a __host__ __device__ function("abs") is not allowed. The experimental flag '--expt-relaxed-constexpr' can be used to allow this.
/home/han/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/include/unsupported/Eigen/CXX11/../../../Eigen/src/Core/MathFunctions.h(1270): warning: calling a constexpr __host__ function("imag") from a __host__ __device__ function("abs") is not allowed. The experimental flag '--expt-relaxed-constexpr' can be used to allow this.
/home/han/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/include/unsupported/Eigen/CXX11/../../../Eigen/src/Core/MathFunctions.h(1270): warning: calling a constexpr __host__ function from a __host__ __device__ function is not allowed. The experimental flag '--expt-relaxed-constexpr' can be used to allow this.
/home/han/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/include/unsupported/Eigen/CXX11/../../../Eigen/src/Core/MathFunctions.h(1270): warning: calling a constexpr __host__ function from a __host__ __device__ function is not allowed. The experimental flag '--expt-relaxed-constexpr' can be used to allow this.
/home/han/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/include/unsupported/Eigen/CXX11/../../../Eigen/src/Core/MathFunctions.h(1295): warning: calling a constexpr __host__ function("real") from a __host__ __device__ function("exp") is not allowed. The experimental flag '--expt-relaxed-constexpr' can be used to allow this.
/home/han/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/include/unsupported/Eigen/CXX11/../../../Eigen/src/Core/MathFunctions.h(1296): warning: calling a constexpr __host__ function("imag") from a __host__ __device__ function("exp") is not allowed. The experimental flag '--expt-relaxed-constexpr' can be used to allow this.
/home/han/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/include/unsupported/Eigen/CXX11/../../../Eigen/src/Core/MathFunctions.h(1297): warning: calling a constexpr __host__ function("imag") from a __host__ __device__ function("exp") is not allowed. The experimental flag '--expt-relaxed-constexpr' can be used to allow this.
/home/han/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/include/unsupported/Eigen/CXX11/../../../Eigen/src/Core/MathFunctions.h(1295): warning: calling a constexpr __host__ function from a __host__ __device__ function is not allowed. The experimental flag '--expt-relaxed-constexpr' can be used to allow this.
/home/han/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/include/unsupported/Eigen/CXX11/../../../Eigen/src/Core/MathFunctions.h(1296): warning: calling a constexpr __host__ function from a __host__ __device__ function is not allowed. The experimental flag '--expt-relaxed-constexpr' can be used to allow this.
/home/han/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/include/unsupported/Eigen/CXX11/../../../Eigen/src/Core/MathFunctions.h(1297): warning: calling a constexpr __host__ function from a __host__ __device__ function is not allowed. The experimental flag '--expt-relaxed-constexpr' can be used to allow this.
/home/han/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/include/unsupported/Eigen/CXX11/../../../Eigen/src/Core/MathFunctions.h(1298): warning: calling a constexpr __host__ function from a __host__ __device__ function is not allowed. The experimental flag '--expt-relaxed-constexpr' can be used to allow this.
/home/han/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/include/unsupported/Eigen/CXX11/../../../Eigen/src/Core/MathFunctions.h(1303): warning: calling a constexpr __host__ function("real") from a __host__ __device__ function("exp") is not allowed. The experimental flag '--expt-relaxed-constexpr' can be used to allow this.
/home/han/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/include/unsupported/Eigen/CXX11/../../../Eigen/src/Core/MathFunctions.h(1304): warning: calling a constexpr __host__ function("imag") from a __host__ __device__ function("exp") is not allowed. The experimental flag '--expt-relaxed-constexpr' can be used to allow this.
/home/han/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/include/unsupported/Eigen/CXX11/../../../Eigen/src/Core/MathFunctions.h(1305): warning: calling a constexpr __host__ function("imag") from a __host__ __device__ function("exp") is not allowed. The experimental flag '--expt-relaxed-constexpr' can be used to allow this.
/home/han/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/include/unsupported/Eigen/CXX11/../../../Eigen/src/Core/MathFunctions.h(1303): warning: calling a constexpr __host__ function from a __host__ __device__ function is not allowed. The experimental flag '--expt-relaxed-constexpr' can be used to allow this.
/home/han/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/include/unsupported/Eigen/CXX11/../../../Eigen/src/Core/MathFunctions.h(1304): warning: calling a constexpr __host__ function from a __host__ __device__ function is not allowed. The experimental flag '--expt-relaxed-constexpr' can be used to allow this.
/home/han/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/include/unsupported/Eigen/CXX11/../../../Eigen/src/Core/MathFunctions.h(1305): warning: calling a constexpr __host__ function from a __host__ __device__ function is not allowed. The experimental flag '--expt-relaxed-constexpr' can be used to allow this.
/home/han/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/include/unsupported/Eigen/CXX11/../../../Eigen/src/Core/MathFunctions.h(1306): warning: calling a constexpr __host__ function from a __host__ __device__ function is not allowed. The experimental flag '--expt-relaxed-constexpr' can be used to allow this.
/home/han/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/include/unsupported/Eigen/CXX11/src/Tensor/TensorRandom.h(133): warning: calling a constexpr __host__ function from a __host__ __device__ function is not allowed. The experimental flag '--expt-relaxed-constexpr' can be used to allow this.
/home/han/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/include/unsupported/Eigen/CXX11/src/Tensor/TensorRandom.h(138): warning: calling a constexpr __host__ function from a __host__ __device__ function is not allowed. The experimental flag '--expt-relaxed-constexpr' can be used to allow this.
/home/han/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/include/unsupported/Eigen/CXX11/src/Tensor/TensorRandom.h(208): warning: calling a constexpr __host__ function from a __host__ __device__ function is not allowed. The experimental flag '--expt-relaxed-constexpr' can be used to allow this.
/home/han/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/include/unsupported/Eigen/CXX11/src/Tensor/TensorRandom.h(213): warning: calling a constexpr __host__ function from a __host__ __device__ function is not allowed. The experimental flag '--expt-relaxed-constexpr' can be used to allow this.
(py27) han@MS:/media/han/E/mWork/mCode/Faster-RCNN_TF/lib$ cd ../
(py27) han@MS:/media/han/E/mWork/mCode/Faster-RCNN_TF$ python ./tools/demo.py --model pretrain_model/VGGnet_fast_rcnn_iter_70000.ckpt
Traceback (most recent call last):
File "./tools/demo.py", line 11, in <module>
from networks.factory import get_network
File "/media/han/E/mWork/mCode/Faster-RCNN_TF/tools/../lib/networks/__init__.py", line 8, in <module>
from .VGGnet_train import VGGnet_train
File "/media/han/E/mWork/mCode/Faster-RCNN_TF/tools/../lib/networks/VGGnet_train.py", line 2, in <module>
from networks.network import Network
File "/media/han/E/mWork/mCode/Faster-RCNN_TF/tools/../lib/networks/network.py", line 3, in <module>
import roi_pooling_layer.roi_pooling_op as roi_pool_op
File "/media/han/E/mWork/mCode/Faster-RCNN_TF/tools/../lib/roi_pooling_layer/roi_pooling_op.py", line 5, in <module>
_roi_pooling_module = tf.load_op_library(filename)
File "/home/han/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/python/framework/load_library.py", line 56, in load_op_library
lib_handle = py_tf.TF_LoadLibrary(library_filename)
tensorflow.python.framework.errors_impl.NotFoundError: /media/han/E/mWork/mCode/Faster-RCNN_TF/tools/../lib/roi_pooling_layer/roi_pooling.so: undefined symbol: _ZTIN10tensorflow8OpKernelE解决方法
参考网上教程
https://blog.csdn.net/zhongxinleishi/article/details/78972944
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/docs_src/extend/adding_an_op.md
http://blog.sina.com.cn/s/blog_5d5a19fe0102xf02.html
应该是tensorflow版本兼容性问题，我的tensorflow版本是1.8，cuda 9.0 ,cudnn 7.0.5.我修改了make.sh文件，参考其他教程，添加下面的内容(添加位置参考下面的修改后的make.sh)：
TF_LIB=$(python -c 'import tensorflow as tf; print(tf.sysconfig.get_lib())')
TF_CFLAGS=( $(python -c 'import tensorflow as tf; print(" ".join(tf.sysconfig.get_compile_flags()))') )
TF_LFLAGS=( $(python -c 'import tensorflow as tf; print(" ".join(tf.sysconfig.get_link_flags()))') )
-D_GLIBCXX_USE_CXX11ABI=0
${TF_CFLAGS[@]} ${TF_LFLAGS[@]} -O2 \修改后的完整版make.sh为：
TF_INC=$(python -c 'import tensorflow as tf; print(tf.sysconfig.get_include())')
TF_LIB=$(python -c 'import tensorflow as tf; print(tf.sysconfig.get_lib())')
TF_CFLAGS=( $(python -c 'import tensorflow as tf; print(" ".join(tf.sysconfig.get_compile_flags()))') )
TF_LFLAGS=( $(python -c 'import tensorflow as tf; print(" ".join(tf.sysconfig.get_link_flags()))') )
CUDA_PATH=/usr/local/cuda/
CXXFLAGS=''
if [[ "$OSTYPE" =~ ^darwin ]]; then
CXXFLAGS+='-undefined dynamic_lookup'
fi
cd roi_pooling_layer
if [ -d "$CUDA_PATH" ]; then
nvcc -std=c++11 -c -o roi_pooling_op.cu.o roi_pooling_op_gpu.cu.cc \
-I $TF_INC -D GOOGLE_CUDA=1 -x cu -Xcompiler -fPIC $CXXFLAGS \
-arch=sm_37 -L$TF_LIB -ltensorflow_framework\
-D_GLIBCXX_USE_CXX11ABI=0
g++ -std=c++11 -shared -o roi_pooling.so roi_pooling_op.cc \
-D_GLIBCXX_USE_CXX11_ABI=0\
roi_pooling_op.cu.o -I $TF_INC -D GOOGLE_CUDA=1 -fPIC $CXXFLAGS \
${TF_CFLAGS[@]} ${TF_LFLAGS[@]} -O2 \
-lcudart -L $CUDA_PATH/lib64
else
g++ -std=c++11 -shared -o roi_pooling.so roi_pooling_op.cc \
-I $TF_INC -fPIC $CXXFLAGS
fi
cd ..3 test_net.py测试网络并计算AP
测试VOC2007数据集中的test部分，运行代码如下：
python ./tools/test_net.py --device gpu --device_id 0 --weights model/VGGnet_fast_rcnn_iter_70000.ckpt --imdb voc_2007_test --cfg experiments/cfgs/faster_rcnn_end2end.yml --network VGGnet_test需要准备好数据集VOC2007
下载数据集保存到Faster-RCNN_TF/data/VOCdevkit2007
$VOCdevkit2007/
$VOCdevkit2007/VOC2007/ # 包括Annotations, ImageSets, JPEGImages等子文件夹新建文件夹如下，用来保存测试生成的结果，如果文件夹不存在，则会在测试结束时报错No such file。相关代码在
lib/datasets/psacal_voc.py中
$VOCdevkit2007/result/VOC2007/Main测试过程记录
# 省略之间的大部分内容
im_detect: 4945/4952 0.154s 0.001s
im_detect: 4946/4952 0.154s 0.001s
im_detect: 4947/4952 0.154s 0.001s
im_detect: 4948/4952 0.154s 0.001s
im_detect: 4949/4952 0.154s 0.001s
im_detect: 4950/4952 0.154s 0.001s
im_detect: 4951/4952 0.154s 0.001s
im_detect: 4952/4952 0.154s 0.001s
Evaluating detections
Writing aeroplane VOC results file
Writing bicycle VOC results file
Writing bird VOC results file
Writing boat VOC results file
Writing bottle VOC results file
Writing bus VOC results file
Writing car VOC results file
Writing cat VOC results file
Writing chair VOC results file
Writing cow VOC results file
Writing diningtable VOC results file
Writing dog VOC results file
Writing horse VOC results file
Writing motorbike VOC results file
Writing person VOC results file
Writing pottedplant VOC results file
Writing sheep VOC results file
Writing sofa VOC results file
Writing train VOC results file
Writing tvmonitor VOC results file
VOC07 metric? Yes
Reading annotation for 1/4952
Reading annotation for 101/4952
Reading annotation for 201/4952
Reading annotation for 301/4952
Reading annotation for 401/4952
Reading annotation for 501/4952
Reading annotation for 601/4952
Reading annotation for 701/4952
Reading annotation for 801/4952
Reading annotation for 901/4952
Reading annotation for 1001/4952
Reading annotation for 1101/4952
Reading annotation for 1201/4952
Reading annotation for 1301/4952
Reading annotation for 1401/4952
Reading annotation for 1501/4952
Reading annotation for 1601/4952
Reading annotation for 1701/4952
Reading annotation for 1801/4952
Reading annotation for 1901/4952
Reading annotation for 2001/4952
Reading annotation for 2101/4952
Reading annotation for 2201/4952
Reading annotation for 2301/4952
Reading annotation for 2401/4952
Reading annotation for 2501/4952
Reading annotation for 2601/4952
Reading annotation for 2701/4952
Reading annotation for 2801/4952
Reading annotation for 2901/4952
Reading annotation for 3001/4952
Reading annotation for 3101/4952
Reading annotation for 3201/4952
Reading annotation for 3301/4952
Reading annotation for 3401/4952
Reading annotation for 3501/4952
Reading annotation for 3601/4952
Reading annotation for 3701/4952
Reading annotation for 3801/4952
Reading annotation for 3901/4952
Reading annotation for 4001/4952
Reading annotation for 4101/4952
Reading annotation for 4201/4952
Reading annotation for 4301/4952
Reading annotation for 4401/4952
Reading annotation for 4501/4952
Reading annotation for 4601/4952
Reading annotation for 4701/4952
Reading annotation for 4801/4952
Reading annotation for 4901/4952
Saving cached annotations to /media/han/E/mWork/mCode/Faster-RCNN_TF/data/VOCdevkit2007/annotations_cache/annots.pkl
AP for aeroplane = 0.6980
AP for bicycle = 0.7866
AP for bird = 0.6652
AP for boat = 0.5798
AP for bottle = 0.4797
AP for bus = 0.7663
AP for car = 0.7984
AP for cat = 0.7842
AP for chair = 0.4755
AP for cow = 0.7353
AP for diningtable = 0.6464
AP for dog = 0.7980
AP for horse = 0.8046
AP for motorbike = 0.7386
AP for person = 0.7702
AP for pottedplant = 0.3747
AP for sheep = 0.6597
AP for sofa = 0.6533
AP for train = 0.7625
AP for tvmonitor = 0.6625
Mean AP = 0.6820
~~~~~~~~
Results:
0.698
0.787
0.665
0.580
0.480
0.766
0.798
0.784
0.476
0.735
0.646
0.798
0.805
0.739
0.770
0.375
0.660
0.653
0.762
0.662
0.682
~~~~~~~~
--------------------------------------------------------------
Results computed with the **unofficial** Python eval code.
Results should be very close to the official MATLAB eval code.
Recompute with `./tools/reval.py --matlab ...` for your paper.
-- Thanks, The Management
--------------------------------------------------------------4 自己训练VOC2007
./experiments/scripts/faster_rcnn_end2end.sh gpu 0 VGG16 pascal_voc训练记录
(py27) han@MS:/media/han/E/mWork/mCode/Faster-RCNN_TF$ ./experiments/scripts/faster_rcnn_end2end.sh gpu 0 VGG16 pascal_voc
+ set -e
+ export PYTHONUNBUFFERED=True
+ PYTHONUNBUFFERED=True
+ DEV=gpu
+ DEV_ID=0
+ NET=VGG16
+ DATASET=pascal_voc
+ array=($@)
+ len=4
+ EXTRA_ARGS=
+ EXTRA_ARGS_SLUG=
+ case $DATASET in
+ TRAIN_IMDB=voc_2007_trainval
+ TEST_IMDB=voc_2007_test
+ PT_DIR=pascal_voc
+ ITERS=70000
++ date +%Y-%m-%d_%H-%M-%S
+ LOG=experiments/logs/faster_rcnn_end2end_VGG16_.txt.2018-07-31_21-08-01
+ exec
++ tee -a experiments/logs/faster_rcnn_end2end_VGG16_.txt.2018-07-31_21-08-01
tee: experiments/logs/faster_rcnn_end2end_VGG16_.txt.2018-07-31_21-08-01: No such file or directory
+ echo Logging output to experiments/logs/faster_rcnn_end2end_VGG16_.txt.2018-07-31_21-08-01
Logging output to experiments/logs/faster_rcnn_end2end_VGG16_.txt.2018-07-31_21-08-01
+ python ./tools/train_net.py --device gpu --device_id 0 --weights data/pretrain_model/VGG_imagenet.npy --imdb voc_2007_trainval --iters 70000 --cfg experiments/cfgs/faster_rcnn_end2end.yml --network VGGnet_train
voc_2007_train
voc_2007_val
voc_2007_trainval
voc_2007_test
kitti_train
kitti_val
kitti_trainval
kitti_test
nthu_71
nthu_370
Called with args:
Namespace(cfg_file='experiments/cfgs/faster_rcnn_end2end.yml', device='gpu', device_id=0, imdb_name='voc_2007_trainval', max_iters=70000, network_name='VGGnet_train', pretrained_model='data/pretrain_model/VGG_imagenet.npy', randomize=False, set_cfgs=None, solver=None)
Using config:
{'DATA_DIR': '/media/han/E/mWork/mCode/Faster-RCNN_TF/data',
'DEDUP_BOXES': 0.0625,
'EPS': 1e-14,
'EXP_DIR': 'faster_rcnn_end2end',
'GPU_ID': 0,
'IS_MULTISCALE': False,
'MATLAB': 'matlab',
'MODELS_DIR': '/media/han/E/mWork/mCode/Faster-RCNN_TF/models/pascal_voc',
'PIXEL_MEANS': array([[[102.9801, 115.9465, 122.7717]]]),
'RNG_SEED': 3,
'ROOT_DIR': '/media/han/E/mWork/mCode/Faster-RCNN_TF',
'TEST': {'BBOX_REG': True,
'DEBUG_TIMELINE': False,
'HAS_RPN': True,
'MAX_SIZE': 1000,
'NMS': 0.3,
'PROPOSAL_METHOD': 'selective_search',
'RPN_MIN_SIZE': 16,
'RPN_NMS_THRESH': 0.7,
'RPN_POST_NMS_TOP_N': 300,
'RPN_PRE_NMS_TOP_N': 6000,
'SCALES': [600],
'SVM': False},
'TRAIN': {'ASPECT_GROUPING': True,
'BATCH_SIZE': 128,
'BBOX_INSIDE_WEIGHTS': [1.0, 1.0, 1.0, 1.0],
'BBOX_NORMALIZE_MEANS': [0.0, 0.0, 0.0, 0.0],
'BBOX_NORMALIZE_STDS': [0.1, 0.1, 0.2, 0.2],
'BBOX_NORMALIZE_TARGETS': True,
'BBOX_NORMALIZE_TARGETS_PRECOMPUTED': True,
'BBOX_REG': True,
'BBOX_THRESH': 0.5,
'BG_THRESH_HI': 0.5,
'BG_THRESH_LO': 0.0,
'DEBUG_TIMELINE': False,
'DISPLAY': 10,
'FG_FRACTION': 0.25,
'FG_THRESH': 0.5,
'GAMMA': 0.1,
'HAS_RPN': True,
'IMS_PER_BATCH': 1,
'LEARNING_RATE': 0.001,
'MAX_SIZE': 1000,
'MOMENTUM': 0.9,
'PROPOSAL_METHOD': 'gt',
'RPN_BATCHSIZE': 256,
'RPN_BBOX_INSIDE_WEIGHTS': [1.0, 1.0, 1.0, 1.0],
'RPN_CLOBBER_POSITIVES': False,
'RPN_FG_FRACTION': 0.5,
'RPN_MIN_SIZE': 16,
'RPN_NEGATIVE_OVERLAP': 0.3,
'RPN_NMS_THRESH': 0.7,
'RPN_POSITIVE_OVERLAP': 0.7,
'RPN_POSITIVE_WEIGHT': -1.0,
'RPN_POST_NMS_TOP_N': 2000,
'RPN_PRE_NMS_TOP_N': 12000,
'SCALES': [600],
'SNAPSHOT_INFIX': '',
'SNAPSHOT_ITERS': 5000,
'SNAPSHOT_PREFIX': 'VGGnet_fast_rcnn',
'STEPSIZE': 50000,
'USE_FLIPPED': True,
'USE_PREFETCH': False},
'USE_GPU_NMS': True}
<bound method pascal_voc.default_roidb of <datasets.pascal_voc.pascal_voc object at 0x7fda0feb6ed0>>
Loaded dataset `voc_2007_trainval` for training
Appending horizontally-flipped training examples...
voc_2007_trainval gt roidb loaded from /media/han/E/mWork/mCode/Faster-RCNN_TF/data/cache/voc_2007_trainval_gt_roidb.pkl
done
Preparing training data...
done
Output will be saved to `/media/han/E/mWork/mCode/Faster-RCNN_TF/output/faster_rcnn_end2end/voc_2007_trainval`
/gpu:0
Tensor("Placeholder:0", shape=(?, ?, ?, 3), dtype=float32)
Tensor("conv5_3/conv5_3:0", shape=(?, ?, ?, 512), dtype=float32)
Tensor("rpn_cls_score/rpn_cls_score:0", shape=(?, ?, ?, 18), dtype=float32)
Tensor("Placeholder_2:0", shape=(?, 5), dtype=float32)
Tensor("Placeholder_1:0", shape=(?, 3), dtype=float32)
Tensor("Placeholder:0", shape=(?, ?, ?, 3), dtype=float32)
Tensor("rpn_conv/3x3/rpn_conv/3x3:0", shape=(?, ?, ?, 512), dtype=float32)
Tensor("rpn_cls_score/rpn_cls_score:0", shape=(?, ?, ?, 18), dtype=float32)
Tensor("rpn_cls_prob:0", shape=(?, ?, ?, ?), dtype=float32)
Tensor("rpn_cls_prob_reshape:0", shape=(?, ?, ?, 18), dtype=float32)
Tensor("rpn_bbox_pred/rpn_bbox_pred:0", shape=(?, ?, ?, 36), dtype=float32)
Tensor("Placeholder_1:0", shape=(?, 3), dtype=float32)
Tensor("rpn_rois:0", shape=(?, 5), dtype=float32)
Tensor("Placeholder_2:0", shape=(?, 5), dtype=float32)
Tensor("conv5_3/conv5_3:0", shape=(?, ?, ?, 512), dtype=float32)
(<tf.Tensor 'roi-data/rois:0' shape=(?, 5) dtype=float32>, <tf.Tensor 'roi-data/Cast:0' shape=<unknown> dtype=int32>, <tf.Tensor 'roi-data/PyFunc:2' shape=<unknown> dtype=float32>, <tf.Tensor 'roi-data/PyFunc:3' shape=<unknown> dtype=float32>, <tf.Tensor 'roi-data/PyFunc:4' shape=<unknown> dtype=float32>)
[<tf.Tensor 'conv5_3/conv5_3:0' shape=(?, ?, ?, 512) dtype=float32>, <tf.Tensor 'roi-data/rois:0' shape=(?, 5) dtype=float32>]
Tensor("drop7/mul:0", shape=(?, 4096), dtype=float32)
Use network `VGGnet_train` in training
Filtered 0 roidb entries: 10022 -> 10022
2018-07-31 21:08:56.623549: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2018-07-31 21:08:56.701373: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:898] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2018-07-31 21:08:56.701736: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1356] Found device 0 with properties:
name: GeForce GTX 1060 6GB major: 6 minor: 1 memoryClockRate(GHz): 1.7085
pciBusID: 0000:01:00.0
totalMemory: 5.93GiB freeMemory: 5.14GiB
2018-07-31 21:08:56.701748: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1435] Adding visible gpu devices: 0
2018-07-31 21:08:57.187696: I tensorflow/core/common_runtime/gpu/gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix:
2018-07-31 21:08:57.187717: I tensorflow/core/common_runtime/gpu/gpu_device.cc:929] 0
2018-07-31 21:08:57.187722: I tensorflow/core/common_runtime/gpu/gpu_device.cc:942] 0: N
2018-07-31 21:08:57.188150: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 4912 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1060 6GB, pci bus id: 0000:01:00.0, compute capability: 6.1)
Computing bounding-box regression targets...
[0. 0. 0. 0.]
[0. 0. 0. 0.]
[0. 0. 0. 0.]
[0. 0. 0. 0.]
[0. 0. 0. 0.]
[0. 0. 0. 0.]
[0. 0. 0. 0.]
[0. 0. 0. 0.]
[0. 0. 0. 0.]
[0. 0. 0. 0.]
[0. 0. 0. 0.]
[0. 0. 0. 0.]
[0. 0. 0. 0.]
[0. 0. 0. 0.]
[0. 0. 0. 0.]
[0. 0. 0. 0.]
[0. 0. 0. 0.]]
[0. 0. 0. 0.]
bbox target stdevs:
[[0.1 0.1 0.2 0.2]
[0.1 0.1 0.2 0.2]
[0.1 0.1 0.2 0.2]
[0.1 0.1 0.2 0.2]
[0.1 0.1 0.2 0.2]
[0.1 0.1 0.2 0.2]
[0.1 0.1 0.2 0.2]
[0.1 0.1 0.2 0.2]
[0.1 0.1 0.2 0.2]
[0.1 0.1 0.2 0.2]
[0.1 0.1 0.2 0.2]
[0.1 0.1 0.2 0.2]
[0.1 0.1 0.2 0.2]
[0.1 0.1 0.2 0.2]
[0.1 0.1 0.2 0.2]
[0.1 0.1 0.2 0.2]
[0.1 0.1 0.2 0.2]
[0.1 0.1 0.2 0.2]
[0.1 0.1 0.2 0.2]
[0.1 0.1 0.2 0.2]
[0.1 0.1 0.2 0.2]]
[0.1 0.1 0.2 0.2]
Normalizing targets
done
Solving...
/home/han/anaconda3/envs/py27/lib/python2.7/site-packages/tensorflow/python/ops/gradients_impl.py:100: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
"Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
2018-07-31 21:08:59.738460: W tensorflow/core/framework/allocator.cc:101] Allocation of 411041792 exceeds 10% of system memory.
2018-07-31 21:08:59.953198: W tensorflow/core/framework/allocator.cc:101] Allocation of 411041792 exceeds 10% of system memory.
2018-07-31 21:09:00.094394: W tensorflow/core/framework/allocator.cc:101] Allocation of 411041792 exceeds 10% of system memory.
Loading pretrained model weights from data/pretrain_model/VGG_imagenet.npy
assign pretrain model weights to conv5_1
assign pretrain model biases to conv5_1
assign pretrain model weights to fc6
assign pretrain model biases to fc6
assign pretrain model weights to conv5_3
assign pretrain model biases to conv5_3
assign pretrain model weights to fc7
assign pretrain model biases to fc7
ignore fc8
ignore fc8
assign pretrain model weights to conv5_2
assign pretrain model biases to conv5_2
assign pretrain model weights to conv4_1
assign pretrain model biases to conv4_1
assign pretrain model weights to conv4_2
assign pretrain model biases to conv4_2
assign pretrain model weights to conv4_3
assign pretrain model biases to conv4_3
assign pretrain model weights to conv3_3
assign pretrain model biases to conv3_3
assign pretrain model weights to conv3_2
assign pretrain model biases to conv3_2
assign pretrain model weights to conv3_1
assign pretrain model biases to conv3_1
assign pretrain model weights to conv1_1
assign pretrain model biases to conv1_1
assign pretrain model weights to conv1_2
assign pretrain model biases to conv1_2
assign pretrain model weights to conv2_2
assign pretrain model biases to conv2_2
assign pretrain model weights to conv2_1
assign pretrain model biases to conv2_1
2018-07-31 21:09:16.592446: W tensorflow/core/common_runtime/bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.09GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2018-07-31 21:09:18.131959: W tensorflow/core/common_runtime/bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.49GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2018-07-31 21:09:19.693919: W tensorflow/core/common_runtime/bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.52GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
iter: 10 / 70000, total loss: 1.2254, rpn_loss_cls: 0.3864, rpn_loss_box: 0.0024, loss_cls: 0.5635, loss_box: 0.2731, lr: 0.001000
speed: 0.753s / iter
2018-07-31 21:09:24.066658: W tensorflow/core/common_runtime/bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.09GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2018-07-31 21:09:25.726283: W tensorflow/core/common_runtime/bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.90GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2018-07-31 21:09:25.833259: W tensorflow/core/common_runtime/bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 2.78GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2018-07-31 21:09:25.884801: W tensorflow/core/common_runtime/bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.71GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2018-07-31 21:09:27.742113: W tensorflow/core/common_runtime/bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.49GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
2018-07-31 21:09:28.199178: W tensorflow/core/common_runtime/bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.49GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
iter: 20 / 70000, total loss: 2.0806, rpn_loss_cls: 0.3751, rpn_loss_box: 0.0652, loss_cls: 0.9635, loss_box: 0.6768, lr: 0.001000
speed: 0.682s / iter
2018-07-31 21:09:30.392418: W tensorflow/core/common_runtime/bfc_allocator.cc:219] Allocator (GPU_0_bfc) ran out of memory trying to allocate 3.34GiB. The caller indicates that this is not a failure, but may mean that there could be performance gains if more memory were available.
iter: 30 / 70000, total loss: 1.2051, rpn_loss_cls: 0.1684, rpn_loss_box: 0.0309, loss_cls: 0.7065, loss_box: 0.2992, lr: 0.001000
speed: 0.691s / iter
iter: 40 / 70000, total loss: 0.7352, rpn_loss_cls: 0.4540, rpn_loss_box: 0.1017, loss_cls: 0.1793, loss_box: 0.0002, lr: 0.001000
speed: 0.637s / iter
iter: 50 / 70000, total loss: 1.0909, rpn_loss_cls: 0.3517, rpn_loss_box: 0.0146, loss_cls: 0.5463, loss_box: 0.1783, lr: 0.001000
speed: 0.611s / iter
iter: 60 / 70000, total loss: 0.7737, rpn_loss_cls: 0.4024, rpn_loss_box: 0.0482, loss_cls: 0.2567, loss_box: 0.0665, lr: 0.001000
speed: 0.585s / iter
iter: 70 / 70000, total loss: 0.3217, rpn_loss_cls: 0.2266, rpn_loss_box: 0.0038, loss_cls: 0.0905, loss_box: 0.0008, lr: 0.001000
speed: 0.566s / iter
iter: 80 / 70000, total loss: 2.4348, rpn_loss_cls: 0.2842, rpn_loss_box: 0.0690, loss_cls: 1.4211, loss_box: 0.6605, lr: 0.001000
speed: 0.552s / iter
# 省略中间部分
iter: 69990 / 70000, total loss: 0.3971, rpn_loss_cls: 0.0115, rpn_loss_box: 0.0245, loss_cls: 0.0678, loss_box: 0.2931, lr: 0.000100
speed: 0.240s / iter
iter: 70000 / 70000, total loss: 0.0669, rpn_loss_cls: 0.0059, rpn_loss_box: 0.0127, loss_cls: 0.0159, loss_box: 0.0324, lr: 0.000100
speed: 0.240s / iter
Wrote snapshot to: /media/han/E/mWork/mCode/Faster-RCNN_TF/output/faster_rcnn_end2end/voc_2007_trainval/VGGnet_fast_rcnn_iter_70000.ckpt
done solving
real 311m0.563s
user 1377m18.319s
sys 1379m38.735s
+ set +x
grep: experiments/logs/faster_rcnn_end2end_VGG16_.txt.2018-07-31_22-46-08: No such file or directory
+ python ./tools/test_net.py --device gpu --device_id 0 --weights --imdb voc_2007_test --cfg experiments/cfgs/faster_rcnn_end2end.yml --network VGGnet_test
voc_2007_train
voc_2007_val
voc_2007_trainval
voc_2007_test
kitti_train
kitti_val
kitti_trainval
kitti_test
nthu_71
nthu_370
usage: test_net.py [-h] [--device DEVICE] [--device_id DEVICE_ID]
[--def PROTOTXT] [--weights MODEL] [--cfg CFG_FILE]
[--wait WAIT] [--imdb IMDB_NAME] [--comp]
[--network NETWORK_NAME]
test_net.py: error: argument --weights: expected one argument5 自己训练70000次迭代后，检测VOC_test_2007计算AP
Writing aeroplane VOC results file
Writing bicycle VOC results file
Writing bird VOC results file
Writing boat VOC results file
Writing bottle VOC results file
Writing bus VOC results file
Writing car VOC results file
Writing cat VOC results file
Writing chair VOC results file
Writing cow VOC results file
Writing diningtable VOC results file
Writing dog VOC results file
Writing horse VOC results file
Writing motorbike VOC results file
Writing person VOC results file
Writing pottedplant VOC results file
Writing sheep VOC results file
Writing sofa VOC results file
Writing train VOC results file
Writing tvmonitor VOC results file
VOC07 metric? Yes
AP for aeroplane = 0.7384
AP for bicycle = 0.7871
AP for bird = 0.6821
AP for boat = 0.5594
AP for bottle = 0.5461
AP for bus = 0.7854
AP for car = 0.8031
AP for cat = 0.7994
AP for chair = 0.5188
AP for cow = 0.7300
AP for diningtable = 0.6734
AP for dog = 0.7782
AP for horse = 0.7990
AP for motorbike = 0.7478
AP for person = 0.7671
AP for pottedplant = 0.4376
AP for sheep = 0.6763
AP for sofa = 0.6506
AP for train = 0.7645
AP for tvmonitor = 0.7188
Mean AP = 0.6982