GTX1070+CUDA8.0+Ubuntu16.04+Caffe+SSD 深度学习框架搭建 细节一步到位版

版权声明:本文为博主原创文章,未经博主允许不得转载。 https://blog.csdn.net/sinat_31802439/article/details/52958791

在此首先感谢各种深度学习讨论QQ群中网友的帮助,感谢乐于在网上共享一些编译代码时遇到的一些疑难杂症的中外同志们。。

这个是真的不太好装。。


1.关闭独立显卡,启动集成显卡。

开机,按F2,进入BISO设置,高级,显卡管理,显卡启用选择ICPU。

用一个DVI-D的数据线连接电脑主板和一台屏幕比较小比较老的显示器。(别问我为什么。。)

启动电脑。

2.安装Ubuntu16.04.

14.04试过5次,最终还是不行。

用Ultral刻录一个Ubuntu16.04的安装盘。

开机,按F2选择启动项,用U盘启动。

安装Ubuntu16.04

3.NVIDIA 驱动安装

1) Ctrl+alt+F1 进入字符界面,关闭图形界面
sudo service lightdm stop //必须有,不然会安装失败
2) 安装 nvidia driver
sudo chmod 755 NVIDIA-Linux-x86_64-367.27.run
sudo ./NVIDIA-Linux-x86_64-367.27.run
//获取权限
//安装驱动
Accept
Continue installation
安装完成之后
sudo service lightdm start

重启电脑 关闭集成显卡 开启独立显卡

4、 Cuda8.0 安装

1) 在终端运行指令 sudo sh cuda_8.0.27_linux.run --no-opengl-libs 不加这个选项会进入循环登陆(别问我怎么知道的)
选择
Do you accept the previously read EULA?
accept/decline/quit: accept
Install NVIDIA Accelerated Graphics Driver for Linux-x86_64
361.62?
(y)es/(n)o/(q)uit: n

Install the CUDA 8.0 Toolkit?
(y)es/(n)o/(q)uit: y
Enter Toolkit Location
[ default is /usr/local/cuda-8.0 ]:
Do you want to install a symbolic link at /usr/local/cuda?
(y)es/(n)o/(q)uit: y
Install the CUDA 8.0 Samples?
(y)es/(n)o/(q)uit: y
Enter CUDA Samples Location
[ default is /home/zhou ]:
Installing the CUDA Toolkit in /usr/local/cuda-8.0 ...
Missing recommended library: libGLU.so
Missing recommended library: libX11.so
Missing recommended library: libXi.so
Missing recommended library: libXmu.so
Installing the CUDA Samples in /home/zhou ...
Copying samples to /home/zhou/NVIDIA_CUDA-8.0_Samples now...
Finished copying samples.
===========
= Summary =
===========
Driver: Not Selected
Toolkit: Installed in /usr/local/cuda-8.0
Samples: Installed in /home/zhou, but missing recommended
libraries
Please make sure that
- PATH includes /usr/local/cuda-8.0/bin
- LD_LIBRARY_PATH includes /usr/local/cuda-8.0/lib64, or, add
/usr/local/cuda-8.0/lib64 to /etc/ld.so.conf and run ldconfig as
root

To uninstall the CUDA Toolkit, run the uninstall script in
/usr/local/cuda-8.0/bin
Please see CUDA_Installation_Guide_Linux.pdf in
/usr/local/cuda-8.0/doc/pdf for detailed information on setting up
CUDA.
***WARNING: Incomplete installation! This installation did not
install the CUDA Driver. A driver of version at least 361.00 is
required for CUDA 8.0 functionality to work.
To install the driver using this installer, run the following
command, replacing with the name of this run file:
sudo .run -silent -driver
Logfile is /tmp/cuda_install_2961.log
安装完成,但是缺少一些库。

2) 安装所缺少的库
sudo apt-get install freeglut3-dev build-essential libx11-dev
libxmu-dev libxi-dev libgl1-mesa-glx libglu1-mesa
libglu1-mesa-dev

安装完成。

3)
sudo apt-get install vim
4) 设置环境变量
在终端输入这两句:
export PATH=/usr/local/cuda-8.0/bin:$PATH
export
LD_LIBRARY_PATH=/usr/local/cuda-8.0/lib64:$LD_LIBRARY_PATH
然后修改文件中环境变量设置sudo vi /etc/profile
输入上面 export 的两句,保存,退出。
sudo ldconfig //环境变量立即生效
5) 验证安装是否完成
nvidia-smi

nvcc –V

显示:
nvcc -V nvcc: NVIDIA (R) Cuda compiler driver Copyright
(c) 2005-2016
NVIDIA Corporation Built on Wed_May__4_21:01:56_CDT_2016

6) 测试 cuda 的 samples
cd ‘/home/zhou/NVIDIA_CUDA-8.0_Samples’
make

7) 安装 cudnn5.0
安装 cuDNN 比较简单,解压后把相应的文件拷贝到对应的 CUDA 目录下即
可:
tar -zxvf cudnn-8.0-linux-x64-v5.0-ga.tgz
显示以下信息:
*cuda/include/cudnn.h
cuda/lib64/libcudnn.so
cuda/lib64/libcudnn.so.5
cuda/lib64/libcudnn.so.5.0.5
cuda/lib64/libcudnn_static.a*
继续执行以下指令:
sudo cp cuda/include/cudnn.h /usr/local/cuda/include/
sudo cp cuda/lib64/libcudnn* /usr/local/cuda/lib64/
sudo chmod a+r /usr/local/cuda/include/cudnn.h
sudo chmod a+r /usr/local/cuda/lib64/libcudnn*
8) 验证一下,哈哈哈哈哈啊哈哈哈哈哈
cuda 的 samples 里面有个 deviceQuery 运行之后会显示信息,最后一行
出行 pass 说明成功啦~~~~

5、 SSD 安装

sudo apt-get update

1) 安装依赖项
sudo apt-get install libprotobuf-dev libleveldb-dev libsnappy-dev
libopencv-dev libhdf5-serial-dev protobuf-compiler
sudo apt-get install --no-install-recommends libboost-all-dev

2) BLAS 安装
sudo apt-get install libatlas-base-dev
3) 安装 pycaffe 接口所需要的依赖项
sudo apt-get install -y python-numpy python-scipy
python-matplotlib python-sklearn python-skimage python-h5py
python-protobuf python-leveldb python-networkx python-nose
python-pandas python-gflags cython ipython
4) 继续安装依赖项
sudo apt-get install libgflags-dev libgoogle-glog-dev liblmdb-dev
5) opencv3.0 安装
参照 http://ouxinyu.github.io/Blogs/20151108001.html
(1)使用他提供的修改版的安装包(即 百度云 下载,密码:wysa)(下面
的安装方式使用该包完成,安装包修改了 dependencies.sh 文件并增加了
OpenCV 3.0.0 的安装文件)
(2)切换到文件保存的文件夹(/home/zhou/opencv3.0),然后安装依
赖项
sudo sh dependencies.sh
(3)安装 OpenCV3.0
sudo sh opencv3_0_0.sh
==。 就应该直接自己下载文件的,我这渣网速。
虽然最后下完了,但是在半路又需要下载文件
ippicv_linux_20141027.tgz
因为网络不稳定,所以我在自己的电脑上下载了之后复制到需要的文件夹:
cp ippicv_linux_20141027.tgz
/home/zhou/opencv3.0/OpenCV/opencv-3.0.0/3rdparty/ippicv/downlo
ads/linux-8b449a536a2157bcad08a2b9f266828b
重新运行指令:
sudo sh opencv3_0_0.sh
然后开始正常安装,并开始 make,但是 make 到 72%出现错误,
graphcuts.cpp 文件中许多变量没有声明,google 之后发现是因为

opencv3.0 还不支持 cuda8.0,但是有个同学已经对其进行修改。
请参考:
https://github.com/opencv/opencv/pull/6510/commits/10896129b39655e19e4e7
c529153cb5c2191a1db
cd
'/home/zhou/opencv3.0/OpenCV/opencv-3.0.0/modules/cudalegacy/sr
c'
sudo vi graphcuts.cpp
进行修改

再次运行 sudo sh opencv3_0_0.sh
这次运行正常,至此 opencv3.0 安装成功!!!

**6)**caffe 编译

1、安装Git

终端输入:

sudo apt-get install git

2、安装SSD

在主文件下终端输入(即/home/***(您的服务器名字)这个目录):

git clone https://github.com/weiliu89/caffe.git

cd caffe

git checkout ssd(出现“分支”则说明copy-check成功)


第二部分:配置SSD(caffe)
终端输入:
cd /home/**(您服务器的名字)/caffe
cp Makefile.config.example Makefile.config

打开Makefile.config,修改之处可以参考我的Makefile.config文件,文件地址:( 链接:http://pan.baidu.com/s/1mgVN57i   密码:34e6 )保存退出。

Makefile.config文件的第85行,添加/usr/include/hdf5/serial/ 到 INCLUDE_DIRS,也就是把下面第一行代码改为第二行代码。

<code class="hljs ruby has-numbering" style="display: block; padding: 0px; background: transparent; color: inherit; box-sizing: border-box; font-family: "Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal;"><span class="hljs-constant" style="box-sizing: border-box;">INCLUDE_DIRS</span> <span class="hljs-symbol" style="color: rgb(0, 102, 102); box-sizing: border-box;">:</span>= <span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$(</span><span class="hljs-constant" style="box-sizing: border-box;">PYTHON_INCLUDE</span>) /usr/local/<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">include</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right: 1px solid rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right: 1px solid rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>
<code class="hljs ruby has-numbering" style="display: block; padding: 0px; background: transparent; color: inherit; box-sizing: border-box; font-family: "Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal;"><span class="hljs-constant" style="box-sizing: border-box;">INCLUDE_DIRS</span> <span class="hljs-symbol" style="color: rgb(0, 102, 102); box-sizing: border-box;">:</span>= <span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$(</span><span class="hljs-constant" style="box-sizing: border-box;">PYTHON_INCLUDE</span>) /usr/local/<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">include</span> /usr/<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">include</span>/hdf5/serial/</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right: 1px solid rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right: 1px solid rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

Step 2

Makefile文件的第173行,把 hdf5_hl 和hdf5修改为hdf5_serial_hl 和 hdf5_serial,也就是把下面第一行代码改为第二行代码。

<code class="hljs fix has-numbering" style="display: block; padding: 0px; background: transparent; color: inherit; box-sizing: border-box; font-family: "Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal;"><span class="hljs-attribute" style="box-sizing: border-box;">LIBRARIES +</span>=<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;"> glog gflags protobuf boost_system boost_filesystem m hdf5_hl hdf5</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right: 1px solid rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right: 1px solid rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>
<code class="hljs fix has-numbering" style="display: block; padding: 0px; background: transparent; color: inherit; box-sizing: border-box; font-family: "Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal;"><span class="hljs-attribute" style="box-sizing: border-box;">LIBRARIES +</span>=<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;"> glog gflags protobuf boost_system boost_filesystem m hdf5_serial_hl hdf5_serial</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right: 1px solid rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right: 1px solid rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

I solve it by add boost_regex to LIBRARIES in Makefile
263 LIBRARIES += boost_thread stdc++ boost_regex

cd /home/**(您服务器的名字)/caffe
mkdir build
cd build
cmake ..
(cmake和..中间又一个空格
make all -j16("‐j16"是使用 CPU 的多核进行编译,可以极大地加速编译的速度)
make install
make runtest
(这一步不是必须的)
make pycaffe(编译pycaffe)

第三部分:下载数据文件

1、 预训练模型下载

下载地址:链接:http://pan.baidu.com/s/1slpaEO9 密码:loxo

在caffe/models文件夹下新建文件夹,命名为VGGNet,将刚刚下载下来的文件放入这个VGGNet文件夹当中

2、下载VOC2007和VOC2012数据集

在主文件夹下(即/home/**(您服务器的名字)/)新建文件夹,命名为data

终端输入:

cd /home/**(您服务器的名字)/data

wget http://host.robots.ox.ac.uk/pascal/VOC/voc2012/VOCtrainval_11-May-2012.tar

wget http://host.robots.ox.ac.uk/pascal/VOC/voc2007/VOCtrainval_06-Nov-2007.tar

wget http://host.robots.ox.ac.uk/pascal/VOC/voc2007/VOCtest_06-Nov-2007.tar
解压这三个文件,终端输入:
cd /home/**(您服务器的名字)/data
tar -xvf VOCtrainval_11-May-2012.tar
tar -xvf VOCtrainval_06-Nov-2007.tar
tar -xvf VOCtest_06-Nov-2007.tar
(请严格按照 这个顺序 解压)

 

第四部分:生成LMDB文件

终端输入:
cd /home/**(您服务器的名字)/caffe
./data/VOC0712/create_list.sh
./data/VOC0712/create_data.sh

在运行第三步时如果出现no module named caffe或者是no module named caffe-proto,则在终端输入:

export PYTHONPATH=$PYTHONPATH:/home/**(您服务器的名字)/caffe/Python

然后再次运行第三步

 

第五部分:训练测试演示
1、 训练
打开caffe/examples/ssd/ ssd_pascal.py 这个文件,找到 gpus=’0,1,2,3’ 这一行,如果您的服务器有 一块 显卡,则将 123 删去,如果有 两个 显卡,则删去 23 ,以此类推。如果您服务器没有gpu支持,则注销以下几行,程序会以cpu形式训练。(这个是解决问题 cudasuccess(10vs0) 的方法)
#Ifnum_gpus >0:

                # batch_size_per_device =int(math.ceil(float(batch_size) / num_gpus))

#iter_size =int(math.ceil(float(accum_batch_size) / (batch_size_per_device * num_gpus)))

 # solver_mode =P.Solver.GPU
  # device_id =int(gpulist[0])

保存后终端运行:
cd  /home/**(您服务器的名字)/caffe
python examples/ssd/ssd_pascal.py

如果出现问题 cudasuccess(2vs0) 则说明您的显卡计算量有限,再次打开caffe/examples/ssd/ ssd_pascal.py 这个文件,找到 batch_size =32 这一行,修改数字 32 ,可以修改为 16 ,或者 8 ,甚至为 4 (相信大家这个修改可以理解,我就不作说明了),保存后再次终端运行 python examples/ssd/ssd_pascal.py
2、 测试
终端输入:
python examples/ssd/score_ssd_pascal.py (演示detection的训练结果,数值在 0.718 左右)
3、 演示
演示网络摄像头识别效果,终端输入:
python examples/ssd/ssd_pascal_webcam.py
demo结果如下( 这里我只帖一张演示图,不过这是我从摄像头实时检测结果中截取的 ):



Makefile.config文件的第85行,添加/usr/include/hdf5/serial/ 到 INCLUDE_DIRS,也就是把下面第一行代码改为第二行代码。

<code class="hljs ruby has-numbering" style="display: block; padding: 0px; background: transparent; color: inherit; box-sizing: border-box; font-family: "Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal;"><span class="hljs-constant" style="box-sizing: border-box;">INCLUDE_DIRS</span> <span class="hljs-symbol" style="color: rgb(0, 102, 102); box-sizing: border-box;">:</span>= <span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$(</span><span class="hljs-constant" style="box-sizing: border-box;">PYTHON_INCLUDE</span>) /usr/local/<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">include</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right: 1px solid rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right: 1px solid rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>
<code class="hljs ruby has-numbering" style="display: block; padding: 0px; background: transparent; color: inherit; box-sizing: border-box; font-family: "Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal;"><span class="hljs-constant" style="box-sizing: border-box;">INCLUDE_DIRS</span> <span class="hljs-symbol" style="color: rgb(0, 102, 102); box-sizing: border-box;">:</span>= <span class="hljs-variable" style="color: rgb(102, 0, 102); box-sizing: border-box;">$(</span><span class="hljs-constant" style="box-sizing: border-box;">PYTHON_INCLUDE</span>) /usr/local/<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">include</span> /usr/<span class="hljs-keyword" style="color: rgb(0, 0, 136); box-sizing: border-box;">include</span>/hdf5/serial/</code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right: 1px solid rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right: 1px solid rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

Step 2

Makefile文件的第173行,把 hdf5_hl 和hdf5修改为hdf5_serial_hl 和 hdf5_serial,也就是把下面第一行代码改为第二行代码。

<code class="hljs fix has-numbering" style="display: block; padding: 0px; background: transparent; color: inherit; box-sizing: border-box; font-family: "Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal;"><span class="hljs-attribute" style="box-sizing: border-box;">LIBRARIES +</span>=<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;"> glog gflags protobuf boost_system boost_filesystem m hdf5_hl hdf5</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right: 1px solid rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right: 1px solid rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>
<code class="hljs fix has-numbering" style="display: block; padding: 0px; background: transparent; color: inherit; box-sizing: border-box; font-family: "Source Code Pro", monospace;font-size:undefined; white-space: pre; border-radius: 0px; word-wrap: normal;"><span class="hljs-attribute" style="box-sizing: border-box;">LIBRARIES +</span>=<span class="hljs-string" style="color: rgb(0, 136, 0); box-sizing: border-box;"> glog gflags protobuf boost_system boost_filesystem m hdf5_serial_hl hdf5_serial</span></code><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right: 1px solid rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul><ul class="pre-numbering" style="box-sizing: border-box; position: absolute; width: 50px; background-color: rgb(238, 238, 238); top: 0px; left: 0px; margin: 0px; padding: 6px 0px 40px; border-right: 1px solid rgb(221, 221, 221); list-style: none; text-align: right;"><li style="box-sizing: border-box; padding: 0px 5px;">1</li></ul>

猜你喜欢

转载自blog.csdn.net/sinat_31802439/article/details/52958791