OPENCV training model

1 Introduction

​Using Cascade Classifier consists of two main phases: training phase and detection phase. OpenCV applications that need to be used: opencv_createsamples, opencv_annotation, opencv_traincascade and opencv_visualisation. opencv_createsamples and opencv_traincascade have been disabled since OpenCV 4.0, but since the model format of 3.4 and 4.x is the same, the program of version 3.4 can be used for model training.

2. Prepare training data

To train a model, we need a set of positive examples (containing the actual objects you want to detect) and a set of negative examples (containing everything you don't want to detect). Negative sample sets have to be prepared manually, while positive sample sets are created using opencv_createssamples.

2.1. Sample requirements

• It is best to use a grayscale image for the sample image, and it is best to do some preprocessing according to the actual situation;
• The larger the number of samples, the better, try to be higher than 1000;
• The greater the variability between samples, the better;
• The best ratio of positive and negative samples is 1:3;
• The best sample size is 20x20;

2.2. Get the training program

WINDOWS:
	下载地址https://opencv.org/releases/，目前4.x.x版本中没有训练程序，此处采用3.4.16版本，下载后安装就可以找到对应的exe文件；
LINUX：
	下载地址https://opencv.org/releases/，目前4.x.x版本中没有训练程序，此处采用3.4.16版本，编译后生成的目录为“build/bin”

2.3. Prepare directory and program

• negdata: the directory where the negative samples are placed
• posdata: the directory where the positive samples are placed
• opencv_annotation: used to visually select objects of interest in any given image and create info description files
• opencv_createsamples: an executable program that generates a sample description file
• opencv_traincascade: executable program for sample training
• opencv_visualisation: Visualize the cascade after training, you can see which features are selected and how complex its stages are

3. Collect sample data

3.1. Positive samples

Positive samples are created by the opencv_createssamples application. The application supports two methods for generating positive sample datasets:

• Method 1: Generate a bunch of positive sample images from a positive sample image
• Method 2: Provide all the positive sample images yourself, and use a tool to cut them out, resize them and put them into the binary format required by openv

3.1.1. Positive sample generation method 1

Method 1 works very well for fixed objects, but it tends to fail quickly for less rigid objects, in which case method 2 is recommended. OPENCV even states that 100 real images can produce a better model than 1000 artificially generated ones. If you still decide to go with Method 1, please keep the following in mind:

• Note that you need more than one positive sample before feeding it to the application, since it only works with perspective transformation.
• If you want a robust model, then take samples that cover the various types that may occur in the object class. For example, training a face model should take into account different races, ages, emotions, and perhaps beard styles, which also applies to using the second method.

Approach 1 takes a single object image and randomly rotates the object, varies the image intensity, and places the image on an arbitrary background to create a large number of positive samples from a given object image. The amount and range of randomness can be controlled via command line arguments to the opencv_createsamples application.

​ Parameter description of opencv_createsamples:

	-vec	用于训练的正样本的输出文件名
	-img	源对象图像
	-bg 	背景描述文件，包含一个图像列表，这些图像用作对象随机扭曲版本的背景
	-num 	生成的正样本数量
    -bgcolor	背景色(目前采用灰度图像)，背景色为透明色。由于可能存在压缩伪影，因此可以通过-bgthresh指定颜色容错量。bgcolor-bgthresh和bgcolor+bgthresh范围内的所有像素都被解释为透明的。
    -bgthresh < background_color_threshold >
    -inv		如果指定，颜色将被反转
	-randinv	如果指定，颜色将随机反转
    -maxidev	前景样本中像素的最大强度偏差
    -maxxangle	朝向x轴的最大旋转角度，必须以弧度为单位
    -maxyangle	朝向y轴的最大旋转角度，必须以弧度为单位
    -maxzangle	朝向z轴的最大旋转角度，必须以弧度为单位
    -show		调试选项。如果指定，将显示每个样品。按Esc将继续示例创建过程，但不会显示每个示例
    -w			输出样本的宽度(像素)
    -h			输出样本的高度(像素)

When running opencv_createsamples with method 1, use the following procedure to create sample object instances:

• The given source image is randomly rotated around all three axes. The chosen angle is limited by -maxxangle, -maxyangle and -maxzangle
• Pixels with brightness in the range [bg_color-bg_color_threshold; Bg_color + bg_color_threshold] are interpreted as transparent. White noise is added to the intensity of the foreground
• If -inv is specified, foreground pixel intensities are inverted. If -randinv key is specified, the algorithm randomly chooses whether to invert the sample
• Finally, place the obtained image from the background description file onto an arbitrary background, resize it to the desired size specified by -w and -h, and store into the vec file specified by -vec

3.1.2. Positive sample generation method 2

Positive samples can also be obtained from a collection of previously labeled images, which is required when building robust object models. The collection is described by a text file similar to the background description file. Each line of this file corresponds to an image. The first element of the line is the filename, then the number of object annotations, then numbers describing the coordinates of the object's bounding rectangle (x, y, width, height).

​ Enter the posdata directory, and then create pos.txt, the directory format is as follows:

/posdata
	00001.jpg 
	00002.jpg 
	00003.jpg 
	pos.txt

The content of pos.txt is as follows:

00001.jpg 3 548 605 95 100 771 641 125 94 1199 634 182 269
00002.jpg 5 333 464 115 148 844 610 205 259 1139 460 98 138 1315 461 93 101 943 503 57 78
00003.jpg 5 5 491 103 156 472 517 80 114 845 449 151 177 877 631 148 203 1292 482 68 91

​ The image 00001.jpg contains 3 objects, namely {548, 605, 95, 100}, {771, 641, 125, 94}, {1199, 634, 182, 269}

To create positive samples from such a collection, the -info parameter should be specified instead of -img. Note that in method 2, parameters such as -bg, -bgcolor, -bgthreshold, -inv, -randinv, -maxxangle, -maxyangle, -maxzangle, etc. will be ignored and no longer used. The scheme of method 2 to create an example is as follows:

• Clip the provided bounding box from the original image, get the object instance from the given image
• Resize them to the target sample size (defined by -w and -h) and store in the output vec file defined by the -vec parameter
• The only parameters that matter for method 2 are -w, -h, -show and -num

Generate VEC instructions:

opencv_createsamples -info ${FILEINFO_POS} -vec ${FILEVEC_POS} -num ${NUM_POS} -bgcolor 0 -bgthresh 0 ${SIZ_POS}

3.1.3.opencv_annotation tool

​ The process of creating an info description file can also be done by using the opencv_annotation tool. This is an open-source tool for visually selecting regions of interest for object instances in any given image.

​ Using this tool is very simple. The tool accepts several required and some optional parameters:

-?, -h, --help, --usage (value:true)
					显示帮助信息
-a, --annotations 	必选参数，指定生成info描述文件的路径
-i,--images			必选参数，包含样本的目录路径
-m,--maxWindowHeight	可选参数，如果输入图像的高度大于这里给定的分辨率，请使用resizeFactor调整图像的大小
-r,--resizeFactor		可选参数，当使用maxWindowHeight参数时，用于调整输入图像大小的因子(默认：一半大小)。

​ Example:

./opencv_annotation -a=sample_people/posdata/pos.txt -i=sample_people/posdata/

The opencv_annotation command will open a window with the first image and the mouse cursor for annotation. The left mouse button is used to select the first corner of the object, then continue drawing until determined, and stop on the second left click. After each selection, you have the following options:
​ Press the c key: confirm the annotation, the annotation turns green, and confirm that it has been saved
Press the d key: delete the last annotation from the annotation list (convenient to delete wrong annotations)
Press the n key: Continue to the next image
Press ESC: Exit the annotation software
Finally, you will have a usable annotation file, which can be passed to the -info parameter of opencv_createsamples.

3.2. Negative samples

Negative samples are taken from any image, which should not contain the object you want to detect. These negative sample images are recorded in a negative sample description file, and each line of the description file contains the path of an image (it can be an absolute path or a relative path ). Note that negative samples and sample images are also called background samples or background images.

The described images may be of different sizes, however each image should be equal to or larger than the desired training window size (model size, in most cases the average size of the object).

​ Negative samples can't just find some pictures as negative samples. It is better to choose different negative samples according to different items. For example, if a project is to do face detection at an airport, then it is best to take some picture data from the scene and collect negative samples from it. For different projects, different positive samples and negative samples are collected.

​ Enter the negdata directory, and then create neg.txt, the directory format is as follows:

/negdata
	00001.jpg
	00002.jpg
	00003.jpg
	00004.jpg
	neg.txt

The content of neg.txt is as follows:

00001.jpg
00002.jpg
00003.jpg
00004.jpg

4. Model training

General parameters:

-data		训练好的分类器存储的位置，这个文件夹应该事先手动创建
-vec 		带有正样本的vec文件
-bg 		背景描述文件，包含负样本图像的文件
-numPos 	每个分类器阶段训练中使用的正样本数量。
-numNeg		每个分类器阶段训练中使用的负样本数量。
-numStages 	要训练的级联阶段的数量。
-precalcValBufSize 	预计算特征值的缓冲区大小(单位为Mb)。
-precalcIdxBufSize	预计算特征索引的缓冲区大小(单位:Mb)。
	注意：precalcValBufSize和precalcIdxBufSize分配的内存越多，训练过程就越快，但是不应超过可用的系统内存。
-baseFormatSave		仅在使用Haar特征时有效。如果指定了该参数，则级联将以旧格式保存。
-numThreads 训练期间使用的最大线程数。注意，实际使用的线程数可能更少，这取决于您的机器和编译选项。默认情况下，如果您使用TBB支持构建OpenCV，则选择最大可用线程，这是此优化所需要的。
-acceptanceRatioBreakValue	用于确定你的模型应该保持学习的精确程度以及何时停止。一个好的指导方针是训练不超过10e-5，以确保模型不会在你的训练数据上过度训练。缺省情况下，该值设置为-1，表示禁用该特性。

​Cascading parameters

-stageType		级别（stage）参数。目前只支持将BOOST分类器作为级联的类型；
-featureType	特征的类型：HAAR-类Haar特征；LBP-局部二进制模式（默认Harr）；
如果遇到的教程提到旧的opencv_haartraining工具(已弃用)，请忽略该教程并使用opencv_traincascade工具。openv_traincascade同时支持HAAR和LBP特征。与HAAR特征相比，LBP特征产生整数精度，产生浮点精度，因此LBP的训练和检测速度都比HAAR特征快几倍。对于LBP和HAAR的检测质量，主要取决于所使用的训练数据和所选择的训练参数。
-w 	训练样本的度(像素，默认24）；
-h	训练样本的度(像素，默认24）；
注意：训练样本的尺寸必须跟训练样本创建（使用 opencv_createsamples 程序创建）时的尺寸保持一致

​Boosted classifier parameters

-bt	
	Boosted分类器的类型（DAB-Discrete AdaBoost, RAB-Real AdaBoost, LB-LogitBoost,GAB-Gentle AdaBoost为默认）；
-minHitRate	
	每个阶段的最小期望命中率（默认值为0.995）。总命中率可以估计为(minHitRate^ number_of_stages)
-maxFalseAlarmRate 
	每一级希望得到的最大误检率（默认值为0.5），总的误检率大约为 maxFalseAlarmRate^number_of_stages；
-weightTrimRate	
	是否应该使用微调及其权重。一个不错的选择是0.95。
-maxDepth 		
	弱树的最大深度。一个合适的选择是1，这是树桩的情况。
-maxWeakCount 
	每个级联阶段弱树的最大计数（默认值为100）。boosted分类器(阶段)将有许多弱树(<=maxWeakCount)，以实现给定的-maxFalseAlarmRate。

Haar-like feature parameters

-mode 
	选择训练中使用的Haar特征集的类型。BASIC只使用直立特征，而ALL使用全套直立和45度旋转特征集。

​ After the opencv_traincascade application has finished its work, the trained cascade will be saved in the cascade.xml file in the -data folder. Other files in this folder are created for training breaks and you can delete them after training is over.

​

Training instructions:

opencv_traincascade -data ${PATH_OUT} -vec ${FILEVEC_POS} -bg ${FILEINFO_NEG}  -numPos ${NUM_POS} -numNeg ${NUM_NEG} -numStages 20 -feattureType HAAR ${SIZ_POS} -minHitRate 0.995 -maxFalseAlarmRate 0.5

5. Visualization cascade classifier opencv_visualisation

It is useful to visualize the post-training cascade to see which features it selects and how complex its stages are. For this purpose, OpenCV provides an opencv_visualisation application. This application has the following commands:

-?, -h, --help, --usage
	显示帮助信息
-d, --data
	(可选)如果提供了一个数据文件夹，必须事先手动创建，将存储阶段输出和特征的视频
-i, --image 
	(必选):指向对象模型的参考图像的路径。应该是一个维度为[-w，-h]的注释（annotation），同时传递给opencv_createssamples和opencv_traincascade应用程序。
-m, --model 
	(必选):训练模型的路径，它应该在openv_traincascade应用程序的-data参数提供的文件夹中。

​ example

./opencv_visualisation -m=output/sample_people/cascade.xml -d=output/sample_people/ -i=sample_people/posdata/object.jpg

Some limitations of current visualization tools

• Only handle cascaded classifier models trained with the opencv_traincascade tool, including stumps as decision trees [default setting].
• The provided image needs to be a sample window with the original model dimensions and is passed to the --image parameter.

6. Problem solving

6.1. Question 1

问题描述：Traincascade Error: Bad argument (Can not get new positive sample. The most possible reason is insufficient count of samples in given vec-file.

Solution: The main reason is that the value of numPos is not set correctly. Simply set numPos to the number of positive samples, and set numPos to the number of positive samples * 4/5.

6.2. Question 2

Problem description: (-215:Assertion failed) !ssize.empty() in function 'resize'

Solution: The sample point data in the description file is wrong;

7. Postscript

​ Based on human head (positive sample 4100, negative sample 23240), car (positive sample 516, negative sample 1044), pedestrian ((positive sample 1126, negative sample 1000)) and actual environment (positive sample 79, negative sample 19) training, It doesn't feel like the end result is very good. Those who know the reason or have ideas can communicate together.