Reprinted: https://blog.csdn.net/lanxuecc/article/details/52222369
Haar features
A Survey of Haar Feature Principles
The Haar feature is a feature that reflects the grayscale change of the image, and the pixel is divided into modules to calculate the difference. It is divided into three categories: edge features, linear features, center features, and diagonal features. A feature template is formed by combining black and white rectangular boxes, and the feature value of this template is represented by the sum of black rectangular pixels and the subtracted white rectangular pixels in the feature template. For example, some features of the face can be simply described by the difference feature of the rectangular module, such as: the eyes are darker than the cheeks, the sides of the bridge of the nose are darker than the bridge of the nose, the mouth is darker than the surrounding color, etc. However, the rectangular feature is only sensitive to some simple graphic structures, such as edges and line segments, so it can only describe the image structure with obvious pixel module gradient changes in specific directions (horizontal, vertical, diagonal).
As shown above, the image Haar features of modules A, B, and D are: v=Sum white-Sum black
The image Haar features of module C are: v=Sum white (left)+Sum white (right)-2*Sum black
Here to ensure The number of pixels in the module of the white rectangle is the same as the number of pixels in the module of the black rectangle, so multiply by 2
For an image, by changing the size and position of the feature template, a large number of features can be exhausted to represent an image. The feature template in the figure above is called "feature prototype"; the feature obtained by expanding (translation and scaling) the feature prototype in the image sub-window is called "rectangular feature"; the value of the rectangular feature is called "feature value". For example, in a 24*24 size image, you can calculate the feature of A in the picture above with a rectangular template starting from coordinates (0, 0) with a width of 20 and a height of 20. You can also use coordinates (0, 2) to start with a rectangular template with a width of 20 and a height of 20. To calculate the features of A in the above picture, you can also start from the coordinates (0, 0) with a width of 22 and a height of 22 to calculate the features of A in the above picture, so that the rectangular feature value changes with the category, size and position, making a very small one Small images contain very many rectangular features. The rectangle eigenvalues are a function of three factors: rectangle template category, rectangle location, and rectangle size.
Haar feature calculation method
First introduce the integral graph: as shown in the following code, it is the structure of storing image integral graph in opencv. You can see that there are two integral graphs sum integral graph and titled sum integral graph. The sum integral graph is used to calculate general vertical or horizontal rectangular features, such as (a), (b), (c), (d) below. . . , and the titled sum integral graph is used to calculate the integral graph with a slope of 45 degrees, such as (1c), (1d), (2c) below. . .
/* Prepared for training samples */
typedef struct CvHaarTrainingData
{
CvSize winsize; /* training image size */
int maxnum; /* maximum number of samples */
CvMat sum; /* sum images (each row represents image) */
CvMat tilted; /* tilted sum images (each row represents image) */
CvMat normfactor; /* normalization factor */
CvMat cls; /* classes. 1.0 - object, 0.0 - background */
CvMat weights; /* weights */
CvMat* valcache; /* precalculated feature values (CV_32FC1) */
CvMat* idxcache; /* presorted indices (CV_IDX_MAT_TYPE) */
} CvHaarTrainigData;- 1
- 2
- 3
- 4
- 5
- 6
- 7
- 8
- 9
- 10
- 11
- 12
- 13
- 14
The main idea of the sum integral graph is to store the sum of pixels in the rectangular area formed by the image from the starting point to each point as an element of an array in memory. When calculating the sum of pixels in a certain area, you can directly index the elements of the array. There is no need to recalculate the pixel sum for this area. For example: for an X*Y image, its integral graph is also an integral graph whose X*Y element is int, and the corresponding integral graph (x, y) position value represents the original image rectangle [(0,0), (x,y)] the sum of all pixels within. As shown below:
Represented by the formula: where p(i,j) represents the pixel value of the original image at the position (i,j), and f(x,y) represents the value of the integral image at the position (x,y)
The integral map can compute different features in the same time (constant time) at multiple scales, thus greatly improving the detection speed. As shown in the figure below, the two A features of rectangular boxes of different sizes are not located in the picture. Their calculation methods are:
[f(x,y)+f(i,j)-f(s,t)-f(l, m)]-[f(s,t)+f(h,k)-f(i,j)-f(n,o)]
[f(X,Y])+f(I,J)-f (S,T)-f(L,M)]-[f(S,T)+f(H,K)-f(I,J)-f(N,O)]
Two features of different sizes can be seen The amount of calculation is the same! ! !
The titled sum class integral graph calculates a rectangular feature with a 45° rotation angle. In order to facilitate the calculation of the rectangular feature with a 45° rotation angle, we define the point value of (x, y) in the titled sum class integral graph as RSAT(x, y), which represents the 45° area in the upper left corner of (x, y) in the original image and Pixel sum of the 45° area in the lower left corner. 
The formula is as follows:
The calculation method of the integral graph for each point has the following recursive formula to save performance and reduce repeated calculations:
The eigenvalue of the calculated matrix feature is the difference between the rectangle RSAT(x, y) located in the cross row. The rectangular features in the following figure are:
[RSAT(m,n)-RSAT(i,j)-RSAT(k,l)+RSAT(s,t)]-[RSAT(o,p)-RSAT(m,n )-RSAT(x,y)+RSAT(k,l)]
In the same way, the integral map can also use the same time (constant time) to calculate different 45-degree rectangular features at multiple scales.
Opencv source code corresponding to Haar feature
Haar features proposed by Viola:
Haar-like features proposed by Lienhart et al.:
The function icvCreateIntHaarFeatures is responsible for creating several haar features in the opencv source code. This function is executed after entering the classifier training function. It calculates the vertex coordinates of the rect of all HaarFeatures according to the winsize graph to determine the feature and stores it in CvIntHaarFeatures in the structure array. In the follow-up, for different images, you only need to use the vertices of these rectangular boxes to obtain the value of the integral graph at this point, and then the specific value of the feature can be calculated. When symmetric is 0, it means that all features are created, and when it is 1, it means that the target graph is vertically symmetric, so you only need to create all the features whose center is in the left half. mode==0 represents the original rectangular feature proposed by Viola, mode==1 represents all vertical haar features, and mode==2 represents all features
The specific names of each feature are as follows:
The following is a detailed analysis of the logic of the code through comments in the source code::::
/*
* icvCreateIntHaarFeatures
*
* Create internal representation of haar features
*
* mode:
* 0 - BASIC = Viola
* 1 - CORE = All upright
* 2 - ALL = All features
*/
static
CvIntHaarFeatures* icvCreateIntHaarFeatures( CvSize winsize,
int mode,
int symmetric )
{
CvIntHaarFeatures* features = NULL;
CvTHaarFeature haarFeature;
/*内存存储器是一个可用来存储诸如序列,轮廓,图形,子划分等动态增长数据 结构的底层结构。
它是由一系列以同等大小的内存块构成,呈列表型*/
CvMemStorage* storage = NULL;
CvSeq* seq = NULL;
CvSeqWriter writer;
int s0 = 36; /* minimum total area size of basic haar feature */
int s1 = 12; /* minimum total area size of tilted haar features 2 */
int s2 = 18; /* minimum total area size of tilted haar features 3 */
int s3 = 24; /* minimum total area size of tilted haar features 4 */
int x = 0;
int y = 0;
int dx = 0;
int dy = 0;
float factor = 1.0F;
factor = ((float) winsize.width) * winsize.height / (24 * 24);
#if 0
s0 = (int) (s0 * factor);
s1 = (int) (s1 * factor);
s2 = (int) (s2 * factor);
s3 = (int) (s3 * factor);
#else
s0 = 1;
s1 = 1;
s2 = 1;
s3 = 1;
#endif
/* CV_VECTOR_CREATE( vec, CvIntHaarFeature, size, maxsize ) */
storage = cvCreateMemStorage();/*创建一个内存存储器*/
cvStartWriteSeq( 0, sizeof( CvSeq ), sizeof( haarFeature ), storage, &writer );
/*定义了writer工具每次写入数据的大小，以及写入到哪个内存存储器。
在之后调用 CV_WRITE_SEQ_ELEM( haarFeature, writer )时就可以自动将一个haarFeature类型的数据写入内存存储器中*/
for( x = 0; x < winsize.width; x++ )
{
for( y = 0; y < winsize.height; y++ )/*x，y确定了特征矩形的左上角坐标，所以这是遍历整个窗口*/
{
for( dx = 1; dx <= winsize.width; dx++ )
{/*dx，dy确定了特征矩形的大小，所以这是遍历不矩形大小的特征*/
for( dy = 1; dy <= winsize.height; dy++ )
{/*接下来是针对不同的模版分别计算特征*/
// haar_x2
if ( (x+dx*2 <= winsize.width) && (y+dy <= winsize.height) ) {
if (dx*2*dy < s0) continue;
if (!symmetric || (x+x+dx*2 <=winsize.width)) {/*haar_x2::对应上述haar_x2图特征Sum黑-Sum白*/
haarFeature = cvHaarFeature( "haar_x2",
x, y, dx*2, dy, -1,
x+dx, y, dx , dy, +2 );
/* CV_VECTOR_PUSH( vec, CvIntHaarFeature, haarFeature, size, maxsize, step ) */
CV_WRITE_SEQ_ELEM( haarFeature, writer );
}
}
// haar_y2
if ( (x+dx <= winsize.width) && (y+dy*2 <= winsize.height) ) {
if (dx*2*dy < s0) continue;
if (!symmetric || (x+x+dx <= winsize.width)) {/*haar_y2::对应上图图haar_y2特征Sum黑-Sum白*/
haarFeature = cvHaarFeature( "haar_y2",
x, y, dx, dy*2, -1,
x, y+dy, dx, dy, +2 );
CV_WRITE_SEQ_ELEM( haarFeature, writer );
}
}
// haar_x3
if ( (x+dx*3 <= winsize.width) && (y+dy <= winsize.height) ) {
if (dx*3*dy < s0) continue;
if (!symmetric || (x+x+dx*3 <=winsize.width)) {/*haar_x3::对应上图图haar_x3特征，Sum黑*2-Sum白左-Sum白右*/
haarFeature = cvHaarFeature( "haar_x3",
x, y, dx*3, dy, -1,
x+dx, y, dx, dy, +3 );
CV_WRITE_SEQ_ELEM( haarFeature, writer );
}
}
// haar_y3
if ( (x+dx <= winsize.width) && (y+dy*3 <= winsize.height) ) {
if (dx*3*dy < s0) continue;
if (!symmetric || (x+x+dx <= winsize.width)) {/*haar_y3::对应上图图haar_y3特征，Sum黑*2-Sum白上-Sum白下*/
haarFeature = cvHaarFeature( "haar_y3",
x, y, dx, dy*3, -1,
x, y+dy, dx, dy, +3 );
CV_WRITE_SEQ_ELEM( haarFeature, writer );
}
}
if( mode != 0 /*BASIC*/ ) {
// haar_x4
if ( (x+dx*4 <= winsize.width) && (y+dy <= winsize.height) ) {
if (dx*4*dy < s0) continue;
if (!symmetric || (x+x+dx*4 <=winsize.width)) {/*haar_x4::对应上图图haar_x4特征，Sum黑-Sum白左-Sum白右*/
haarFeature = cvHaarFeature( "haar_x4",
x, y, dx*4, dy, -1,
x+dx, y, dx*2, dy, +2 );
CV_WRITE_SEQ_ELEM( haarFeature, writer );
}
}
// haar_y4
if ( (x+dx <= winsize.width ) && (y+dy*4 <= winsize.height) ) {
if (dx*4*dy < s0) continue;
if (!symmetric || (x+x+dx <=winsize.width)) {/*haar_y4::对应上图图haar_y4特征，Sum黑-Sum白上-Sum白下*/
haarFeature = cvHaarFeature( "haar_y4",
x, y, dx, dy*4, -1,
x, y+dy, dx, dy*2, +2 );
CV_WRITE_SEQ_ELEM( haarFeature, writer );
}
}
}
// x2_y2
if ( (x+dx*2 <= winsize.width) && (y+dy*2 <= winsize.height) ) {
if (dx*4*dy < s0) continue;
if (!symmetric || (x+x+dx*2 <=winsize.width)) {/*haar_x2_y2::对应上图图haar_x2_y2特征，Sum黑左上-Sum白上右-Sum白下左+Sum黑右下*/
haarFeature = cvHaarFeature( "haar_x2_y2",
x , y, dx*2, dy*2, -1,
x , y , dx , dy, +2,
x+dx, y+dy, dx , dy, +2 );
CV_WRITE_SEQ_ELEM( haarFeature, writer );
}
}
if (mode != 0 /*BASIC*/) {
// point
if ( (x+dx*3 <= winsize.width) && (y+dy*3 <= winsize.height) ) {
if (dx*9*dy < s0) continue;
if (!symmetric || (x+x+dx*3 <=winsize.width)) {/*point::对应上图图point特征，Sum黑中*8-Sum白*/
haarFeature = cvHaarFeature( "haar_point",
x , y, dx*3, dy*3, -1,
x+dx, y+dy, dx , dy , +9);
CV_WRITE_SEQ_ELEM( haarFeature, writer );
}
}
}
if (mode == 2 /*ALL*/) {
// tilted haar_x2 (x, y, w, h, b, weight)
if ( (x+2*dx <= winsize.width) && (y+2*dx+dy <= winsize.height) && (x-dy>= 0) ) {
if (dx*2*dy < s1) continue;
if (!symmetric || (x <= (winsize.width / 2) )) {/*tilted haar_x2::对应上图图tilted haar_x2特征，在tilted积分图中求得*/
haarFeature = cvHaarFeature( "tilted_haar_x2",
x, y, dx*2, dy, -1,
x, y, dx , dy, +2 );
CV_WRITE_SEQ_ELEM( haarFeature, writer );
}
}
// tilted haar_y2 (x, y, w, h, b, weight)
if ( (x+dx <= winsize.width) && (y+dx+2*dy <= winsize.height) && (x-2*dy>= 0) ) {
if (dx*2*dy < s1) continue;
if (!symmetric || (x <= (winsize.width / 2) )) {/*tilted haar_y2::对应上图图tilted haar_y2特征，在tilted积分图中求得*/
haarFeature = cvHaarFeature( "tilted_haar_y2",
x, y, dx, 2*dy, -1,
x, y, dx, dy, +2 );
CV_WRITE_SEQ_ELEM( haarFeature, writer );
}
}
// tilted haar_x3 (x, y, w, h, b, weight)
if ( (x+3*dx <= winsize.width) && (y+3*dx+dy <= winsize.height) && (x-dy>= 0) ) {
if (dx*3*dy < s2) continue;
if (!symmetric || (x <= (winsize.width / 2) )) {/*tilted haar_x3::对应上图图tilted haar_x3特征，在tilted积分图中求得*/
haarFeature = cvHaarFeature( "tilted_haar_x3",
x, y, dx*3, dy, -1,
x+dx, y+dx, dx , dy, +3 );
CV_WRITE_SEQ_ELEM( haarFeature, writer );
}
}
// tilted haar_y3 (x, y, w, h, b, weight)
if ( (x+dx <= winsize.width) && (y+dx+3*dy <= winsize.height) && (x-3*dy>= 0) ) {
if (dx*3*dy < s2) continue;
if (!symmetric || (x <= (winsize.width / 2) )) {/*tilted haar_y3::对应上图图tilted haar_y3特征，在tilted积分图中求得*/
haarFeature = cvHaarFeature( "tilted_haar_y3",
x, y, dx, 3*dy, -1,
x-dy, y+dy, dx, dy, +3 );
CV_WRITE_SEQ_ELEM( haarFeature, writer );
}
}
// tilted haar_x4 (x, y, w, h, b, weight)
if ( (x+4*dx <= winsize.width) && (y+4*dx+dy <= winsize.height) && (x-dy>= 0) ) {
if (dx*4*dy < s3) continue;
if (!symmetric || (x <= (winsize.width / 2) )) {/*同上*/
haarFeature = cvHaarFeature( "tilted_haar_x4",
x, y, dx*4, dy, -1,
x+dx, y+dx, dx*2, dy, +2 );
CV_WRITE_SEQ_ELEM( haarFeature, writer );
}
}
// tilted haar_y4 (x, y, w, h, b, weight)
if ( (x+dx <= winsize.width) && (y+dx+4*dy <= winsize.height) && (x-4*dy>= 0) ) {
if (dx*4*dy < s3) continue;
if (!symmetric || (x <= (winsize.width / 2) )) {/*同上*/
haarFeature = cvHaarFeature( "tilted_haar_y4",
x, y, dx, 4*dy, -1,
x-dy, y+dy, dx, 2*dy, +2 );
CV_WRITE_SEQ_ELEM( haarFeature, writer );
}
}
/*
// tilted point
if ( (x+dx*3 <= winsize.width - 1) && (y+dy*3 <= winsize.height - 1) && (x-3*dy>= 0)) {
if (dx*9*dy < 36) continue;
if (!symmetric || (x <= (winsize.width / 2) )) {/*tilted point::对应上图图tilted point特征，在tilted积分图中求得*/
haarFeature = cvHaarFeature( "tilted_haar_point",
x, y, dx*3, dy*3, -1,
x, y+dy, dx , dy, +9 );
CV_WRITE_SEQ_ELEM( haarFeature, writer );
}
}
*/
}
}
}
}
}
seq = cvEndWriteSeq( &writer );
features = (CvIntHaarFeatures*) cvAlloc( sizeof( CvIntHaarFeatures ) +
( sizeof( CvTHaarFeature ) + sizeof( CvFastHaarFeature ) ) * seq->total );
features->feature = (CvTHaarFeature*) (features + 1);
features->fastfeature = (CvFastHaarFeature*) ( features->feature + seq->total );
features->count = seq->total;
features->winsize = winsize;
cvCvtSeqToArray( seq, (CvArr*) features->feature );
cvReleaseMemStorage( &storage );
/*特征的rect由坐标表示转换为由像素索引表示*/
icvConvertToFastHaarFeature( features->feature, features->fastfeature,
features->count, (winsize.width + 1) );
return features;
}