OpenGL.Shader: Zhige teaches you to write a filter live client (12) Visual filter: the realization of the principle of bilateral filtering of skin whitening

OpenGL.Shader: Zhige teaches you to write a live filter client(12)

 

The work and life arrangements are too full, and I emptied my blog for a period of time. I have only recently had time to continue studying the filters. This article brings a more popular filter effect dermabrasion-simple learning of bilateral filtering.

1. What is bilateral filtering?

Let’s take a look at the more official explanation: Bilateral filter is a non-linear filtering method that combines the spatial proximity of the image and the similarity of the pixel value. It also considers spatial information and grayscale. Similarity, to achieve the purpose of edge preservation and noise removal. It is simple, non-iterative, and partial.

You may not understand "non-linearity" at first glance. In fact, the mean filter/Gaussian filter introduced above is a linear filter, which is simply understood as: for all pixel values ​​of a picture, the same filter matrix is ​​used, and the convolution is performed according to a fixed ratio of weight coefficients. Non-linear filtering is to add a parameter to determine whether adjacent pixels are similar, so that the weight coefficient in the convolution process is no longer a fixed ratio, but is dynamically adjusted on demand.

After combining the above understanding of the vernacular, the following international practice starts with the formula of bilateral filtering.

g(i, j) represents the output result;
S(i, j) refers to the (2N+1)(2N+1) convolution operation centered on (i,j); N is the radius of the convolution matrix The length
(k, l) represents the input point(s) in the range; f(k, l) is the value corresponding to the point (k, l).
w(i, j, k, l) represents the value calculated by two Gaussian functions (note: this is not the final weight)

We transform the above formula, assuming that w(i,j,k,l) ​​in the formula is m, then

Let m1+m2+m3… +mn = M, then we

can see that this is obviously a convolution operation of the image matrix and the kernel. Among them, m1/M represents the weight of the first point (or the last point, see how to implement it later), and the image matrix and the kernel are weighted and summed through the convolution operator to finally get the output value.

Next, we will discuss the most critical w(i, j, k, l). In fact, w(i, j, k, l) = ws * wr.

 

Let me talk about WS first. The space is close to the Gaussian function, also called the domain of definition. If you observe carefully, it is the normal distribution model of Gaussian filtering . It represents the mathematical model of the convolution performed in a specific space. The schematic diagram is as follows. If we perform the convolution of this mathematical model on the entire image, it is ordinary Gaussian filtering.

       

 

 

 

Let’s talk about WR, pixel value similarity Gaussian function, also called value domain or frequency domain. The logic diagram is as follows, which means the current input point (k, l) value f (k, l) and output point (i, j) The difference between the value f (i, j). The larger the difference, the smaller wr tends to 0; the smaller the difference, the larger wr tends to 1. If f(i,j) = f(k,l), wr=1.

Key understanding: It is to compare the difference between the value f(k,l) of the current point and the value f(i,j) of the input point. In image processing, it is to compare the pixel value and the coordinate of the coordinate (i,j) as (K, l) pixel value to determine whether the edges are similar.

 

 2. Bilateral filtering in GL

The realization of bilateral filtering on OpenGL.Shader does not lie in algorithms, but in ideas. The previous article introduced the calculation of multiple FBOs to achieve Gaussian filter dimensionality reduction, which may be difficult to understand. This time, implementing bilateral filtering will start from simplicity and go from simplicity. Strive to let everyone understand the idea, and after getting the code, you can "change it yourself".

The first is the vertex shader:

attribute vec4 position; 
attribute vec4 inputTextureCoordinate; 
const int GAUSSIAN_SAMPLES = 9; 
uniform vec2 singleStepOffset; 
varying vec2 textureCoordinate; 
varying vec2 blurCoordinates[GAUSSIAN_SAMPLES]; 
 
void main() 
{ 
    gl_Position = position; 
    textureCoordinate = inputTextureCoordinate.xy; 
     
    int multiplier = 0; 
    vec2 blurStep; 
    for (int i = 0; i < GAUSSIAN_SAMPLES; i++) 
    { 
        multiplier = (i - ((GAUSSIAN_SAMPLES - 1) / 2)); 
        blurStep = float(multiplier) * singleStepOffset; 
        blurCoordinates[i] = inputTextureCoordinate.xy + blurStep; 
    } 
}

Bilateral filtering is basically the same as the previous Gaussian filtering. It also takes 9 sampling points and directly claims a singleStepOffset offset step of vec2. Calculate the vertices of the first four steps and the last four steps of the input point. There is nothing to say, let's focus on the fragment shader.

uniform sampler2D SamplerY; 
uniform sampler2D SamplerU; 
uniform sampler2D SamplerV; 
uniform sampler2D SamplerRGB; 
mat3 colorConversionMatrix = mat3( 
                   1.0, 1.0, 1.0, 
                   0.0, -0.39465, 2.03211, 
                   1.13983, -0.58060, 0.0); 
vec3 yuv2rgb(vec2 pos) 
{ 
   vec3 yuv; 
   yuv.x = texture2D(SamplerY, pos).r; 
   yuv.y = texture2D(SamplerU, pos).r - 0.5; 
   yuv.z = texture2D(SamplerV, pos).r - 0.5; 
   return colorConversionMatrix * yuv; 
} 
// yuv转rgb
const lowp int GAUSSIAN_SAMPLES = 9; 
varying highp vec2 textureCoordinate; 
varying highp vec2 blurCoordinates[GAUSSIAN_SAMPLES]; 
uniform mediump float distanceNormalizationFactor; 
 
void main() 
{ 
    lowp vec4 centralColor; // Input the pixel value of the center point 
    lowp float gaussianWeightTotal; // Gaussian weight set
    lowp vec4 sampleSum; // Convolution and
    lowp vec4 sampleColor ; // sample point pixel value
    lowp float gaussianWeight; // sample point Gaussian weight
    lowp float distanceFromCentralColor; 
     
    centralColor = vec4(yuv2rgb(blurCoordinates[4]), 1.0); 
    gaussianWeightTotal = 0.22; 
    sampleSum = centralColor * 0.22; 
     
    sampleColor = vec4( yuv2rgb(blurCoordinates[0]), 1.0); 
    distanceFromCentralColor = min(distance(centralColor, sampleColor) * distanceNormalizationFactor, 1.0); 
    gaussianWeight = 0.03 * (1.0 - distanceFromCentralColor); 
    gaussianWeightTotal += gaussianWeight; 
    sampleSum += sampleColor * gaussianWeight; 
     
    sampleColor = vec4(yuv2rgb(blurCoordinates[1]), 1.0); 
    distanceFromCentralColor = min(distance(centralColor, sampleColor) * distanceNormalizationFactor, 1.0); 
    gaussianWeight = 0.07 * (1.0 - distanceFromCentralColor); 
    gaussianWeightTotal += gaussianWeight; 
    sampleSum += sampleColor * gaussianWeight; 
    
    sampleColor = vec4(yuv2rgb(blurCoordinates[2]), 1.0); 
    distanceFromCentralColor = min(distance(centralColor, sampleColor) * distanceNormalizationFactor, 1.0); 
    gaussianWeight = 0.12 * (1.0 - distanceFromCentralColor); 
    gaussianWeightTotal += gaussianWeight; 
    sampleSum += sampleColor * gaussianWeight; 
     
    sampleColor = vec4(yuv2rgb(blurCoordinates[3]), 1.0); 
    distanceFromCentralColor = min(distance(centralColor, sampleColor) * distanceNormalizationFactor, 1.0); 
    gaussianWeight = 0.17 * (1.0 - distanceFromCentralColor); 
    gaussianWeightTotal += gaussianWeight; 
    sampleSum += sampleColor * gaussianWeight; 
     
    sampleColor = vec4(yuv2rgb(blurCoordinates[5]), 1.0); 
    distanceFromCentralColor = min(distance(centralColor, sampleColor) * distanceNormalizationFactor, 1.0); 
    gaussianWeight = 0.17 * (1.0 - distanceFromCentralColor); 
    gaussianWeightTotal += gaussianWeight; 
    sampleSum += sampleColor * gaussianWeight; 
     
    sampleColor = vec4(yuv2rgb(blurCoordinates[6]), 1.0); 
    distanceFromCentralColor = min(distance(centralColor, sampleColor) * distanceNormalizationFactor, 1.0); 
    gaussianWeight = 0.12 * (1.0 - distanceFromCentralColor); 
    gaussianWeightTotal += gaussianWeight; 
    sampleSum += sampleColor * gaussianWeight; 
     
    sampleColor = vec4(yuv2rgb(blurCoordinates[7]), 1.0); 
    distanceFromCentralColor = min(distance(centralColor, sampleColor) * distanceNormalizationFactor, 1.0); 
    gaussianWeight = 0.07 * (1.0 - distanceFromCentralColor); 
    gaussianWeightTotal += gaussianWeight; 
    sampleSum += sampleColor * gaussianWeight; 
     
    sampleColor = vec4(yuv2rgb(blurCoordinates[8]), 1.0); 
    distanceFromCentralColor = min(distance(centralColor, sampleColor) * distanceNormalizationFactor, 1.0); 
    gaussianWeight = 0.03 * (1.0 - distanceFromCentralColor); 
    gaussianWeightTotal += gaussianWeight; 
    sampleSum += sampleColor * gaussianWeight; 
     
    gl_FragColor = sampleSum / gaussianWeightTotal; 
}

Looking at a long section of shader code, it is actually a set of template code. First look at the two parts of the code:

centralColor = vec4(yuv2rgb(blurCoordinates[4]), 1.0); 
gaussianWeightTotal = 0.22; 
sampleSum = centralColor * 0.22; 

// ... ...

gl_FragColor = sampleSum / gaussianWeightTotal 

If we omit the convolution of the outer 8 sampling points, the output value = the pixel value of the input center point, keeping the original input image effect. Then we add the sampling point convolution, taking the blurCoordinates[3] and blurCoordinates[5] of a singleStepOffset outside the center point as an example to analyze the sampling point code logic:

sampleColor = vec4(yuv2rgb(blurCoordinates[3]), 1.0); 
distanceFromCentralColor = min(distance(centralColor, sampleColor) * distanceNormalizationFactor, 1.0); 
gaussianWeight = 0.17 * (1.0 - distanceFromCentralColor); 
gaussianWeightTotal += gaussianWeight; 
sampleSum += sampleColor * gaussianWeight; 

The first line of code , get the pixel value of the sampling point.
(Key point) The second line of code uses the GLSL built-in function disatance to calculate the distance between two vec2/3/4, which can also be understood as the difference between two variables. The distance function can be used to calculate the similarity of two colors. The larger the result , The greater the difference between the two colors, the smaller the result, and the smaller the difference between the two colors. Then multiply by the custom distanceNormalizationFactor difference quantization factor. How to understand this factor? In fact, it is a modified parameter that can dynamically change its effect range. If you don't understand, you can see the animation below.

Here is a debugging technique for GLSL. The value that needs to be debugged is directly output to gl_FragColor for display, and the effect is the most straightforward to observe with eyes.

gl_FragColor = vec4( min(distance(centralColor, sampleColor)*distanceNormalizationFactor, 1.0),   0.0, 0.0, 1.0);

// debug.gif

(Sub-emphasis) The third line of code , combined with the second line of code understanding, uses the GLSL built-in function min, according to the mathematical meaning of bilateral filtering, calculates the difference between the sampling point and the central input point, and normalizes it to one The similarity distanceFromCentralColor, but one thing to note is that the closer the pixel value of the sampling point and the center input point is, the closer the distanceFromCentralColor is to 0, and the closer the Gaussian weight is to the original value.
So the weight involved in the convolution = the original Gaussian weight*(1.0-distanceFromCentralColor) The
fourth and fifth lines of code merge the weights involved in the convolution and perform the convolution operation.

The rest is to rewrite several GpuBaseFilter functions, please refer to https://github.com/MrZhaozhirong/NativeCppApp                       /src/main/cpp/gpufilter/filter/GpuBilateralBlurFilter.hpp for details 

    void setAdjustEffect(float percent) {
        // 动态调整色值阈值参数
        mThreshold_ColorDistanceNormalization = range(percent*100.0f, 10.0f, 1.0f);
    }

    void onDraw(GLuint SamplerY_texId, GLuint SamplerU_texId, GLuint SamplerV_texId,
                void* positionCords, void* textureCords)
    {
        if (!mIsInitialized)
            return;
        glUseProgram(getProgram());
        // 把step offset的步伐直接用vec2表示，其值直接输入1/w，1/h
        glUniform2f(mSingleStepOffsetLocation, 1.0f/mOutputWidth, 1.0f/mOutputHeight);
        glUniform1f(mColorDisNormalFactorLocation, mThreshold_ColorDistanceNormalization);
        // 绘制的模板代码
        glVertexAttribPointer(mGLAttribPosition, 2, GL_FLOAT, GL_FALSE, 0, positionCords);
        glEnableVertexAttribArray(mGLAttribPosition);
        glVertexAttribPointer(mGLAttribTextureCoordinate, 2, GL_FLOAT, GL_FALSE, 0, textureCords);
        glEnableVertexAttribArray(mGLAttribTextureCoordinate);

        glActiveTexture(GL_TEXTURE0);
        glBindTexture(GL_TEXTURE_2D, SamplerY_texId);
        glUniform1i(mGLUniformSampleY, 0);
        glActiveTexture(GL_TEXTURE1);
        glBindTexture(GL_TEXTURE_2D, SamplerU_texId);
        glUniform1i(mGLUniformSampleU, 1);
        glActiveTexture(GL_TEXTURE2);
        glBindTexture(GL_TEXTURE_2D, SamplerV_texId);
        glUniform1i(mGLUniformSampleV, 2);
        // onDrawArraysPre
        glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);
        glDisableVertexAttribArray(mGLAttribPosition);
        glDisableVertexAttribArray(mGLAttribTextureCoordinate);
        glBindTexture(GL_TEXTURE_2D, 0);
    }

 

Let me talk about another knowledge point:

In GpuGaussianBlurFilter, the Gaussian convolution kernel is passed into the Shader from the external code, and its value is

convolutionKernel = new GLfloat[9]{
                0.0947416f, 0.118318f, 0.0947416f,
                0.118318f,  0.147761f, 0.118318f,
                0.0947416f, 0.118318f, 0.0947416f,
        };

In GpuGaussianBlurFilter2, the Gaussian kernel is not imported from the outside, but is written directly in the Shader. Its values ​​are:
0.05 , 0.09, 0.12, 0.15, 0.18, 0.15, 0.12, 0.09, 0.05

In this GpuBilateralBlurFilter, do not know there is a small partner did not pay attention, that the Gaussian kernel is written directly Shader among whose value is:
0.03,0.07,0.12,0.17,0.22,0.17,0.12,0.07,0.03

I step by step to increase the weight of the core value and reduce the weight of the edge, think about why?