OpenGL.Shader: Zhige teaches you to write a live filter client(9)
In the last article, I learned about convolution and wrote a simple pull sharpening filter. This chapter records the blur algorithm-mean blur, also called mean filtering, also called box blur (BoxBlur).
1. Signal and noise
Before talking about blur, I feel it is necessary to learn the concept of signal and noise in computer vision . If we feel that there are too many noises while making a call, then there is a lot of noise in the call data at this time, which has already reached the level of affecting normal calls. Even when the noise is particularly loud, the signal is easily submerged in the noise. The image is also a kind of data, and there are also signals and noise in the image. In layman's terms, signal and noise are a pair of enemies, and the space of the image is limited. The more the signal, the less the noise, and vice versa.
1.1 Signal
Signals are a good thing, because this is the data we want. The more signals there are, the less noise interference will be and the higher the quality of the data. We can use the concept of signal-to-noise ratio to measure the quality of data. The so-called signal-to-noise ratio refers to the ratio of energy between signal and noise. Intuitively speaking, the less noise, the greater the signal-to-noise ratio, and the better the quality of the data. We can see that picture a shows the image after Gaussian noise interference, while picture b is the original image without noise interference.
1.2 Noise
In the above picture, you have seen the difference between the two pictures that have been subjected to noise interference and that that have not been subjected to noise interference. The image with noise interference makes it difficult for us to obtain the original information expressed by the picture, so that the degree of certainty of the information expressed by the image is reduced, that is, the so-called information entropy increases.
In real life, the pictures acquired by the image acquisition device will more or less introduce noise, which is mainly caused by the noise generated by the interference of the photosensitive element of the image acquisition device such as the camera. The main performance is For black and white noise, etc.
The black and white noise that appears randomly in the image is called salt and pepper noise. "Pepper" stands for black and "salt" stands for white . Therefore, the concept of salt and pepper noise is used to represent the black and white noise in the image, and its position in the picture is random of. Some color changes may also appear randomly in the image. The most typical cause of such noise is Gaussian noise, which is caused by Gaussian noise superimposed on the original picture.
The so-called Gaussian noise means that the noise probability density of the image superimposed obeys the Gaussian distribution, that is, the normal distribution . This is the most common type of noise in nature. For example, there may be this type of noise in photos taken with a camera at night.
2. Image filtering
As mentioned earlier, noise is a type of data we don't want. However, noise is often introduced in actual operation. For example, the picture is transmitted through a low-quality channel, which introduces the noise existing in the channel; the image acquisition equipment introduces noise due to some electronic reasons. The presence of noise will inevitably cause interference to our normal image processing . Filtering out as much noise as possible is an important step in our image preprocessing. The following describes a common method of filtering noise-mean filtering.
The mean filter mentioned here is more precisely an arithmetic mean filter, which is the simplest kind of image low-pass filter, which can filter out uniform noise and Gaussian noise, but will cause a certain degree of blur to the image. It is a method of averaging the pixels in the specified area in the picture, as shown in the figure below.
The disadvantage of the averaging filter is that it will blur the image, because it averages all the points. In fact, in most cases, the proportion of noise is small, and all points are processed with the same weight, which will inevitably lead to blurred images. Moreover, the larger the width of this filter, the more blurred the filtered picture, which means that the details of the image are lost, making the image more "moderate".
Of course, according to this feature, the weight of this filter can also be changed to achieve a focused effect. A feasible way to achieve this is to modify the parameters of this filter in accordance with the Gaussian distribution, then this filter is called a Gaussian filter, which will be studied in the next chapter.
A set of concepts is interspersed here: Why high-pass filter (HPF) and low-pass filter (LPF)?
High-pass filter: Improve the brightness of the modified pixel according to the brightness difference between the pixel and the surrounding pixels.
The main function is sharpening, but it will bring noise.
Low-pass filter: When the brightness difference between the pixel and the surrounding pixels is less than a certain value, the brightness of the pixel is smoothly modified.
The main function is blurring/denoising.
3. Simple implementation of GPU.Shader mean filtering
The next step is to implement the mean filtering on the OpenGL shader, not to mention nonsense, just upload the code:
attribute vec4 position;
attribute vec4 inputTextureCoordinate;
uniform float widthFactor; // width factor = 1 / screen width
uniform float heightFactor; // height factor = 1 / screen height
uniform float blurSize; // center offset, combined with screen width and height factor dynamically adjusting the average filter convolution kernel size
VARYING vec2 TextureCoordinate;
VARYING vec2 leftTextureCoordinate;
VARYING vec2 rightTextureCoordinate;
VARYING vec2 topTextureCoordinate;
VARYING vec2 topLeftTextureCoordinate;
VARYING vec2 topRightTextureCoordinate;
VARYING vec2 bottomTextureCoordinate;
VARYING vec2 bottomLeftTextureCoordinate;
VARYING vec2 bottomRightTextureCoordinate;
//. 9 The coordinate position corresponding to the grid matrix.
void main()
{ gl_Position = position;
vec2 widthStep = vec2(blurSize*widthFactor, 0.0);
vec2 heightStep = vec2(0.0, blurSize*heightFactor);
// factor大小为屏幕长宽的倒数，根据blurSize可以动态调节step的偏移大小
textureCoordinate = inputTextureCoordinate.xy;
leftTextureCoordinate = inputTextureCoordinate.xy - widthStep;
rightTextureCoordinate = inputTextureCoordinate.xy + widthStep;
topTextureCoordinate = inputTextureCoordinate.xy - heightStep;
topLeftTextureCoordinate = inputTextureCoordinate.xy - heightStep - widthStep;
topRightTextureCoordinate = inputTextureCoordinate.xy - heightStep + widthStep;
bottomTextureCoordinate = inputTextureCoordinate.xy + heightStep;
bottomLeftTextureCoordinate = inputTextureCoordinate.xy + heightStep - widthStep;
bottomRightTextureCoordinate = inputTextureCoordinate.xy + heightStep + widthStep;
}
The vertex shader is responsible for generating the vertex index positions required by the 9-square grid of the convolution matrix. The convolution operation is given to the following fragment shader
precision highp float;
uniform sampler2D SamplerY;
uniform sampler2D SamplerU;
uniform sampler2D SamplerV;
mat3 colorConversionMatrix = mat3(
1.0, 1.0, 1.0,
0.0, -0.39465, 2.03211,
1.13983, -0.58060, 0.0);
vec3 yuv2rgb(vec2 pos)
{
vec3 yuv;
yuv.x = texture2D(SamplerY, pos).r;
yuv.y = texture2D(SamplerU, pos).r - 0.5;
yuv.z = texture2D(SamplerV, pos).r - 0.5;
return colorConversionMatrix * yuv;
}
varying vec2 textureCoordinate;
varying vec2 leftTextureCoordinate;
varying vec2 rightTextureCoordinate;
varying vec2 topTextureCoordinate;
varying vec2 topLeftTextureCoordinate;
varying vec2 topRightTextureCoordinate;
varying vec2 bottomTextureCoordinate;
varying vec2 bottomLeftTextureCoordinate;
varying vec2 bottomRightTextureCoordinate;
void main()
{
mediump vec3 textureColor = yuv2rgb(textureCoordinate);
mediump vec3 leftTextureColor = yuv2rgb(leftTextureCoordinate);
mediump vec3 rightTextureColor = yuv2rgb(rightTextureCoordinate);
mediump vec3 topTextureColor = yuv2rgb(topTextureCoordinate);
mediump vec3 topLeftTextureColor = yuv2rgb(topLeftTextureCoordinate);
mediump vec3 topRightTextureColor = yuv2rgb(topRightTextureCoordinate);
Vec3 bottomTextureColor = YUV2RGB mediump (bottomTextureCoordinate);
mediump Vec3 bottomLeftTextureColor = YUV2RGB (bottomLeftTextureCoordinate);
mediump Vec3 bottomRightTextureColor = YUV2RGB (bottomRightTextureCoordinate);
// color values lifted, and then weighted average, to remember the last output divided. 9
Vec3 fragmentColor = (leftTextureColor + textureColor + rightTextureColor);
fragmentColor += (topLeftTextureColor + topTextureColor + topRightTextureColor);
fragmentColor += (bottomLeftTextureColor + bottomTextureColor + bottomRightTextureColor);
gl_FragColor = vec4(fragmentColor/9.0, 1.0);
}
The above is a relatively intuitive and simple implementation of mean filtering. Compared with the Laplacian sharpening in the previous article, it should be easier to understand.
The video practice effect is converted into a gif with very low recognition, so I won't put it out here. If you are interested, you can run the demo project to see the effect.
Project address: https://github.com/MrZhaozhirong/NativeCppApp Mean Blur Filter cpp/gpufilter/filter/GpuBoxBlurFilter.hpp
That is All.
Interest discussion group: 703531738. Code: Zhige 13567
References: