I have recently been studying the principles of image compression and read a lot of information, from the upper layer size compression and quality compression principles to the lower layer Huffman compression. I walked on Chenghua Avenue and then went to Erxian Bridge. I read it all. Today I will summarize it. Let me share your technology. The following content may subvert your understanding of image compression.
/ Basic knowledge of pictures /
First look at this section with a few questions:
1. What is the difference between bit depth and color depth? Are they the same thing?
2. Why can’t Bitmap be saved directly? What is the relationship between Bitmap and PNG and JPG?
3. The formula for the memory size occupied by pictures: picture resolution * size of each pixel. Is this correct?
4. Why sometimes the same app, the same picture on the same interface in app > consumes different memory on different devices?
5. For the same picture, if the controls displayed on the interface are of different sizes, will its memory size also change accordingly?
Introduction to ARGB
ARGB color model: The most common color model, device-related, four channels, the values are [0, 255], that is, converted into binary bits 0000 0000 ~ 1111 1111.
A: Alpha (transparency) R: Red (red) G: Green (green) B: Blue (blue)
Bitmap concept
The essence of the Bitmap object is the expression of the content of a picture in the memory of the mobile phone. It regards the content of the image as consisting of a limited number of pixels that store data; each pixel stores the ARGB value of the pixel position. Once the ARGB value of each pixel is determined, the content of this picture is determined accordingly.
color mode
Bitmap.Config is an enumerated internal class of Bitmap, which represents the storage scheme of the ARGB channel value for each pixel. There are four values:
ALPHA_8: Each pixel occupies 8 bits (1 byte), stores transparency information, and has no color information.
RGB_565: No transparency, R=5, G=6, B=5, then one pixel occupies 5+6+5=16 bits (2 bytes) and can represent 2^16 colors.
ARGB_4444: It consists of 4 4-bits, that is, A=4, R=4, G=4, B=4. Then one pixel occupies 4+4+4+4=16 bits (2 bytes) and can represent 2 ^16 colors.
ARGB_8888: It consists of 4 8-bits, that is, A=8, R=8, G=8, B=8. Then one pixel occupies 8+8+8+8=32 bits (4 bytes) and can represent 2 ^24 colors.
Bit depth and color depth
When you check the information of an image on Windows, you will find bit depth, but not color depth:
Here is an introduction to the concepts of bit depth and color depth:
Color depth: As the name suggests, it is the "depth of color", which refers to how many bits are used to store the ARGB value for each pixel. It is an attribute of the image itself. Color depth can be used to measure the color processing capabilities of an image (i.e., the richness of the colors). Typical color depths are 8-bit, 16-bit, 24-bit and 32-bit. The value of the above Bitmap.Config parameter refers to the color depth. For example, the color depth of ARGB_8888 mode is 32 bits, and the color depth of RGB_565 mode is 16 bits. Color depth is a digital image parameter.
Bit depth refers to the number of binary digits required for each pixel when recording the color of a digital image. When these data are recorded in a computer according to a certain arrangement, they form a computer file of a digital image. The number of bits used by each pixel in the computer is the "bit depth". Bit depth is a physical hardware parameter mainly used for storage.
For example: a certain picture is 100 pixels * 100 pixels, has a color depth of 32 bits (ARGB_8888), and the bit depth is 24 bits when saving, then:
-
The size of this image in memory is: 100 * 100 * (32 / 8) Byte
-
The size occupied in the file is 100 * 100 * (24/ 8) * compression rate Byte
Expand your knowledge
24-bit color can be called true color, and the color depth is 24. It can be combined into 2 to the 24th power of colors, that is: 16777216 colors, which exceeds the number of colors that the human eye can distinguish.
The size of the Bitmap in memory
Many articles on the Internet will introduce the formula for calculating the memory size occupied by an image: resolution * size of each pixel, but is this really the case?
We all know that our mobile phone screens have a certain resolution (for example: 1920×1080), and images also have their own pixels (for example, the resolution of captured images is 4032×3024).
It is most appropriate if a 1920×1080 picture is loaded onto a 1920×1080 screen, and the display effect is best at this time.
If you put a 4032×3024 image on a 1920×1080 screen, you will not get a better display effect (the same as the 1920×1080 image display effect), but it will waste more memory. If you press ARGB_8888 To display, 48MB of memory space (404830364 bytes) is required. Such a large memory consumption can easily cause OOM. We will talk about memory optimization for large image loading later, but we will not introduce it in detail here.
In Android's native Bitmap operation, when the image source is a different resource directory in res, the resolution of the image when it is loaded into the memory will go through a layer of conversion, so although the calculation formula of the final image size is still resolution * pixels size, but the resolution at this time is no longer the resolution of the image itself. For details, please see ByteDance interviewer: How to calculate the memory size occupied by a picture. The rules are as follows:
New resolution = horizontal resolution of the original image * (dpi of the device / dpi corresponding to the directory) * vertical resolution of the original image * (dpi of the device / dpi corresponding to the directory).
When using Glide, if there is a control that sets the image display, the resolution of the image will be automatically reduced and loaded according to the size of the control. The resolution conversion rules for images whose source is res are also invalid for it.
When using fresco, no matter where the image comes from, even if it is res, the memory size occupied by the image is still calculated based on the resolution of the original image.
Other image sources, such as disks, files, streams, etc., calculate the memory size of the image based on the resolution of the original image.
So how to calculate the memory occupied by Bitmap?
Let’s look at the source code of BitmapFactory.decodeResource():
BitmapFactory.java
public static Bitmap decodeResourceStream(Resources res, TypedValue value,InputStream is, Rect pad, Options opts) {
if (opts == null) {
opts = new Options();
}
if (opts.inDensity == 0 && value != null) {
final int density = value.density;
if (density == TypedValue.DENSITY_DEFAULT) {
//inDensity默认为图片所在文件夹对应的密度
opts.inDensity = DisplayMetrics.DENSITY_DEFAULT;
} else if (density != TypedValue.DENSITY_NONE) {
opts.inDensity = density;
}
}
if (opts.inTargetDensity == 0 && res != null) {
//inTargetDensity为当前系统密度。
opts.inTargetDensity = res.getDisplayMetrics().densityDpi;
}
return decodeStream(is, pad, opts);
}
BitmapFactory.cpp 此处只列出主要代码。
static jobject doDecode(JNIEnv* env, SkStreamRewindable* stream, jobject padding, jobject options) {
//初始缩放系数
float scale = 1.0f;
if (env->GetBooleanField(options, gOptions_scaledFieldID)) {
const int density = env->GetIntField(options, gOptions_densityFieldID);
const int targetDensity = env->GetIntField(options, gOptions_targetDensityFieldID);
const int screenDensity = env->GetIntField(options, gOptions_screenDensityFieldID);
if (density != 0 && targetDensity != 0 && density != screenDensity) {
//缩放系数是当前系数密度/图片所在文件夹对应的密度；
scale = (float) targetDensity / density;
}
}
//原始解码出来的Bitmap；
SkBitmap decodingBitmap;
if (decoder->decode(stream, &decodingBitmap, prefColorType, decodeMode)
!= SkImageDecoder::kSuccess) {
return nullObjectReturn("decoder->decode returned false");
}
//原始解码出来的Bitmap的宽高；
int scaledWidth = decodingBitmap.width();
int scaledHeight = decodingBitmap.height();
//要使用缩放系数进行缩放，缩放后的宽高；
if (willScale && decodeMode != SkImageDecoder::kDecodeBounds_Mode) {
scaledWidth = int(scaledWidth * scale + 0.5f);
scaledHeight = int(scaledHeight * scale + 0.5f);
}
//源码解释为因为历史原因；sx、sy基本等于scale。
const float sx = scaledWidth / float(decodingBitmap.width());
const float sy = scaledHeight / float(decodingBitmap.height());
canvas.scale(sx, sy);
canvas.drawARGB(0x00, 0x00, 0x00, 0x00);
canvas.drawBitmap(decodingBitmap, 0.0f, 0.0f, &paint);
// now create the java bitmap
return GraphicsJNI::createBitmap(env, javaAllocator.getStorageObjAndReset(),
bitmapCreateFlags, ninePatchChunk, ninePatchInsets, -1);
}
/ Introduction to image compression methods in Android /
Image compression is a very common development scenario in Android. There are two main compression methods: one is quality compression, and the other is downsampling compression.
The former is to change the storage volume of the image without changing the image size, while the latter is to reduce the image size to achieve the same purpose.
quality compression
In Android, quality compression of images is usually implemented as follows:
ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
//quality is 0~100, 0 represents the minimum volume, 100 represents the highest quality, and the corresponding volume is also the largest
bitmap.compress(Bitmap.CompressFormat.JPEG, quality , outputStream);
In the above code, the compression format we selected is CompressFormat.JPEG. In addition, there are two options:
First, CompressFormat.PNG, the PNG format is lossless. It cannot perform quality compression. The quality parameter has no effect and will be ignored, so the file size of the final image saved will not change;
Second, CompressFormat.WEBP, this format is an image format introduced by Google. It saves more space than JPEG. After actual testing, it can be optimized by about 30%.
In some application scenarios, bitmap needs to be converted into ByteArrayOutputStream. You need to decide whether to use CompressFormat.PNG or Bitmap.CompressFormat.JPEG according to the image format you want to compress. In this case, the quality is 100.
Android quality compression logic, the function compress finally comes to a native function after a series of java layer calls, as follows:
//Bitmap.cpp
static jboolean Bitmap_compress(JNIEnv* env, jobject clazz, jlong bitmapHandle,
jint format, jint quality,
jobject jstream, jbyteArray jstorage) {
LocalScopedBitmap bitmap(bitmapHandle);
SkImageEncoder::Type fm;
switch (format) {
case kJPEG_JavaEncodeFormat:
fm = SkImageEncoder::kJPEG_Type;
break;
case kPNG_JavaEncodeFormat:
fm = SkImageEncoder::kPNG_Type;
break;
case kWEBP_JavaEncodeFormat:
fm = SkImageEncoder::kWEBP_Type;
break;
default:
return JNI_FALSE;
}
if (!bitmap.valid()) {
return JNI_FALSE;
}
bool success = false;
std::unique_ptr<SkWStream> strm(CreateJavaOutputStreamAdaptor(env, jstream, jstorage));
if (!strm.get()) {
return JNI_FALSE;
}
std::unique_ptr<SkImageEncoder> encoder(SkImageEncoder::Create(fm));
if (encoder.get()) {
SkBitmap skbitmap;
bitmap->getSkBitmap(&skbitmap);
success = encoder->encodeStream(strm.get(), skbitmap, quality);
}
return success ? JNI_TRUE : JNI_FALSE;
}
You can see that the function encoder->encodeStream(...) was finally called to save the encoding locally. This function calls the skia engine to encode and compress the image. The introduction to skia will be explained later.
Size compression
Nearest Neighbor Resampling
BitmapFactory.Options options = new BitmapFactory.Options();
//Or inDensity is used with inTargetDensity, the algorithm is the same as inSampleSize
options.inSampleSize = 2; //Set the image scaling ratio (width and height), Google recommends using multiples of 2:
Bitmap bitmap = BitmapFactory.decodeFile("xxx.png");
Bitmap compress = BitmapFactory.decodeFile("xxx.png", options);
Let’s focus here on inSampleSize. Literally, it means: "Set the sampling size". Its function is: after setting the value of inSampleSize (int type), if it is set to 4, the width and height will be 1/4 of the original, the width and height will be reduced, and the natural memory will also be reduced.
Referring to the explanation in Google's official documentation, we can see that x (x is a multiple of 2) pixels finally correspond to one pixel. Since the sampling rate is set to 1/2, two pixels generate one pixel.
The neighboring sampling method is relatively crude. One of the pixels is directly selected as the generated pixel, and the other pixel is directly discarded. This causes the picture to become pure green, that is, the red pixels are discarded.
The algorithm used in adjacent sampling is called adjacent point interpolation algorithm.
Bilinear Resampling
Bilinear Resampling is generally used in two ways in Android:
Bitmap bitmap = BitmapFactory.decodeFile("xxx.png");
Bitmap compress = Bitmap.createScaledBitmap(bitmap, bitmap.getWidth()/2, bitmap.getHeight()/2, true);
或者直接使用 matrix 进行缩放
Bitmap bitmap = BitmapFactory.decodeFile("xxx.png");
Matrix matrix = new Matrix();
matrix.setScale(0.5f, 0.5f);
bm = Bitmap.createBitmap(bitmap, 0, 0, bit.getWidth(), bit.getHeight(), matrix, true);
Looking at the source code, you can see that the createScaledBitmap function ultimately uses the second method of matrix for scaling. Bilinear sampling uses a bilinear interpolation algorithm. This algorithm does not directly and roughly select a pixel like the adjacent point interpolation algorithm. It refers to the values of 2x2 points around the corresponding position of the source pixel, takes the corresponding weight according to the relative position, and obtains the target image after calculation.
The bilinear interpolation algorithm has an anti-aliasing function in image scaling and is the simplest and most common image scaling algorithm. When the bilinear interpolation algorithm is used for adjacent 2x2 pixels, the resulting surface will be in the neighborhood. are consistent, but the slopes are not consistent, and the smoothing effect of the bilinear interpolation algorithm may degrade the details of the image. This phenomenon is especially obvious during upsampling.
The advantages of bilinear sampling compared to neighboring sampling are:
Its coefficients can be decimals, not necessarily integers. The effect is particularly obvious under certain compression restrictions.
To deal with the difference in display effect of pictures with more text, bilinear sampling effect is better
There are also bicubic sampling and **Lanczos** sampling, etc. For specific analysis, please refer to the image compression analysis in Android (Part 2) shared by a QQ music guru.
Section summary
In Android, the first two sampling methods can be selected according to the actual situation. If the time requirements are not high, bilinear sampling is preferred to scale the image. If the requirements for image quality are very high and bilinear sampling cannot meet the requirements, you can consider introducing several other algorithms to process images. However, it should be noted that the latter two algorithms use convolution kernels to calculate and generate For pixels, the amount of calculation is relatively large, and Lanczos has the largest amount of calculation. In the actual development process, the algorithm can be selected according to the needs. We often use size compression and quality compression in combination.
Now we are going to enter actual combat, refer to Luban, an Android image compression tool that imitates the compression strategy of WeChat Moments, and enter our next chapter of Luban compression algorithm analysis.
/ Background of Luban Compression /
Luban Compression - Android image compression tool, imitating WeChat Moments compression strategy.
At present, when doing App development, the element of pictures cannot be avoided. However, as the resolution of mobile phone photos increases, image compression has become a very important issue. Any image can be several MB, or even dozens of MB. When such photos are loaded into the app, it is conceivable that loading a few photos will be difficult. For pictures, the mobile phone memory is not enough, which naturally causes OOM. Therefore, Android picture compression is extremely important.
There are many articles about simply cropping and compressing pictures. However, it is difficult to control how much to crop and compress. If the image is over-cropped, the image will be too small, and if the quality is over-compressed, the display effect will be too poor. So I naturally thought about how the App giant WeChat would handle it. Luban reverse-calculated the compression algorithm by sending nearly 100 pictures of different resolutions in WeChat Moments and comparing the original pictures with the WeChat-compressed pictures.
Effect and contrast
Because it is a reverse calculation, the effect is not exactly the same as WeChat, but it is very close to the effect after compression of WeChat Moments. See the comparison below for details!
/ Luban algorithm analysis /
WeChat algorithm analysis
The first step is to perform sampling rate compression;
The second step is to compress the width and height in equal proportions (WeChat limits the maximum length and width or minimum length and width of the original image and thumbnail image);
The third step is to compress the image quality (usually 75 or 70);
The fourth step is to adopt the webP format.
After these four processes, the effect is basically the same as that of WeChat Moments, including file size and display effect.
Luban algorithm analysis
The current steps of Luban compression only occupy the second and third steps of the WeChat algorithm. The algorithm logic is as follows:
Determine whether the image proportion value is within the following range.
-
[1, 0.5625) that is, the image is in the ratio range [1:1 ~ 9:16)
-
[0.5625, 0.5) That is, the image is within the ratio range [9:16 ~ 1:2)
-
[0.5, 0) that is, the image is in the ratio range [1:2 ~ 1:∞)
Brief explanation: Get the proportion coefficient of the picture. If it is in the interval [1, 0.5625), the picture is in the ratio range [1:1 ~ 9:16), and the ratio is analogous. If this coefficient is less than 0.5, then give it Put it within the ratio range [1:2 ~ 1:∞).
Determine whether the longest side of the image exceeds the boundary value.
-
[1, 0.5625) The boundary values are: 1664 * n (n=1), 4990 * n (n=2), 1280 * pow(2, n-1) (n≥3)
-
[0.5625, 0.5) The boundary value is: 1280 * pow(2, n-1) (n≥1)
-
[0.5, 0) The boundary value is: 1280 * pow(2, n-1) (n≥1)
Step 2: When I went up and took a look, I was confused. What is 1664, what is n, and what is pow. . . It is estimated that only the author himself can understand this writing. In fact, it is to judge whether the longest side of the picture passes the boundary value. This boundary value is an experience value imitating WeChat. That is to say, 1664 and 4990 are both experience values, imitating WeChat's strategy.
As for n, the value returned is the value of options.inSampleSize, which is the coefficient of sampling compression. It is of type int. Google recommends that it be a multiple of 2. Therefore, in order to comply with this suggestion, the code returns an operation of 4 if it is less than 10240. Finally, let’s talk about pow, which is actually (long side/1280). This 1280 is also an empirical value, which is deduced from the reverse. The logic is clear after the explanation. What a trap, hahaha
Calculate the actual side length of the compressed image, based on the calculation result in step 2. If it exceeds a certain boundary value:
-
width / pow(2, n-1)
-
height/ pow(2, n-1)
Step 3: This seems useless. It is better to calculate the actual side length of the compressed image. People have also said that the calculation result in step 2 shall prevail. In fact, it is just for you. At first glance, there are so many steps. Hahahaha, I am bluffing you. Woolen cloth!
Calculate the actual file size of the compressed image, based on the results of steps 2 and 3. The larger the image ratio, the larger the file.
size = (newW * newH) / (width * height) * m；
-
[1, 0.5625) then width & height corresponds to 1664, 4990, 1280 * n (n≥3), m corresponds to 150, 300, 300;
-
[0.5625, 0.5) then width = 1440, height = 2560, m = 200;
-
[0.5, 0) then width = 1280, height = 1280 / scale, m = 500; Note: scale is a proportional value
Step 4: This feels useless. This m should be the compression ratio. But the whole process is to verify whether the size exceeds your expectations after compression. If it exceeds your expectations, the compression will be repeated.
Determine whether the size in step 4 is too small.
-
[1, 0.5625) then the minimum size corresponds to 60, 60, 100
-
[0.5625, 0.5) then the minimum size is 100
-
[0.5, 0) then the minimum size is 100
Step 5: This step is useless, it is also used for subsequent loop compression. This size is calculated above. The value formula corresponding to the minimum size is: size = (newW * newH) / (width * height) * m. The corresponding three values are the three groups divided according to the proportion of the picture above. Then calculated.
Pass the previously obtained values of compressed image width, height, and size into the compression process, and compress the image until the above values are met.
The last step is useless. Just by looking at the words, you can tell that it is for loop compression. Maybe WeChat does this too? Now that you already have expectations, why not just follow them in one step? But how to adjust the cropping coefficient and compression coefficient to achieve the optimal effect? This function has been added to my project. It is still in internal testing and is not open source. It will be open sourced for everyone to use after it is stabilized in the future.
Bringing algorithms into open source code
Let’s look directly at the engine.java class where the algorithm is located:
// Calculate the sampling compression value, which is to imitate WeChat’s experience value, core content
private int computeSize() { //Complete width and length srcWidth = srcWidth % 2 == 1 ? srcWidth + 1 : srcWidth; srcHeight = srcHeight % 2 == 1 ? srcHeight + 1 : srcHeight; // Get the long side and short side int longSide = Math.max(srcWidth, srcHeight ); int shortSide = Math.min(srcWidth, srcHeight); // Get the proportion coefficient of the image, if it is in the interval [1, 0.5625), that is The picture is in [1:1 ~ 9:16) ratio float scale = ((float) shortSide / longSide); // Start to judge what kind of picture is in In the ratio, it is the first step mentioned above if (scale <= 1 && scale > 0.5625) { // Determine whether the longest side of the picture passes the boundary value. This boundary value is an empirical value imitating WeChat, which is the second step mentioned above. if (longSide < 1664) { // What is returned is the value of options.inSampleSize, which is the coefficient of sampling compression. It is int type. Google recommends it to be a multiple of 2. Return 1; a> // The logic above 10240 is not mentioned, it is also an experience value, don’t worry about it, you can adjust it at will return 2; } else if (longSide < 4990) { return 4; } else { return longSide / 1280 == 0 ? 1 : longSide / 1280; } // These judgments are all reverse derivation The experience value can also be said to be a strategy } else if (scale <= 0.5625 && scale > 0.5) { return longSide / 1280 == 0 ? 1 : longSide / 1280; } else { // The proportion of the picture at this time is a long picture, using the strategy of rounding up return (int) Math.ceil(longSide / (1280.0 / scale) ); } } // Image rotation method private Bitmap rotatingImage(Bitmap bitmap, int angle) { Matrix matrix = new Matrix(); // Rotate the incoming bitmap by angle matrix.postRotate(angle ); // Return a new bitmap return Bitmap.createBitmap(bitmap, 0, 0, bitmap.getWidth(), bitmap.getHeight(), matrix, true); } // Compression method, returns a File File compress() throws IOException { // Create an options object BitmapFactory.Options options = new BitmapFactory.Options(); // Get sampling compression The value of options.inSampleSize = computeSize(); // Sample and compress the image and put it into a bitmap. Parameter 1 is the format of the bitmap image. The previous Obtained Bitmap tagBitmap = BitmapFactory.decodeStream(srcImg.open(), null, options); // Create an output stream object ByteArrayOutputStream stream = new ByteArrayOutputStream(); // Determine whether it is a JPG picture if (Checker.SINGLE.isJPG(srcImg.open( ))) { // Checker.SINGLE.getOrientation This method is to detect whether the image has been rotated and correct it tagBitmap = rotatingImage(tagBitmap, Checker.SINGLE.getOrientation (srcImg.open())); } //Quality compression of the image, parameter 1: Determine whether it is in PNG format or by whether there is a transparent channel JPG format, // Parameter 2: The compression quality is fixed at 60, Parameter 3: After compression, the bitmap is written to the byte stream tagBitmap.compress (focusAlpha ? Bitmap.CompressFormat.PNG : Bitmap.CompressFormat.JPEG, 60, stream);
// Recycle the bitmap after use
tagBitmap.recycle();
// Write the image stream to File and then refresh the buffer area, close the file stream and Byte stream
FileOutputStream fos = new FileOutputStream(tagImg);
fos.write(stream.toByteArray());
fos.flush();
fos.close();
stream.close();
return tagImg;
}
/ Luban original framework problem analysis /
Original framework problem analysis
-
There is no pre-judgment of the memory before decoding.
-
Quality Compression Hardcoded 60
-
No image output format selection provided
-
Reasonable parallel compression of multiple files is not supported, and the output order and compression order cannot be guaranteed to be consistent.
-
Detect the file format and image angle and repeatedly create the InputStream multiple times, adding unnecessary overhead and increasing the risk of OOM.
-
Memory leaks may occur, and you need to handle the life cycle reasonably.
-
If the image has a size limit, it can only be compressed repeatedly.
-
The original framework still uses RxJava1.0
Technical transformation plan
-
Before decoding, use the obtained image width and height to calculate the memory usage. If the memory exceeds the limit, use RGB-565 to try decoding.
-
For quality compression, provide an interface for passing in quality coefficients.
-
Supports multiple formats for image output, not limited to File
-
Use coroutines to implement asynchronous compression and parallel compression tasks. You can cancel the coroutines at the appropriate time to terminate the task.
-
Refer to Glide's reuse of byte arrays and InputStream's mark() and reset() to optimize repeated opening overhead.
-
Use LiveData to implement monitoring and automatically log out of monitoring.
-
Calculate the size before compression, and reversely derive the size compression coefficient and mass compression coefficient.
-
RxJava3 and coroutines are now available, but most projects already have thread pools. We need to use the thread pool in the project instead of importing a third-party library and building a thread pool, which causes a waste of resources.
summary
When Luban Compression first came out, it was said to be "probably the image compression algorithm closest to WeChat Moments." However, this library has not been maintained for three or four years. With the iteration of products, WeChat is no longer the same WeChat it used to be. Luban The compressed library will also be updated. Therefore, in order to adapt to the current project, I will compress the images into a new version of the library based on the above technical transformation plan, which is more powerful.
Luban also has a turbo branch. This branch is mainly for compatibility with system versions before Android 7.0 and imports the jni version of libjpeg-turbo.
libjpeg-turbo is an efficient JPEG image processing library written in C. The Android system internally used the non-turbo version of libjpeg before version 7.0, and Huffman encoding was turned off for performance. Systems after 7.0 use the libjpeg-turbo library internally and enable Huffman encoding.
So what is Huffman coding? What is the skio engine mentioned earlier?
/ Explanation of underlying Huffman compression /
In the previous essential basic knowledge of Android image compression, the mentioned Skia is an important part of Android. Huffman compression is mentioned in the analysis of Luban compression algorithm, so what is the relationship between them?
Android Skia graphics engine
Skia is a 2D vector graphics processing function library. After being acquired by Google in 2005 and maintained by itself, it is an image engine implemented in C++. It implements various image processing functions and is widely used in Google's own and other companies' products (such as: Chrome, Firefox, Android, etc.), based on which it is easy to develop image processing functions for operating systems, browsers, etc.
Skia provides basic drawing and simple encoding and decoding functions in Android, and can be connected to other third-party encoding and decoding libraries or hardware encoding and decoding libraries, such as libpng, libjpeg, libgif, etc. Therefore, this function calls bitmap.compress(Bitmap.CompressFormat.JPEG...), which actually calls the libjpeg.so dynamic library for encoding and compression.
The final logic of Android encoding to save pictures is Java layer function → Native function → Skia function → corresponding third library function (such as libjpeg). So skia is like a glue layer used to link various third-party encoding and decoding libraries, but Android will also make some modifications to these libraries, such as modifying the memory management method, etc.
Android used to use a functionally stripped version of libjpeg to some extent. The default method for compressing images was standard huffman instead of optimized huffman. That is to say, the default Huffman table was used and was not based on the actual image. To calculate the corresponding Huffman table, Google considered the performance bottleneck of mobile phones in the early stage. Calculating the weight of the image takes up a lot of CPU resources and is also very time-consuming, because at this time it is necessary to calculate the weight of all pixels argb of the image. This is also One of the reasons why Android's image compression rate is worse than that of iOS.
/ Image compression and Huffman algorithm /
Here is a brief introduction to the Huffman algorithm. The Huffman algorithm is one of the commonly used algorithms in multimedia processing. For example, there may be five values a, b, c, d, e in a file. Their binary expression is:
a. 1010 b. 1011 c. 1100 d. 1101 e. 1110
We can see that the first digit is 1, which is actually wasted. The optimal expression under the fixed-length algorithm is:
a. 010 b. 011 c. 100 d. 101 e. 110
In this way, we can save one bit of loss. So where is the improvement of Huffman's algorithm compared to the fixed-length algorithm? In the Huffman algorithm, we can give weight to the information, that is, weight the information. Suppose a occupies 60%, b occupies 20%, c occupies 20%, d, e are both 0%:
a:010 (60%) b:011 (20%) c:100 (20%) d:101 (0%) e:110 (0%)
In this case, we can use the Huffman tree algorithm to optimize again as:
a:1 b:01 c:00
So the idea is of course to use short codes for letters that appear frequently, use long codes for letters that appear less frequently, and remove those that do not appear. Finally, the Huffman code of abcde corresponds to: 1 01 00
abcde under fixed-length encoding: 010 011 100 101 110. The encoding after using the Huffman tree weighting is 1 01 00. This is the overall idea of the Huffman algorithm (for a detailed introduction to the algorithm, please refer to the Huffman tree and coding explanations and examples).
Therefore, a very important idea of this algorithm is that we must know the weight of each element. If we can know the weight of each element, then we can dynamically generate an optimal Huffman table based on the weight.
But how to get each element? For pictures, it is the weight of argb in each pixel. You can only cycle the pixel information of the entire picture. This is undoubtedly very performance-consuming, so early Android used the default Huffman table. Perform image compression.
/ libjpeg and optimize_coding /
When libjpeg compresses images, there is a parameter called optimize_coding. Regarding this parameter, libjpeg.doc has the following explanation:
TRUE causes the compressor to compute optimal Huffman coding tables
for the image. This requires an extra pass over the data and
therefore costs a good deal of space and time. The default is
FALSE, which tells the compressor to use the supplied or default
Huffman tables. In most cases optimal tables save only a few percent
of file size compared to the default tables. Note that when this is
TRUE, you need not supply Huffman tables at all, and any you do
supply will be overwritten.
It can be seen from the above that if optimize_coding is set to TRUE, the Huffman table will first be calculated based on the image data during the image compression process. Since this calculation consumes significant space and time, the default value is set to FALSE.
So what is the impact of the optimize_coding parameter? After actual testing, Skia's official staff set optimize_coding=TRUE and FALSE respectively for compression, and found that the image size when FALSE was approximately 2 times + that of TRUE. In other words, for pictures with the same file size, the quality of pictures without Huffman encoding will be 2 times + lower than with Huffman encoding.
Starting from Android 7.0 version, the optimize_code flag has been set to TRUE, which means that the image is used to generate the Huffman table by default instead of using the default Huffman table.
The above content is based on the content in Image Compression Analysis in Android (Part 1). I don’t think it can be better than what he wrote. Thanks to the QQ Music technical team. If there is any offense, please contact us immediately to delete it.
Image compression analysis in Android (Part 1):
https://cloud.tencent.com/developer/article/1006307