In the screen using the content encoding IBC + Palette encoding techniques may significantly improve the compression efficiency. This paper describes the contents of the screen Tencent coding optimization techniques to explore and practice: join IBC + Palette encoding toolset and optimize ME module for the screen content. This paper consists of audio and video Tencent video coding laboratory technical director, Wang Shi Tao speeches in LiveVideoStack line share in finishing from.
Text / Wang Shi Tao
Finishing / LiveVideoStack
Live playback:
https://www2.tutormeetplus.com/v2/render/playback?mode=playback&token=e632113e78b14a2eb10ee64a0f16bd07
Hello everyone, I am from Tencent audio and video lab Wang Shi Tao. Next, we will share the road of exploration ground in the screen content optimization and coding techniques with you.
1. screen content encoding
Before the official start sharing First, a brief screen content encoding: The screen image with traditional cameras to capture images differ greatly, the content of the screen image is an electronic device generates images directly from various types of equipment (including but not limited to, computer, mobile terminal) capturing an image display unit; image screen content generally includes the following types: text animated computer graphics, natural images and graphics images and text bound hybrid computer generated.
As mobile devices and smart terminals continue to penetrate people's lives, the contents of the screen image in our lives more and more common, whether desktop collaboration, cloud games, or the second screen, desktop sharing, online education and other fields has its a very wide range of applications, the screen content with the traditional image of the camera images collected what is the difference?
Camera to capture images usually contain noise sensor, and a continuous tone and generally has a very complex textures; image and to the screen, which typically do not contain noise. This is because many of which are computer-generated images, and the color tone of discrete numbers less; In addition, the line screen image more delicate, is relatively sharp edge; more uniform flat area at the same time most of the screen image, repeating pattern of the same block is greater. Traditional video coding scheme is more natural video cameras capture images of the design, the use of hybrid coding structure prediction + conversion. This hybrid coding structure for coding poor results screen content encoding. As shown in the right side of the above figure, the figure of a Word document is a computer-generated screen image, we can clearly see the text from the figure in some regions has become very blurred, and many details are missing, while the edges of the table has a very significant ringing effect. Because the screen image with a big difference for the camera image acquisition comes naturally. We hope to dramatically improve its coding efficiency and image quality advantage of these unique characteristics of good screen image.
2. HEVC-SCC technology
HEVC coding for the screen image to do a lot of optimization coding tools, first briefly explain some of the major technology HEVC-encoded content on the screen. In several versions of HEVC has some optimization techniques for encoding the screen content: for example v1 version of HEVC launched transform skip coding technology, the principle is to choose whether to skip the frequency domain transform coding; and HEVC -RExt addition to transform skip, also increased the residual rotation, and the residual component DPCM prediction SCC cross and other tools; 2016, HEVC launch of the fourth edition of HEVC-SCC, the version before adding a lot based on the specific contents of the screen encoding tools, including an intra block copy, palette mode, the adaptive color conversion adaptive motion vector accuracy and the like.
Then briefly introduce these tools:
Intra block copy: using the reconstructed blocks of the current frame as a prediction block, IBC made within the current motion compensated encoded image;
Edition toner mode: Palette enumeration mode color value of each coded block to generate a color table, and an index for each sample passed to indicate that it belongs to which color in the color table; generating a color decoder in accordance with Rule table and complete the reconstruction work samples by the color table index.
Adaptive color conversion: converting residual adaptation to a different color space, an image block can be directly encoded in the RGB color space, may be converted in coding adaptively to the YCoCg color space to be encoded;
Adaptive motion vector resolution: Since the contents of the encoding process moving image screen pixel level is usually, so we can control the accuracy of the motion vector MV in the Slice level.
3. IBC toner encoding mode Edition
SCC coding tools can significantly improve efficiency compression coding screen content, which palettes to IBC and to enhance the compression efficiency of the most obvious. According to information, to the screen content, IBC can improve compression efficiency more than 30%, if you can join palette mode version on the basis of IBC and then increased by 15%. Next, I would IBC and palette detail.
IBC prediction coding is reconstructed by the encoded image block of the current frame is obtained, above the PDF of FIG screenshot an example, since English is only a combination of 26 kinds of characters from the English, we encoded, it is easy in the current frame picture I completed coded blocks found in the same English character; if the current encoding can refer to these coded image has been encoded block reconstruction, can greatly improve the coding efficiency. For example, FIG. IB two characters on the last line IB two characters can do its upper left corner of the prediction, so the prediction accuracy is very high. Another point, IBC is carried out at PU level, we can design it as a inter-PU, inter mode lets IBC and common inter-prediction mode can be more flexible to connect, for example, a inter-coded CU may have two PU, the use of a conventional inter prediction, using another IBC.
Of course, IBC with respect to the conventional inter prediction there are many different:
IBC reference block are reconstructed pixels before filtering.
If the current picture is used for reference, which is marked as a long-term reference frame; when the entire image decoded, loop filtering the IBC will be added to the current image and the DPB as short-term reference frame.
IBC predicted block and the current CU can not overlap, in order to prevent non-reconstructed samples are used to predict good.
CU prediction block and the current should be in the same same and a Slice Tile.
IBC block vector must be integer pixel accuracy.
In addition to IBC coding, SCC there is another important tool set, which is the palette mode (Palette mode). Mode for the color palette will have a relatively small number of encoded blocks relatively good results, in general, the number of colors is limited screen content, these colors can be composed of a set of colors, i.e. a palette; then decoding reconstruction of each pixel by the color decoder index table, find the corresponding pixel matching color from the palette to complete the reconstruction of the pixel. As shown in the right diagram in FIG CU four colors, the encoder will search for encoding during each pixel from the color table corresponding to the color value of this pixel reconstruction is completed directly; when small number of colors , the majority of the pixel can be realized by the reconstruction of the color table; As a color color table index + embodiment, does not require changes in the pixel quantized reconstruction can be realized, while the palette significantly improve coding efficiency also reduces the complexity of .
While a vast majority of color coding block can be found with its closest color values in the color table, but sometimes circumstances can not find the color table and encoding pixel color is matching the color will appear. In view of this situation, the palette mode adds a condition known as "escape" mode, this mode will not find a matching pixel color coded specifically for inverse quantization operation to complete the reconstruction of the pixels, so as to solve part of the color point Can not find matching color values in turn leads to the problem of color distortion.
The figure shows a flowchart of the overall encoding palette, which includes the following modules: color table generating module, the mode decision module and the histogram calculation module. During the mode selection, if the final use of the selected encoder mode palette, then the color table and encoding parameters are written into the index stream and transmitted to the decoding side.
4. Screen Tencent coding solutions (TSE)
Before we brief the screen content coding technology and some of the tools in HEVC encoding content on the screen. The toolset, intra blocks and copies the contents of the screen to improve encoding palette encoding compression efficiency caused most obvious. The screen of the coding sequence in most cases if IBC + Palette mode, compression efficiency may be achieved with respect to H265 than 50% improvement. For Tencent, Tencent wireless voted screen, Tencent conferencing coordination office products there are a lot of scenarios like desktop sharing, shared content PPT, Word, etc, which are well suited for screen content encoding. . If we can screen content using encoding technology will undoubtedly greatly enhance the compression coding efficiency and improve product competitiveness, so we started the study and implementation of the screen content encoding. Then share with you the results of research Tencent screen coding technology solutions, we have introduced a special encoder optimized for screen content - Tencent screen content encoder (TSE).
TSE encoder mainly to increase the IBC Palette two tools in the original encoder, the lot and the targeted performance optimization, wherein the optimization of several representative are:
1) based on the motion estimation search hash: For IBC motion estimation, using optimization techniques based on the hash table search. The traditional block matching method is based on the SAD, the method takes too large to meet the cast screen products for wireless high-latency requirements. We instead by using a hash search motion estimation method based on conventional motion estimation, by way hash table search can greatly reduce the number of search points required for motion estimation and motion estimation to reduce time-consuming. According to the statistics before us, while improving the speed of this method, the loss caused by compression efficiency can be basically negligible.
2) fast and efficient color table generation algorithm: Palette overall encoding process is time-consuming maximum color table generating module, SCM reference color table generation software algorithm with a high complexity, consuming a very large coding . Our algorithm for generating the color table to simplify a lot of speed to achieve the purpose of. At the same speed, we need to improve the efficiency of the palette because the color palette during coding, coding efficiency depends on the level of build quality color table, so to optimize color table generation algorithm which is a key technical points. How complex is a core technology of the palette color points of the code table is previously generated. To improve coding efficiency , preferably before we had generated color multiplexing table, this is the current color table to predict the color table previously generated, so that substantial savings can be transmitted codeword consumed color table, while significantly enhance the effect palette coding.
3) efficient lookup table algorithm: When the color table is generated, the encoder needs to find the color table to find the closest color of each pixel, how to quickly find the key point that best matches the color palette also affect the encoding performance one. If YUV three variables to search, then we need to build 24 power scale a look-up table 2, which in practical applications is clearly unrealistic. So we have to look-up table algorithm is optimized to significantly reduce its footprint for memory.
4) a large number of early termination algorithm: We have added a lot of fast algorithms on coding mode selection, such as intra, inter-frame after the decision, according to the verdict and cost value that determines whether to encode + IBC coded palette cost calculation, this can save a lot of time.
5) assembly optimized: In addition to the above optimization, many of which we SIMD key modules optimized to significantly improve encoding speed.
5. Results and Prospects
After the above-described series of optimization, the encoder TSE Tencent performance compared to the X265 has been significantly improved: the compression efficiency is both FIG comparative data acquisition sequence to the right camera, the left side of the screen in the content sequence mode LDP . To the screen content of the sequence, the coding efficiency TSE encoder is compared to x265-ultrafast mode 70% performance improvement, with respect x265-medium has nearly 55% compression performance; observed right side camera collecting comparative data sequence, TSE coding efficiency compared to the encoder x265-ultrafast compression modes are also about 20% to enhance performance.
FIG both the compression efficiency is comparative data: content sequence to the screen, without opening the IBC and PLT, TSE average coding time-consuming only about 33% x265-ultrafast of IBC and open PLT, TSE average coding time-consuming about 50% x265-ultrafast of. Whether or coded speed coding efficiency, TSE advantages are very obvious.
TSE has an advantage not only in terms of compression efficiency and coding speed, while improving the subjective quality is also very obvious. The following figure shows the subjective quality of TSE and x265 comparison:
Can be seen at the same rate, the ringing effect obvious character image coding periphery x265 obtained, the human eye is the intuitive feel dirty and blurred images; TSE but employing the resulting encoded character image clarity sharp, clear eye viewing experience more comfortable, subjective experience significantly better. In summary, TSE encoder can bring savings rate encoding, faster and subjective experience optimization.
TSE technology has been achieved in commercial wireless investment in Tencent and Tencent meeting screen, but to enhance the user experience of exploration for us is endless. We expect in the future be able to develop even better encoder to achieve better compression quality and efficiency in the case as quickly as possible, so as to further enhance the user experience.
LiveVideoStack recruitment
LiveVideoStack is recruiting editors / reporters / operators, together with the world's leading multimedia and technical experts and LiveVideoStack younger partners, to promote eco-development of multimedia technology. Please understand post information directly employed in the BOSS search "LiveVideoStack", or the exchange of research and editor of the package through the micro-letter "Tony_Bao_".
LiveVideoStackCon 2019 Beijing is recruiting lecturers, whether you are a technical school or academic faction, also, or industry experts, no matter how small your team, how new, could come to apply to become LiveVideoStackCon lecturer. [Click to read the original text ] For more information on the General Assembly.