01
background
|
|
|
|
|
|---|---|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2. Main usage scenarios of pictures
|
|
|
|
|
|---|---|---|---|
cover picture |
|
Operation chart |
|
|
|||
resource map |
|
Material picture |
|
|
|||
Avatar picture |
|
UGC |
|
3. Evolution of picture formats
-
Before Q3 2020: iQiyi APP mainly uses JPEG format images -
Q3 2020: iQiyi APP will give priority to using WebP format images -
Q3 2022: iQiyi APP will give priority to using HEIC format images -
Q3 2023: iQiyi APP will give priority to using AVIF format images
4. Picture production and consumption panorama
Image format switching seems to be a simple matter of changing the suffix of the image, but it is a very complicated process to change all the massive iQiyi APP content images to the new format, because the images in the iQiyi APP have changed from From production, uploading, production, distribution to final display, there are multiple strategies in the entire production and consumption process. The overall link is very long, and almost every link needs to be involved in transformation.
1) Production process: including AI synthesis, video frame extraction, operation/producer provision, off-site capture and other image sources, with various formats.
2) Upload link: Different producers will upload through different back-end systems, and the scope of transformation is wide.
3) Production process: Uploaded images generally need to be preprocessed using different strategies according to their purpose. For example, for video cover images, the backend system will use dedicated parameters to perform a series of pre-production operations such as format production, proportion adjustment, intelligent cropping, and zooming, etc., to pre-produce video images in multiple formats and sizes for a video content, and convert the already produced The image format and size information is written into the media library entity and provided to each end for use in various scenarios ; for example, the material images such as logos in the APP are uploaded through the content management system, and are only transcoded to generate the specified format. , not produced in various sizes.
4) Distribution link: CDN and APP business back-end interface are responsible for distribution
CDN needs to support on-demand production of image formats and sizes
For sizes that are not within the pre-production range of the backend system, such as AVIF images with non-standard size 666*666 for cover images, you can produce them in real time on demand by specifying the size and format suffix 666_666.avif.
-
The APP business back-end interface needs to return the image URL of the specified format and size based on the characteristics of the front-end. For different business scenarios, different sizes of cover images will be returned. For example, waterfall flow will use 579*772, the editor of the movie and drama channel will use 405*540, and search for S cards will use 1248*624.
02
WebP format implementation practice
1. Background
In March 2020, iQiyi mobile terminal is preparing to fully apply the WebP image format to replace the JPG, PNG and GIF formats that were mainly used at the time to save image CDN costs.
2. Difficulties
The image URLs used in mobile pages are basically returned by the mobile backend interface, and the mobile backend interface has too many services. The cost of modifying these interfaces to return image URLs in WebP format is too high, and if it is replaced in the future, Other image formats have to be transformed again, which is very time-consuming and labor-intensive.
3. Solution
Network-wide adaptive picture architecture (caplist solution for short)
Principle: The mobile image library automatically spells the caplist parameter after the image URL to tell the image CDN the image format supported by the current device, and the image CDN returns the best image format.
-
For example: the original URL of the image is http://...xxx.jpg, and the image request issued by the image library converts it to http://...xxx.jpg?caplist=webp,jpg. The CDN side receives the image request, parses the parameter caplist, and based on the predefined priority (avif > heic > webp > jpg > png), within the range of image formats supported by caplist, attempts to return images in a format with a higher priority. The process is to first check the cache, and then check whether it supports on-demand production. If it is not satisfied, it will be downgraded, and so on. The specific logic is shown in the figure below. If all attempts fail, a 404 (HTTP_NOT_FOUND) error code will be responded and the image library cannot render the specified image. -
Limitations: Try to avoid downgrades that increase the time required for image requests. Therefore, the high-quality image format suitable for the caplist solution should have two characteristics (WebP format complies with this): -
High pre-production ratio (increased first cache query hit rate) -
On-demand production is time-consuming and low (avoiding downgrades and reducing request time-consuming)
4. Online effect
03
HEIC format implementation practice
1. Background
As early as May 2020, when iQiyi first fully used the WebP image format, it had already investigated the possibility of the HEIC image format with higher file compression rate being implemented on the mobile terminal. The evaluation conclusion at that time was:
The model is well compatible: iOS11+ and Android9+ support HEIC format images at the system level, and there is no problem with compatibility.
High file compression rate: With the same image quality, HEIC images are more than 30% smaller than WebP images.
Decoding performance: Hard decoding on the iOS side is faster than WebP and has good performance; however, the Android side only supports soft decoding, which takes 7.75 times longer than WebP and takes up 3.76 times more memory space than WebP.
Due to the poor decoding performance of the HEIC image format in the Android system, the HEIC image format was only piloted on a small scale in specific scenarios on the iOS side.
Two years later, in May 2022, under the background of cost reduction and efficiency improvement, multiple departments cooperated to promote the implementation of the HEIC image format, and increased the proportion of visits to the HEIC image format through various optimization methods.
2. Difficulties
-
Although the Android system supports the HEIC format, the decoding performance is poor, and the performance of the open source HEIC decoder is not ideal, and the overall implementation cannot meet the requirements. -
HEIC图片格式编码性能较差，耗时较长，不能满足CDN按需生产的要求。之前落地WebP格式过程中非常高效成功的caplist方案非常依赖按需生产能力。如果请求的图片没有预生产过HEIC格式，也不支持按需生产，就会直接降级到低优先级的格式，这会导致图片请求耗时增加。因此，caplist方案在HEIC格式落地中不适用，这大大提高了HEIC格式落地的难度和时间成本。 -
已生产的HEIC图片仅覆盖2020年之后生产的部分图片，占比不够高 -
动图流量占比大，但是iOS和Android系统都不支持HEIC动图
3、解决方案
1）自研HEIC解码器，解决Android端HEIC图片解码性能问题
HEIC图片格式是使用H265编码格式压缩的，而爱奇艺在H265视频编解码方面有丰富经验，于是决定自研HEIC图片解码方案。经过不懈努力，自研解码器的性能相对于系统解码器提升了5倍以上。解码性能虽然仍然比WebP差一点，但是考虑到CDN成本节省，图片下载耗时降低等优势，已具备上线要求。
2）通过后端接口直接返回HEIC格式图片URL的方式落地HEIC格式，解决项目前期caplist方案不能用的问题 由于HEIC格式编码耗时长，不满足caplist方案需要图片格式支持按需生产的前提，那就只能是用最简单直白的方法了，即通过移动后端接口直接返回有预生产好的HEIC格式的图片URL，来让移动前端加载HEIC格式的图片。
但是由于移动后端接口需要聚合的各个业务后端接口非常多，非常分散，而且某个图片是否有预生产好HEIC格式这个信息业务后端接口可能没有传递给移动后端接口，给移动后端接口改造也带来了难度。我们只能通过线上热度数据分析排出要改造的接口的优先级，联合多个业务后端团队有选择地挑选一些影响较大的接口，优先进行改造，以控制改造成本。
3）解决HEIC编码器性能问题，部分使用caplist方案解决项目后期HEIC格式访问量占比增长乏力的问题
Android端自研HEIC解码器的成功研发和落地也为HEIC编码性能较差问题提供了解决思路。技术团队又自研出了高性能HEIC编码方案——Q265编码器，其HEIC编码耗时接近于WebP，比老的X265编码器缩短70%，性能已满足实时生产的需求。这使得caplist方案又能够被采用了。不过，Q265编码器经过几次上线后的迭代改进，中间版本生产出来的HEIC图片有一些问题需要避免使用，因此，caplist覆盖范围必须设置为最终稳定版上线日期之后生产的HEIC图片。加上使用Q265编码器的图片生产工具没有被部署到所有图片CDN服务器，所以，针对HEIC格式的caplist方案是受限制的。
4）针对热度较高的历史图片进行补生产，进一步提升HEIC格式访问量占比
分析发现2020年之前生产的老图片仍然有较高的访问量，但是又没有预生产HEIC格式的图片，并且分布极为分散，难以通过移动后端接口改造的方式来返回HEIC格式给前端，也无法用caplist方案覆盖，因为caplist只能覆盖Q265编码器上线后生产的图片。
最终，我们通过统计线上图片的访问热度，对访问量较高的历史图片补生产了HEIC格式。
5）双端自研动图编解码渲染方案，提高降CDN成本的收益
分析发现移动端有20%-30%的图片流量来自动图，HEIC动图可以像视频编码压缩算法一样支持前后帧合成，文件压缩率比静图更高，但是当时没有现成的HEIC动图的生产端编码和移动端解码方案，需要全新自研。
-
生产端编码方案：自研HEIC动图编码器 使用H265格式编码，使用自研库进行格式封装，配置和调校参数以达到最优效果。实验了全I帧、IP帧、PB帧等多个方案，最终选择了自研IP帧方案，兼顾了兼容性和文件大小。 -
Android端解码方案：自研HEIC动图解码器 
-
iOS端解码方案：自研HEIC动图解码器
4、上线效果
截止2023年3月8日，移动端HEIC格式图片相对于所有图片格式的访问量占比40%+，相比WebP节省流量13%+。HEIC图片格式之所以没能达到WebP格式当时85%的访问量占比，原因如下：
通过移动后端接口改造的手段返回HEIC格式图片给移动前端成本较高，仅覆盖了少数访问量高的业务场景；
通过caplist方案将非HEIC格式图片让CDN优先返回HEIC格式的手段，因Q265编码器上线时间不长，图片CDN服务器未全面部署等原因，仅覆盖了某个日期之后生产的某些图片CDN域名下的图片，范围较小；
补生产的历史图片数量有限，全量补生产代价较大。
04
AVIF格式落地实践
1、背景
比HEIC更新、更先进的图片格式是AVIF（AV1 Image File Format，是由开放媒体联盟开发，采用AV1视讯编码技术压缩图像的一种图像档案格式），其具有更高的图片压缩率，如果能推广落地，又能进一步降低图片CDN成本。经调研，iOS16+，Android12+系统层面对AVIF格式图片进行了支持。
由于在HEIC格式落地的过程中积累了很多成功经验，技术团队对支持更先进的AVIF图片格式信心满满。于是在2023年5月，我们又启动了AVIF图片格式落地专项，基本可以照搬HEIC格式落地的模式。
2、难点
系统解码性能不佳
由于iOS和Android系统解码层面依然存在解码性能差的问题，并且对于系统版本要求过高（Android 12+、iOS 16+），符合要求的线上设备数量占比不够高，导致无法直接使用系统解码作为上线方案。
不支持透明图
由于爱奇艺自研的QAV1编码器之前一直应用于视频编码，对于透明内容没有硬性要求，因此AVIF编解码器刚开始不支持透明内容。而在图片的使用过程中，透明图是比较常见的场景。
-
图片覆盖范围不广 在AVIF全面上线后，根据CDN团队的反馈，caplist方案仅能覆盖有AVIF预生产且支持按需生产的部分图片CDN域名下的部分目录，AVIF格式访问量整体占比依然偏低。通过数据分析发现访问量占比较大的一些目录还未支持AVIF格式图片预生产。另一方面，目前AVIF按需生产的支持范围依然不是很广。这两方面导致caplist方案覆盖范围非常受限。
3、解决方案
-
自研avif解码器，解决移动端AVIF解码性能不佳的问题 得益于爱奇艺QAV1视频编解码器已经广泛使用，可以将成果复用于图片的编解码方案。最终编解码团队自研的AVIF解码器支持iOS9+，Android7+以上的系统版本，并且解码性能优于libheif、libaom、libdav1等开源解码器。 -
改进自研的avif编解码器，解决透明图问题 经过编解码团队的不懈努力，在2023Q4对AVIF透明图的编解码进行了支持。 -
用caplist方案覆盖更多图片CDN目录，提高AVIF格式占比 后台系统在2023Q4对相关目录进行了预生产支持，使得这些目录下预生产日期之后生产的图片可以被caplist方案覆盖到。但是整体覆盖的目录下的图片访问占比仍然偏低。 CDN团队正在研发全按需方案：所有图片CDN域名，所有目录都支持按需生产，且不需要使用caplist参数，通过文件名后缀指定图片格式。届时可以解决不同域名和路径对AVIF的支持。
4、上线效果
通过分析线上数据，观察到图片文件体积优化效果：
静图：HEIC比WebP小30%，AVIF比HEIC小11%，AVIF比WebP小38%；
动图：HEIC比WebP小60%，AVIF比HEIC小30%，AVIF比WebP小72%。
05
总结与展望
通过图片生产消费全流程跨部门通力合作，核心技术攻关，HEIC/AVIF编解码完整方案全自研等实践，爱奇艺逐步将主要图片格式从JPG迁移到最新的AVIF，在保证图片显示质量的前提下，有效地降低了图片CDN带宽成本。
未来，爱奇艺将继续努力提高AVIF图片格式的使用率，实现极致的图片CDN成本控制。
本文分享自微信公众号 - 爱奇艺技术产品团队（iQIYI-TP）。
如有侵权，请联系 [email protected] 删除。
本文参与“OSC源创计划”，欢迎正在阅读的你也加入，一起分享。
{{o.name}}
{{m.name}}