The Technology Behind Facebook's Mobile Photo Previews

When browsing someone's profile or page on Facebook Mobile, the first thing you see is often an image. These images are an integral part of the Facebook experience, but sometimes images are very slow to download and display, especially on slow or mobile networks. And in developing markets like India, many of Facebook's new users are primarily on 2G networks. Recently, Facebook engineers Brian K Cabral and Edward Kandrot wrote an article describing how Facebook solved this problem.

The cover photo is the most visible part of the screen, but it's also one of the slowest to load. There are two main reasons for this: one is that the size of the cover photo is often 100KB, and the transfer speed of a 2G connection may be only 32KB/sec; the other is that the application needs to send two network requests to display the cover photo. It first sends a request to the GraphQL server to get the photo URL, then sends a second network request to fetch the photo from the CDN using that URL. The latency of the second network request is quite high, much longer than the first.

In order to solve the above problems, they hope to be able to generate a 200-byte rendering from the original photo, and then return it directly in the first request response as part of the GraphQL response, which can save the second request. Dramatically reduces user profile and header display latency. Of course, they still end up downloading the full photo from the CDN and showing it, but that can happen in the background. At this point, the question becomes how to compress the photo into 200 bytes.

They wanted the rendering of the photo to have a frosted glass effect. It's fun and consistent with the original photo. The frosted glass effect is easier to achieve with a Gaussian filter, and the blurrier the picture, the lower the resolution and the smaller the size of the picture. However, in order to provide a good user experience, the resolution should not be too low. After many attempts, they figured out that a 42x42 image would do what they wanted, and a higher resolution wouldn't give a better result. However, even if only displaying the DC component of the picture, each pixel would still require 3 bytes, so 42x42x3 would be 5292 bytes, well beyond the 200-byte goal.

They started evaluating standard compression techniques, trying to figure out the best way to compress data down to 200 bytes. Unfortunately, just entropy encoding the picture (such as zlib) can only compress the picture by half, which is still too large. They also evaluated several other non-standard techniques, but in the end, they decided to give JPEG image encoding a try. Sadly, the JPEG header itself is a few hundred bytes in size. However, with the JPEG header removed, the encoded data payload is close to 200 bytes.

So they started to explore if it was possible to use a fixed header for JPEG images so that they could be stored on the client side without being transmitted. The JPEG header contains several tables. In the case of a certain Q value, the quantization table is unchanged. Through experiments, they found that the pictures generated by the Q20 could meet their requirements. Although their images are not fixed size, they are basically limited to 42x42 or less. They also scrutinized the rest of the JPEG header and found that only the Huffman table changed from image to image. The Q value, the picture data, and the picture size determine the frequency values ​​in the Huffman table, and each change results in a different compression ratio and number of payload bytes. They experimented on a set of pictures and eventually came up with a Huffman table that could serve as a standard.

While they deal with a lot of images, there are always situations where this scheme doesn't work. To this end, they added a version number. If any corner cases are found, or a better Huffman table is found in the future, then they can update the version number of the associated image and send the new table to the client. The final format contains a one-byte version number, one-byte width, one-byte height, and a payload of approximately 200 bytes. The server only sends this format as part of the GraphQL response, and the client appends the JPEG body to the predefined JPEG header, producing a normal JPEG image. After standard JPEG decoding, the client can run a predetermined Gaussian blur, stretching its dimensions to fit the window.

Ultimately, they get a format that can meet their needs. This has helped them reduce user profile and page load times by 30% under slow internet speeds. On very fast internet speeds, this ensures that users see a preview of the cover photo immediately, improving the overall experience.