Wow, LZ4 is fast!
I’ve been doing some experiments with LZ4 recently and I must admit that I am truly impressed. For those not familiar with LZ4, it is a compression format from the LZ77 family. Compared to other similar algorithms (such as Google’s Snappy), LZ4’s file format does not allow for very high compression ratios since:
- you cannot reference sequences which are more than 64kb backwards in the stream,
- it encodes lengths with an algorithm that requires 1 + floor(n / 255) bytes to store an integer n instead of the 1 + floor(log(n) / log(2^7)) bytes that variable-length encoding would require.
This might sound like a lot of lost space, but fortunately things are not that bad: there are generally a lot of opportunities to find repeated sequences in a 64kb block, and unless you are working with trivial inputs, you very rarely need to encode lengths which are greater than 15. In case you still doubt LZ4 ability to achieve high compression ratios, the original implementation includes a high compression algorithm that can easily achieve a 40% compression ratio on common ASCII text.
But this file format also allows you to write fast compressors and uncompressors, and this is really what LZ4 excels at: compression and uncompression speed. To measure how faster LZ4 is compared to other famous compression algorithms, I wrote three Java implementations of LZ4:
- a JNI binding to the original C implementation (including the high compression algorithm),
- a pure Java port, using the standard API,
- a pure Java port that uses the sun.misc.Unsafe API to speed up (un)compression.
Then I modified Ning’s JVM compressor benchmark (kudos to Ning for sharing it!) to add my compressors and ran the Calgary compression benchmark.
The results are very impressive:
- the JNI default compressor is the fastest one in all cases but one, and the JNI uncompressor is always the fastest one,
- even when compressed with the high compression algorithm, data is still very fast to uncompress, which is great for read-only data,
- the unsafe Java compressor/uncompressor is by far the fastest pure Java compressor/uncompressor,
- the safe Java compressor/uncompressor has comparable performance to some compressors/uncompressors that use the sun.misc.Unsafe API (such as LZF).
Compression
Uncompression
If you are curious about the compressors whose names start with “LZ4 chunks”, these are compressors that are implemented with Java streams API and compress every 64kb block of the input data separately.
For the full Japex reports, see people.apache.org/~jpountz/lz4.