使用Python和C#对比跨语言压缩算法库LZ4, Snappy, LZO, gzip, glib

版权声明:本文为博主原创文章,未经博主允许不得转载。 https://blog.csdn.net/chenggong2dm/article/details/50731062

写在前面:

    在游戏编程中,数据的处理部分,经常会遇到这样的需求:

    1,服务器端(后端)传给客户端(前端)的数据需要压缩,以节省流量,加速游戏数据通信效率。

    2,这种压缩/解压算法(库)必须尽量少的占用CPU资源,也就是说,压缩解压都要快!游戏卡不得!

    3,这种压缩/解压算法(库)需要跨语言(有点像网络协议)。因为大多数情况下,后端和前端不是一种开发语言。

    4,最好是现成的(不用自己再敲代码去实现),成熟稳定的库。


    忙活了两天,基本锁定在这几个压缩算法库上:LZ4、Snappy、LZO、 gzip、 glib,之后安排同事进行了测试。

    感谢刘金龙同学进行的测试,记录测试数据!

    结论:跨语言、高效压缩解压缩,snappy为首选。


一、LZ4

    速度基本上是最快的。跨语言的互操作性较好。但是目前互操作版不支持C#,有点让人蛋疼。也就是说,unity要是使用这种压缩解压缩方案,要么自己写算法适配,要么等……

    好,看一下官网的描述:


二、Snappy

    看一下官网描述:



    限于篇幅,请大家自行谷歌、百度上述压缩算法的介绍。下面直接说测试过程了。



■ 测试过程:

本测试:

python 和c#都实现了 gzip, zlib, lz4, lzo, snappy 算法的压缩解压缩

Lz4,lzo的python,c#版不能互相解压,gzip,zlib,snappy的可以。

 

------------------------ Python版 ------------------------

压缩率:

压缩前的文件大小 : 104669030


压缩后的文件大小:



总结: gzip,zlib 压缩率差不多分别为49.56%,49.57% lzo次之为68.27%,接着是snappy为72.45%,lz4压缩率最大为79.47%。


是否解压成功:

总结:解压后文件都回复了原来大小,无损压缩:


压缩和解压所用时间:

各个算法压缩解压缩所用的时间:


总结:可以看出来 lz4所用时间最少,snappy次之,接着是lzo,zlib和gzip差不多但是zlib要少一点。

压缩所用系统资源:

 

每0.01秒查看进程利用率 来记录每个算法压缩所需系统资源,第三列为cpu利用率,

下面为记录:

 

root 14303 0.0 5.6 228852 106932pts/1 R+ 11:46 0:00 python2.7 test_gzip

root 14303 14.0 5.6 229464 107544pts/1 R+ 11:46 0:00 python2.7 test_gzip

root 14303 25.0 5.6 231256 108604pts/1 R+ 11:46 0:00 python2.7 test_gzip

root 14303 36.0 5.7 233304 109688pts/1 R+ 11:46 0:00 python2.7 test_gzip

root 14303 47.0 5.8 233304 110776pts/1 R+ 11:46 0:00 python2.7 test_gzip

root 14303 59.0 5.8 237400 111876pts/1 R+ 11:46 0:00 python2.7 test_gzip

root 14303 69.0 6.0 237400 114872pts/1 R+ 11:46 0:00 python2.7 test_gzip

root 14303 80.0 6.0 237400 114872pts/1 R+ 11:46 0:00 python2.7 test_gzip

root 14303 91.0 6.0 237400 115288pts/1 R+ 11:46 0:00 python2.7 test_gzip

root 14303 102 6.1 245592 116428pts/1 R+ 11:46 0:01 python2.7 test_gzip

root 14303 56.5 6.2 245592 118972pts/1 R+ 11:46 0:01 python2.7 test_gzip

root 14303 62.0 6.2 245592 118972pts/1 R+ 11:46 0:01 python2.7 test_gzip

root 14303 67.5 6.3 245592 121020pts/1 R+ 11:46 0:01 python2.7 test_gzip

root 14303 73.0 6.3 245592 121020pts/1 R+ 11:46 0:01 python2.7 test_gzip

root 14303 78.0 6.4 245592 123068pts/1 R+ 11:46 0:01 python2.7 test_gzip

root 14303 84.0 6.4 245592 123220pts/1 R+ 11:46 0:01 python2.7 test_gzip

root 14303 89.5 6.5 261976 124292pts/1 R+ 11:46 0:01 python2.7 test_gzip

root 14303 95.0 6.6 261976 127168pts/1 R+ 11:46 0:01 python2.7 test_gzip

root 14303 100 6.6 261976 127168pts/1 R+ 11:46 0:02 python2.7 test_gzip

root 14303 70.6 6.7 261976 129216pts/1 R+ 11:46 0:02 python2.7 test_gzip

root 14303 74.0 6.7 261976 129216pts/1 R+ 11:46 0:02 python2.7 test_gzip

root 14303 77.6 6.8 261976 131264pts/1 R+ 11:46 0:02 python2.7 test_gzip

root 14303 81.3 6.8 261976 131264pts/1 R+ 11:46 0:02 python2.7 test_gzip

root 14303 85.0 6.9 261976 133312pts/1 R+ 11:46 0:02 python2.7 test_gzip

root 14303 88.6 6.9 261976 133312pts/1 R+ 11:46 0:02 python2.7 test_gzip

root 14303 92.3 7.0 261976 135360pts/1 R+ 11:46 0:02 python2.7 test_gzip

root 14303 96.0 7.0 261976 135360pts/1 R+ 11:46 0:02 python2.7 test_gzip

root 14303 99.6 7.2 261976 137408pts/1 R+ 11:46 0:02 python2.7 test_gzip

root 14303 77.5 7.2 261976 137408pts/1 R+ 11:46 0:03 python2.7 test_gzip

root 14303 80.0 7.3 261976 139456pts/1 R+ 11:46 0:03 python2.7 test_gzip

root 14303 82.7 7.3 261976 139456pts/1 R+ 11:46 0:03 python2.7 test_gzip

root 14303 85.5 7.3 261976 140012pts/1 R+ 11:46 0:03 python2.7 test_gzip

root 14303 88.2 7.3 294744 141160pts/1 R+ 11:46 0:03 python2.7 test_gzip

root 14303 91.0 7.5 294744 143556pts/1 R+ 11:46 0:03 python2.7 test_gzip

root 14303 93.7 7.5 294744 143556pts/1 R+ 11:46 0:03 python2.7 test_gzip

root 14303 96.5 7.6 294744 145604pts/1 R+ 11:46 0:03 python2.7 test_gzip

root 14303 99.2 7.7 294744 147652pts/1 R+ 11:46 0:03 python2.7 test_gzip

root 14303 81.6 7.7 294744 147652pts/1 R+ 11:46 0:04 python2.7 test_gzip

root 14303 83.8 7.8 294744 149700pts/1 R+ 11:46 0:04 python2.7 test_gzip

root 14303 86.0 7.8 294744 149700pts/1 R+ 11:46 0:04 python2.7 test_gzip

root 14303 88.2 7.9 294744 151748pts/1 R+ 11:46 0:04 python2.7 test_gzip

root 14303 90.2 7.9 294744 151748pts/1 R+ 11:46 0:04 python2.7 test_gzip

root 14303 92.6 8.0 294744 153796pts/1 R+ 11:46 0:04 python2.7 test_gzip

root 14303 94.6 8.0 294744 153796pts/1 R+ 11:46 0:04 python2.7 test_gzip

root 14303 96.8 8.1 294744 155844pts/1 R+ 11:46 0:04 python2.7 test_gzip

root 14303 99.0 8.1 294744 155844pts/1 R+ 11:46 0:04 python2.7 test_gzip

root 14303 84.3 8.2 294744 157892pts/1 R+ 11:46 0:05 python2.7 test_gzip

root 14303 86.1 8.2 294744 157892pts/1 R+ 11:46 0:05 python2.7 test_gzip

root 14303 88.0 5.6 228948 107144pts/1 D+ 11:46 0:05 python2.7 test_gzip

root 14303 88.6 0.7 136460 13356pts/1 R+ 11:46 0:05 python2.7 test_gzip

root 14303 90.5 2.1 165460 41232pts/1 R+ 11:46 0:05 python2.7 test_gzip

root 14303 92.5 2.9 181988 56928pts/1 R+ 11:46 0:05 python2.7 test_gzip

root 14303 94.3 4.5 215436 86280pts/1 R+ 11:46 0:05 python2.7 test_gzip

root 14303 96.0 7.0 309784 133808pts/1 R+ 11:46 0:05 python2.7 test_gzip

root 14303 97.8 6.6 256616 127460pts/1 R+ 11:46 0:05 python2.7 test_gzip

root 14303 85.7 10.5 351628 200592pts/1 R+ 11:46 0:06 python2.7 test_gzip

root 14303 87.2 11.2 351628 214928pts/1 R+ 11:46 0:06 python2.7 test_gzip

root 14303 89.1 11.4 358828 219124pts/1 R+ 11:46 0:06 python2.7 test_gzip

root 14303 90.7 7.0 263816 134660pts/1 R+ 11:46 0:06 python2.7 test_gzip

root 14303 91.4 7.0 263816 134664pts/1 D+ 11:46 0:06 python2.7 test_gzip

root 14303 91.4 7.0 263816 134664pts/1 D+ 11:46 0:06 python2.7 test_gzip

root 14303 91.5 7.0 263816 134664pts/1 D+ 11:46 0:06 python2.7 test_gzip

root 14303 91.5 7.0 263816 134664pts/1 D+ 11:46 0:06 python2.7 test_gzip

root 14303 91.7 7.0 263816 134664pts/1 D+ 11:46 0:06 python2.7 test_gzip

root 14303 80.3 7.0 263816 134664pts/1 D+ 11:46 0:06 python2.7 test_gzip

root 14303 80.5 7.0 263816 134664pts/1 D+ 11:46 0:06 python2.7 test_gzip

root 14649 0.0 3.2 224568 62344pts/1 R+ 11:46 0:00 python2.7 test_zlib

root 14649 12.0 5.7 327036 109456pts/1 R+ 11:46 0:00 python2.7 test_zlib

root 14649 23.0 5.7 327036 109456pts/1 R+ 11:46 0:00 python2.7 test_zlib

root 14649 34.0 5.8 327036 111504pts/1 R+ 11:46 0:00 python2.7 test_zlib

root 14649 44.0 5.8 327036 111504pts/1 R+ 11:46 0:00 python2.7 test_zlib

root 14649 56.0 5.9 327036 113552pts/1 R+ 11:46 0:00 python2.7 test_zlib

root 14649 67.0 6.0 327036 115600pts/1 R+ 11:46 0:00 python2.7 test_zlib

root 14649 77.0 6.0 327036 115600pts/1 R+ 11:46 0:00 python2.7 test_zlib

root 14649 89.0 6.1 327036 117648pts/1 R+ 11:46 0:00 python2.7 test_zlib

root 14649 99.0 6.1 327036 117648pts/1 R+ 11:46 0:00 python2.7 test_zlib

root 14649 55.0 6.2 327036 119696pts/1 R+ 11:46 0:01 python2.7 test_zlib

root 14649 60.5 6.3 327036 121744pts/1 R+ 11:46 0:01 python2.7 test_zlib

root 14649 66.0 6.3 327036 121744pts/1 R+ 11:46 0:01 python2.7 test_zlib

root 14649 71.5 6.4 327036 123792pts/1 R+ 11:46 0:01 python2.7 test_zlib

root 14649 77.5 6.4 327036 123792pts/1 R+ 11:46 0:01 python2.7 test_zlib

root 14649 83.0 6.5 327036 125840pts/1 R+ 11:46 0:01 python2.7 test_zlib

root 14649 88.5 6.5 327036 125840pts/1 R+ 11:46 0:01 python2.7 test_zlib

root 14649 94.0 6.7 327036 127888pts/1 R+ 11:46 0:01 python2.7 test_zlib

root 14649 99.5 6.7 327036 127888pts/1 R+ 11:46 0:01 python2.7 test_zlib

root 14649 70.0 6.8 327036 129936pts/1 R+ 11:46 0:02 python2.7 test_zlib

root 14649 73.6 6.9 327036 131984pts/1 R+ 11:46 0:02 python2.7 test_zlib

root 14649 77.3 6.9 327036 131984pts/1 R+ 11:46 0:02 python2.7 test_zlib

root 14649 81.0 7.0 327036 134032pts/1 R+ 11:46 0:02 python2.7 test_zlib

root 14649 84.6 7.0 327036 134032pts/1 R+ 11:46 0:02 python2.7 test_zlib

root 14649 88.3 7.1 327036 136080pts/1 R+ 11:46 0:02 python2.7 test_zlib

root 14649 92.0 7.1 327036 136080pts/1 R+ 11:46 0:02 python2.7 test_zlib

root 14649 95.6 7.2 327036 138128pts/1 R+ 11:46 0:02 python2.7 test_zlib

root 14649 99.3 7.2 327036 138128pts/1 R+ 11:46 0:02 python2.7 test_zlib

root 14649 77.0 7.3 327036 140176pts/1 R+ 11:46 0:03 python2.7 test_zlib

root 14649 79.7 7.4 327036 142224pts/1 R+ 11:46 0:03 python2.7 test_zlib

root 14649 82.5 7.4 327036 142224pts/1 R+ 11:46 0:03 python2.7 test_zlib

root 14649 85.2 7.5 327036 144272pts/1 R+ 11:46 0:03 python2.7 test_zlib

root 14649 88.0 7.5 327036 144272pts/1 R+ 11:46 0:03 python2.7 test_zlib

root 14649 90.7 7.6 327036 146320pts/1 R+ 11:46 0:03 python2.7 test_zlib

root 14649 93.5 7.6 327036 146320pts/1 R+ 11:46 0:03 python2.7 test_zlib

root 14649 96.2 7.7 327036 148368pts/1 R+ 11:46 0:03 python2.7 test_zlib

root 14649 99.0 7.8 327036 150416pts/1 R+ 11:46 0:03 python2.7 test_zlib

root 14649 81.4 7.8 327036 150416pts/1 R+ 11:46 0:04 python2.7 test_zlib

root 14649 83.6 7.9 327036 152464pts/1 R+ 11:46 0:04 python2.7 test_zlib

root 14649 85.8 7.9 327036 152464pts/1 R+ 11:46 0:04 python2.7 test_zlib

root 14649 88.0 8.0 327036 154512pts/1 R+ 11:46 0:04 python2.7 test_zlib

root 14649 90.0 8.0 327036 154512pts/1 R+ 11:46 0:04 python2.7 test_zlib

root 14649 92.2 8.2 327036 156560pts/1 R+ 11:46 0:04 python2.7 test_zlib

root 14649 94.4 8.2 327036 156560pts/1 R+ 11:46 0:04 python2.7 test_zlib

root 14649 96.6 9.9 377588 189092pts/1 R+ 11:46 0:04 python2.7 test_zlib

root 14649 98.0 2.9 275272 56732pts/1 R+ 11:46 0:04 python2.7 test_zlib

root 14649 83.3 3.7 205828 72268 pts/1R+ 11:46 0:05 python2.7 test_zlib

root 14649 85.3 5.0 238596 96856pts/1 R+ 11:46 0:05 python2.7 test_zlib

root 14649 85.5 5.0 238596 96856pts/1 R+ 11:46 0:05 python2.7 test_zlib

root 14649 89.0 6.1 238596 117336pts/1 R+ 11:46 0:05 python2.7 test_zlib

root 14649 90.6 7.2 304132 137820pts/1 R+ 11:46 0:05 python2.7 test_zlib

root 14649 92.6 8.2 275272 157604pts/1 R+ 11:46 0:05 python2.7 test_zlib

root 14649 93.8 8.2 275276 157608pts/1 D+ 11:46 0:05 python2.7 test_zlib

root 14649 94.0 4.7 224592 90032pts/1 R+ 11:46 0:05 python2.7 test_zlib

root 14963 6.0 6.2 325084 120076pts/1 R+ 11:46 0:00 python2.7 test_lz4.

root 14963 17.0 7.7 325084 148748pts/1 R+ 11:46 0:00 python2.7 test_lz4.

root 14963 28.0 9.5 325084 181516pts/1 R+ 11:46 0:00 python2.7 test_lz4.

root 14963 39.0 2.0 201488 39692pts/1 R+ 11:46 0:00 python2.7 test_lz4.

root 14963 51.0 9.8 303704 187968pts/1 R+ 11:46 0:00 python2.7 test_lz4.

root 14963 56.0 9.8 303704 187972pts/1 D+ 11:46 0:00 python2.7 test_lz4.

root 14963 59.0 9.8 303704 187972pts/1 D+ 11:46 0:00 python2.7 test_lz4.

root 14963 59.0 9.8 303704 187972pts/1 D+ 11:46 0:00 python2.7 test_lz4.

root 14963 30.5 9.8 303704 187972pts/1 D+ 11:46 0:00 python2.7 test_lz4.

root 14963 32.0 9.8 303704 187972pts/1 R+ 11:46 0:00 python2.7 test_lz4.

root 15100 10.0 5.9 333336 112888pts/1 R+ 11:46 0:00 python2.7 test_lzo.

root 15100 21.0 6.3 333336 121080pts/1 R+ 11:46 0:00 python2.7 test_lzo.

root 15100 32.0 6.8 333336 131320pts/1 R+ 11:46 0:00 python2.7 test_lzo.

root 15100 43.0 7.3 333336 139512pts/1 R+ 11:46 0:00 python2.7 test_lzo.

root 15100 54.0 7.7 333336 147704pts/1 R+ 11:46 0:00 python2.7 test_lzo.

root 15100 32.5 8.1 333336 155896pts/1 R+ 11:46 0:00 python2.7 test_lzo.

root 15100 38.0 8.6 333336 164088pts/1 R+ 11:46 0:00 python2.7 test_lzo.

root 15100 43.5 9.0 333336 172280pts/1 R+ 11:46 0:00 python2.7 test_lzo.

root 15100 49.0 9.2 294384 176496pts/1 R+ 11:46 0:00 python2.7 test_lzo.

root 15100 50.0 9.2 294384 176500pts/1 D+ 11:46 0:01 python2.7 test_lzo.

root 15100 55.0 5.1 294384 98548pts/1 R+ 11:46 0:01 python2.7 test_lzo.

root 15100 60.5 6.9 294384 133364pts/1 R+ 11:46 0:01 python2.7 test_lzo.

root 15100 66.0 8.7 294384 166132pts/1 R+ 11:46 0:01 python2.7 test_lzo.

root 15100 71.5 9.2 294384 176528pts/1 R+ 11:46 0:01 python2.7 test_lzo.

root 15100 50.0 9.2 294388 176532pts/1 D+ 11:46 0:01 python2.7 test_lzo.

root 15100 50.3 9.2 294388 176532pts/1 D+ 11:46 0:01 python2.7 test_lzo.

root 15100 50.3 9.2 294388 176532pts/1 D+ 11:46 0:01 python2.7 test_lzo.

root 15325 7.0 6.1 351200 116460pts/1 R+ 11:46 0:00 python2.7 test_snap

root 15325 19.0 7.0 351200 134892pts/1 R+ 11:46 0:00 python2.7 test_snap

root 15325 30.0 8.1 351200 155372pts/1 R+ 11:46 0:00 python2.7 test_snap

root 15325 41.0 9.1 351200 173804pts/1 R+ 11:46 0:00 python2.7 test_snap

root 15325 26.0 9.5 305904 181284pts/1 R+ 11:46 0:00 python2.7 test_snap

root 15325 36.0 9.5 305908 181308pts/1 R+ 11:46 0:00 python2.7 test_snap

root 15325 38.5 9.5 305908 181312pts/1 D+ 11:46 0:00 python2.7 test_snap

root 15325 39.5 9.5 305908 181312pts/1 R+ 11:46 0:00 python2.7 test_snap

root 15325 40.5 9.5 305908 181312pts/1 D+ 11:46 0:00 python2.7 test_snap

 

总结:可以看出 snappy和lz4所用资源差不多,最少,lzo次之,zlib和gzip最多。

 

------------------------ C#版------------------------

压缩率:

 压缩前的文件大小


各算法压缩后的文件大小:


总结:可以看出来gzip,zlib 压缩率都为49.57,lzo压缩率68.27,snappy为73.93,lz4压缩率为74.31。

需要注意的是,C#版本压缩后的文件大小,与Python版压缩后的大小,不一致!

 

是否解压成功:

总结:解压后文件都恢复了原来大小,无损压缩:

 

压缩和解压所用时间:

各个算法压缩解压缩所用的时间:

 


 

总结: snappy所用时间最少为0.5秒,lz4次之为0.59,接着是lzo为0.96秒,gzip和zlib远远超过前三个分别为5.71秒和7.15秒。

 

压缩所用系统资源:

gzip:第一个进程为gzip的。


zlib:第一个进程为zlib的。

 

lz4:第一个进程为lz4的。


lzo:第一个进程为lzo的。

 

snappy:第二个进程为snappy的。



总结:可以看出来 gzip和zlib所用资源差不多远远大于lz4,lzo和snappy。

 

各压缩算法跨语言的解压缩:

lzo:

python压缩的在c#不能解压:

一直卡在这界面,解压不了。

 

 

c#压缩的在python中也不能解压。


lz4:

python压缩的在c#中不能解压



c#压缩的在python中也不能解压:


 

gzip:

python 压缩的在c#中解压成功,恢复原文件,无损,记事本能打开,字节不少:

 


c#压缩的在python解压成功,恢复原文件,无损,记事本能打开,字节不少:

 



zlib:

python压缩的在c#中解压成功,恢复原文件,无损,记事本能打开,字节不少:


c#压缩的在python中解压成功,恢复原文件,无损,记事本能打开,字节不少:


snappy:

python压缩的在c#中解压成功,恢复原文件,无损,记事本能打开,字节不少:



c#压缩的在python中解压成功,恢复原文件,无损,记事本能打开,字节不少:

 

 

 

总结:

gzip和zlib压缩率最低,相差不多,同时所消耗资源和时间也是最多的,但他俩都能在python和c#中实现并能互相解压缩。lzo,lz4,snappy三个压缩率比gzip,zlib高,但是所消耗的资源远小于gzip和zlib。lzo和lz4不能跨语言解压缩而snappy可以。

snappy为所找能跨语言压缩解压缩,同时消耗资源较小的压缩算法。

 










猜你喜欢

转载自blog.csdn.net/chenggong2dm/article/details/50731062