Lampros - Weird Bricks

Compression examples and benchmarks

24 January, 2018 | CentOS

You know that moment where you really want to compress something in a hurry and you don't want to really weigh the pros and cons of each utility out there? This is why I decided to take some time and perform some simple tests. 

Disclaimer:

This test is for my own personal usage and some tests have been not performed.

1. Download the latest Linux kernel to use for testing:

cd /tmp
curl -LO https://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.14.15.tar.xz
tar -xf linux-4.14.15.tar.xz

2. Check the decompressed size:

du -hs /tmp/linux-4.14.15
885M /tmp/linux-4.14.15

3. Tar it up using gzip:

cd /opt
time tar -C /tmp/linux-4.14.15 -cvf - . | gzip > linux-4.14.15.gz
real 0m37.709s
user 0m36.079s
sys 0m5.729s
du -h linux-4.14.15.gz
151M linux-4.14.15.gz

4. Tar it up using pigz:

yum install pigz -y
time tar -C /tmp/linux-4.14.15 -cvf - . | pigz > linux-4.14.15.pigz
real 0m19.647s
user 0m36.829s
sys 0m1.448s
du -h linux-4.14.15.pigz
151M linux-4.14.15.pigz

5. Tar it up using lz4:

yum install epel-release -y
yum install lz4 -y
time tar -C /tmp/linux-4.14.15 -cvf - . | lz4 > linux-4.14.15.lz4
real 0m5.170s
user 0m3.613s
sys 0m1.601s
du -h linux-4.14.15.lz4
244M linux-4.14.15.lz4

6. Tar it up using bzip2:

yum install bzip2 -y
time tar -C /tmp/linux-4.14.15 -cvf - . | bzip2 > linux-4.14.15.bz2
real 1m51.363s
user 1m50.324s
sys 0m1.899s
du -h linux-4.14.15.bz2
116M linux-4.14.15.bz2

7. Tar it up using pbzip2:

yum install pbzip2 -y
time tar -C /tmp/linux-4.14.15 -cvf - . | pbzip2 > linux-4.14.15.pbzip2
real 0m54.821s
user 1m45.213s
sys 0m3.951s
du -h linux-4.14.15.pbzip2
116M linux-4.14.15.pbzip2

8. Tar it up using brotli (-6):

yum install http://rpm.chaidas.com/brotli-1.0.1-1.x86_64.rpm -y
time tar -C /tmp/linux-4.14.15 -cvf - . | brotli -6 > linux-4.14.15.brotli6
real 0m43.260s
user 0m42.937s
sys 0m2.670s
du -h linux-4.14.15.brotli6
120M linux-4.14.15.brotli6

9. Tar it up using brotli (-7):

time tar -C /tmp/linux-4.14.15 -cvf - . | brotli -7 > linux-4.14.15.brotli7
real 1m12.779s
user 1m12.251s
sys 0m2.887s
du -h linux-4.14.15.brotli7
116M linux-4.14.15.brotli7

10. Tar it up using brotli (-8):

time tar -C /tmp/linux-4.14.15 -cvf - . | brotli -8 > linux-4.14.15.brotli8
real 1m34.705s
user 1m34.046s
sys 0m2.657s
du -h linux-4.14.15.brotli8
115M linux-4.14.15.brotli8

11. Tar it up using zstd (-8) and use 2 threads (-T2):

time tar -C /tmp/linux-4.14.15 -cvf - . | zstd -T2 -8 > linux-4.14.15.zstd8
real 0m11.060s
user 0m20.718s
sys 0m1.017s
du -h linux-4.14.15.zstd8
124M linux-4.14.15.zstd8

12. Tar it up using zstd (-10) and use 2 threads (-T2):

time tar -C /tmp/linux-4.14.15 -cvf - . | zstd -T2 -10 > linux-4.14.15.zstd10
real 0m18.038s
user 0m34.388s
sys 0m1.127s
du -h linux-4.14.15.zstd10
121M linux-4.14.15.zstd10

13. Tar it up using zstd (-12) and use 2 threads (-T2):

time tar -C /tmp/linux-4.14.15 -cvf - . | zstd -T2 -12 > linux-4.14.15.zstd12
real 0m28.578s
user 0m54.954s
sys 0m1.382s
du -h linux-4.14.15.zstd12
118M linux-4.14.15.zstd12

14. Tar it up using zstd (-14) and use 2 threads (-T2):

time tar -C /tmp/linux-4.14.15 -cvf - . | zstd -T2 -14 > linux-4.14.15.zstd14
real 0m53.609s
user 1m45.363s
sys 0m1.177s
du -h linux-4.14.15.zstd14
117M linux-4.14.15.zstd14

15. Tar it up using xz (-1) and use 2 threads (-T2):

time tar -C /tmp/linux-4.14.15 -cvf - . | xz -T2 -1 > linux-4.14.15.xz1
real 0m28.727s
user 0m55.910s
sys 0m1.164s
du -h linux-4.14.15.xz1
127M linux-4.14.15.xz1

16. Tar it up using xz (-3) and use 2 threads (-T2):

time tar -C /tmp/linux-4.14.15 -cvf - . | xz -T2 -3 > linux-4.14.15.xz3
real 1m5.925s
user 2m9.274s
sys 0m1.138s
du -h linux-4.14.15.xz3
117M linux-4.14.15.xz3

17. Tar it up using xz (-6) and use 2 threads (-T2):

time tar -C /tmp/linux-4.14.15 -cvf - . | xz -T2 -6 > linux-4.14.15.xz6
real 3m28.787s
user 6m52.588s
sys 0m1.285s
du -h linux-4.14.15.xz6
101M linux-4.14.15.xz6

18. Tar it up using lizard:

cd /tmp/
curl -LO https://github.com/inikep/lizard/archive/v1.0.tar.gz
tar -xzf v1.0.tar.gz
cd lizard-1.0/
make -j `nproc`
cp lizard /usr/local/bin/
cd /opt
time tar -C /tmp/linux-4.14.15 -cvf - . | lizard > linux-4.14.15.lizard
real 0m44.007s
user 0m41.987s
sys 0m2.149s
du -h linux-4.14.15.lizard
178M linux-4.14.15.lizard

19. Tar it up using lz4mt (-3):

yum install http://rpm.chaidas.com/lz4mt-0.1-1.x86_64.rpm -y
time tar -C /tmp/linux-4.14.15 -cvf - . | lz4mt -3 - linux-4.14.15.lz4mt-3
real 0m6.346s
user 0m10.991s
sys 0m1.243s
du -h linux-4.14.15.lz4mt-3
188M linux-4.14.15.lz4mt-3

20. Tar it up using lz4mt (-5):

time tar -C /tmp/linux-4.14.15 -cvf - . | lz4mt -5 - linux-4.14.15.lz4mt-5
real 0m8.745s
user 0m15.828s
sys 0m1.124s
du -h linux-4.14.15.lz4mt-5
180M linux-4.14.15.lz4mt-5

21. Tar it up using lz4mt (-7):

time tar -C /tmp/linux-4.14.15 -cvf - . | lz4mt -7 - linux-4.14.15.lz4mt-7
real 0m12.285s
user 0m22.875s
sys 0m1.127s
du -h linux-4.14.15.lz4mt-7
178M linux-4.14.15.lz4mt-7

22. Tar it up using lz4mt (-9):

time tar -C /tmp/linux-4.14.15 -cvf - . | lz4mt -9 - linux-4.14.15.lz4mt-9
real 0m16.557s
user 0m31.103s
sys 0m1.128s
du -h linux-4.14.15.lz4mt-9
177M linux-4.14.15.lz4mt-9

23. Tar it up using plzip (-1):

yum install http://rpm.chaidas.com/plzip-1.6-1.x86_64.rpm -y
time tar -C /tmp/linux-4.14.15 -cvf - . | plzip -1 > linux-4.14.15.plzip1
real 0m39.299s
user 1m17.071s
sys 0m1.059s
du -h linux-4.14.15.plzip1
146M linux-4.14.15.plzip1

24. Tar it up using plzip (-3):

yum install http://rpm.chaidas.com/plzip-1.6-1.x86_64.rpm -y
time tar -C /tmp/linux-4.14.15 -cvf - . | plzip -3 > linux-4.14.15.plzip3
real 1m5.842s
user 2m9.383s
sys 0m1.815s
du -h linux-4.14.15.plzip3
129M linux-4.14.15.plzip3

25. Tar it up using plzip (-6):

yum install http://rpm.chaidas.com/plzip-1.6-1.x86_64.rpm -y
time tar -C /tmp/linux-4.14.15 -cvf - . | plzip -6 > linux-4.14.15.plzip6
real 3m30.852s
user 6m56.902s
sys 0m3.459s
du -h linux-4.14.15.plzip6
128M linux-4.14.15.plzip6

Winners:

Smallest file (regardless of speed):

1st place: XZ (6) - 101MB (3m28 seconds)

2nd place: Brotli (8) - 115MB (1m34 seconds)

3rd place: pbzip2 (default: 9) - 116MB (54 seconds)

Fastest: (regardless of size):

1st place: LZ4 (default) - 244MB - 5 seconds

2nd place: LZ4MT (3) - 188MB - 6 seconds

3rd place: LZ4MT (5) - 180MB - 8 seconds

Best Speed/Compression ratio: (fastest and smallest):

1st place: ZSTD (8) - 124MB - 11 seconds

2nd place: ZSTD (10) - 121MB - 18 seconds

3rd place: ZSTD (12) - 118MB - 28 seconds

CSV Format of results:

Score = size(mb) + time(seconds)

This table is sorted by score - lower is better (smaller file and less time!)

Compression,Size (MB),Time (seconds),Score – lower is better
Zstd (8),124,11,135
Zstd (10),121,18,139
Zstd (12),118,28,146
Xz (1),127,28,155
Brotli (6),120,43,163
pigz,151,19,170
Zstd (14),117,53,170
pbzip2,116,54,170
Xz (3),117,65,182
Plzip (1),146,39,185
Lz4mt (-5),180,8,188
gzip,151,37,188
Brotli (7),116,72,188
Lz4mt (-7),178,12,190
Lz4mt (-9),177,16,193
Lz4mt (-3),188,6,194
Plzip (3),129,65,194
Brotli (8),115,94,209
Lizard,178,44,222
bzip2,116,111,227
lz4,244,5,249
Xz (6),101,208,309
Plzip (6),128,210,338

Sources: