ZPAQ is a "incremental, journaling command-line archiver". I was impressed by it's capabilities and decided to do some quick tests with it:

1. Let's install ZPAQ - I got the RPMS for CentOS from here: http://software.opensuse.org/download.html?project=home:antonbatenev:zpaq-upstream&package=zp

yum install http://download.opensuse.org/repositories/home:/antonbatenev:/zpaq-upstream/ScientificLinux_6/x86_64/zpaq-upstream-7.13-3.1.x86_64.rpm -y

2. Let's download the source code of Firefox 47 for Linux to get some files to play with:

mkdir /tmp/firefox
curl https://archive.mozilla.org/pub/firefox/releases/47.0/firefox-47.0.linux-x86_64.sdk.tar.bz2 | tar jxf - -C /tmp/firefox/

3. OK, before we begin let's see how big the directory /tmp/firefox is:

du -hs /tmp/firefox/
441M /tmp/firefox/

4. Cool, and how many files is that again?

find /tmp/firefox/ -type f | wc -l
6136

5. Nice, let's try compressing this with ZPAQ:

zpaq add firefox47source /tmp/firefox/

Towards the end you'll get output like this:

0.000000 + (447.897244 -> 214.672695 -> 83.551155) = 83.551155 MB
28.797 seconds (all OK)

As you can see the above shows you how long it took, if there were any errors and what the resulting file size is.

6. Let's double check how big the resulting file is with du -h:

du -h firefox47source.zpaq 
80M firefox47source.zpaq

7. Let's compare with gzip, bzip and xz:

Gzip:

time tar czf firefox47gzip.tgz /tmp/firefox/
tar: Removing leading `/' from member names

real 0m34.436s
user 0m33.007s
sys 0m1.347s

Bzip2:

time tar cjf firefox47bzip2.tar.bz2 /tmp/firefox/
tar: Removing leading `/' from member names

real 1m33.266s
user 1m31.595s
sys 0m1.488s

XZ:

[root@zpaq ~]# time tar cJf firefox47XZ.tar.xz /tmp/firefox/
tar: Removing leading `/' from member names

real 6m43.799s
user 6m37.258s
sys 0m3.454s

Let's get the sizes too:

Gzip:

du -h firefox47gzip.tgz 
151M firefox47gzip.tgz

Bzip2:

du -h firefox47bzip2.tar.bz2 
140M firefox47bzip2.tar.bz2

XZ:

du -h firefox47XZ.tar.xz 
111M firefox47XZ.tar.xz

Format          Size (MB) Time (Seconds)
Uncompressed    441       N/A
Gzip            151       34
Bzip2           140       99
XZ              111       403
ZPAQ            80        28

Note: The above tests were run on a 1core/1GB VPS.

8. What if we create an uncompressed tarball first and then compress it with zpaq?

time tar -cf firefox47-uncompressed.tar /tmp/firefox/
tar: Removing leading `/' from member names

real 0m3.699s
user 0m0.061s
sys 0m1.515s

du -h firefox47-uncompressed.tar 
432M firefox47-uncompressed.tar

9. OK, let's compress that with zpaq:

zpaq add firefox47-uncompressed firefox47-uncompressed.tar
0.000000 + (452.638720 -> 220.743936 -> 83.931910) = 83.931910 MB
30.328 seconds (all OK)

du -h firefox47-uncompressed.zpaq 
81M firefox47-uncompressed.zpaq

10. Nice! Now let's pretend we accidentally have two copies of firefox under /tmp/firefox ... that means a bunch of duplicated files.

mkdir /tmp/firefox2
cp -R /tmp/firefox /tmp/firefox2
mv /tmp/firefox2/ /tmp/firefox/

Let's confirm the total size of /tmp/firefox now and number of files:

du -hs /tmp/firefox/
881M /tmp/firefox/

find /tmp/firefox/ -type f | wc -l
12272

Now let's compress with ZPAQ again:

zpaq add firefox47doublesource /tmp/firefox/
0.000000 + (895.794488 -> 214.672695 -> 83.669920) = 83.669920 MB
33.011 seconds (all OK)

du -h firefox47doublesource.zpaq 
80M firefox47doublesource.zpaq

Yup - ZPAQ figured out that the files are identical and didn't use up more space for them.

Let's try the same with gzip and see what happens:

time tar czf firefox47doublesourcegzip.tgz /tmp/firefox/
tar: Removing leading `/' from member names

real 1m10.904s
user 1m6.576s
sys 0m3.709s

du -h firefox47doublesourcegzip.tgz 
302M firefox47doublesourcegzip.tgz

11. OK, but what happens with an uncompressed tarball that has duplicates?

Let's find out!

du -hs /tmp/firefox/
881M /tmp/firefox

time tar -cf firefox47doublesource-uncompressed.tar /tmp/firefox/
tar: Removing leading `/' from member names

real 0m6.654s
user 0m0.191s
sys 0m2.921s

du -h firefox47doublesource-uncompressed.tar 
864M firefox47doublesource-uncompressed.tar

Now let's compress the tarball with ZPAQ:

zpaq add firefox47doublesource-uncompressed firefox47doublesource-uncompressed.tar
0.000000 + (905.297920 -> 270.627222 -> 92.917738) = 92.917738 MB
39.503 seconds (all OK)

du -h firefox47doublesource-uncompressed.zpaq 
89M firefox47doublesource-uncompressed.zpaq

12. How to decompress with zpaq?

Let's imagine for a second that we accidentally deleted /tmp/firefox

rm -rf /tmp/firefox/

Restoring is very simple:

zpaq extract firefox47doublesource.zpaq

13. What if I want to restore to a different directory? for example... /goatmaster ?

zpaq extract firefox47doublesource.zpaq -to /goatmaster/

Will give you a structure like: /goatmaster/tmp/firefox/

Sources: