I always thought that unlike compression tools, tar could not use multiple cores.

It turns out somebody went out and modified tar to do just that -- and it was 4 years ago too! It's called mtar.

Not only does mtar uses pigz instead of gzip, but it also implemented multi threading for reading files.

Allow me to quote the documentation:

"Reading of input files has been multi-threaded for improved performance on slow file systems or when used on network file shares."

This means that if you don't like pigz or gzip you can still benefit from using mtar - the question is - exactly how much?

Testing:

Let's try a simple test - I have created 20k files in the directory /opt - they are a total of 2.5GB. Let's try tar and mtar and see how fast we can tar those up --- please note I won't use compression here.

GNU tar first:

time tar -cf gnutar.tar /opt/
tar: Removing leading `/' from member names

real 0m20.255s
user 0m0.412s
sys 0m7.605s

mtar second - note the 'Y' switch that enables multi-threading - this server has 2 cores:

time bsdtar -cYf mtar.tar /opt/
bsdtar: Removing leading '/' from member names

real 0m11.417s
user 0m0.800s
sys 0m9.614s

20.2 seconds vs 11.4 seconds - that's 43.56% faster than GNU tar.

Not bad!

I'm sold! I'm sold! How do I install it?

You can install it via my repository:

yum -y install http://rpm.chaidas.com/rpm.chaidas.com-0.1-1.x86_64.rpm

yum -y install mtar

No! I want to build it myself!

It's pretty easy to do that too:

1. First install some dependencies that you are going to need:

yum -y install gcc openssl-devel zlib-devel autoconf automake libtool libxml2-devel libtool-ltdl-devel git

2. Clone the github repository:

cd /tmp
git clone git://github.com/johnno1962/mtar.git

3. Configure mtar:

cd mtar
autoreconf -ivf
./configure

4. Edit the Makefile and change the line:

LDFLAGS =

to:

LDFLAGS = -lpthread

5. Build it and install it:

make -j `nproc`
make install

You're good to go!

What if I want to build an RPM out of this?

Then you'll need Ruby, FPM and a couple of small changes - to summarize the steps:

1. Get your dependencies:

yum -y install gcc openssl-devel zlib-devel autoconf automake libtool libxml2-devel libtool-ltdl-devel git ruby ruby-devel rpm-build
gem install fpm --no-ri --no-rdoc

2. Clone, configure, make and install mtar:

cd /tmp
git clone git://github.com/johnno1962/mtar.git
cd mtar
autoreconf -ivf
mkdir -pv /target
./configure --prefix=/target
sed -i 's/LDFLAGS =/LDFLAGS = -lpthread/' Makefile
make -j `nproc`
make install

3. Now use FPM to build your RPM:

cd /target
fpm --verbose -v 1.1 -n mtar \
--replaces bsdtar-\* \
-s dir -t rpm .=/usr/

4. And check your resulting RPM with:

rpm -qlp mtar-1.1-1.x86_64.rpm

You should get output like this:

/usr/bin/bsdcpio
/usr/bin/bsdtar
/usr/include/archive.h
/usr/include/archive_entry.h
/usr/lib/libarchive.a
/usr/lib/libarchive.la
/usr/lib/libarchive.so
/usr/lib/libarchive.so.13
/usr/lib/libarchive.so.13.1.2
/usr/lib/pkgconfig/libarchive.pc
/usr/share/man/man1/bsdcpio.1
/usr/share/man/man1/bsdtar.1
/usr/share/man/man3/archive_entry.3
/usr/share/man/man3/archive_entry_acl.3
/usr/share/man/man3/archive_entry_linkify.3
/usr/share/man/man3/archive_entry_paths.3
/usr/share/man/man3/archive_entry_perms.3
/usr/share/man/man3/archive_entry_stat.3
/usr/share/man/man3/archive_entry_time.3
/usr/share/man/man3/archive_read.3
/usr/share/man/man3/archive_read_data.3
/usr/share/man/man3/archive_read_disk.3
/usr/share/man/man3/archive_read_extract.3
/usr/share/man/man3/archive_read_filter.3
/usr/share/man/man3/archive_read_format.3
/usr/share/man/man3/archive_read_free.3
/usr/share/man/man3/archive_read_header.3
/usr/share/man/man3/archive_read_new.3
/usr/share/man/man3/archive_read_open.3
/usr/share/man/man3/archive_read_set_options.3
/usr/share/man/man3/archive_util.3
/usr/share/man/man3/archive_write.3
/usr/share/man/man3/archive_write_blocksize.3
/usr/share/man/man3/archive_write_data.3
/usr/share/man/man3/archive_write_disk.3
/usr/share/man/man3/archive_write_filter.3
/usr/share/man/man3/archive_write_finish_entry.3
/usr/share/man/man3/archive_write_format.3
/usr/share/man/man3/archive_write_free.3
/usr/share/man/man3/archive_write_header.3
/usr/share/man/man3/archive_write_new.3
/usr/share/man/man3/archive_write_open.3
/usr/share/man/man3/archive_write_set_options.3
/usr/share/man/man3/libarchive.3
/usr/share/man/man3/libarchive_changes.3
/usr/share/man/man3/libarchive_internals.3
/usr/share/man/man5/cpio.5
/usr/share/man/man5/libarchive-formats.5
/usr/share/man/man5/mtree.5
/usr/share/man/man5/tar.5