Lampros - Weird Bricks

Taking S3QL for a spin on an Alma Linux 8.4 container and hooking it up to Backblaze B2

02 October, 2021 | CentOS

Install Docker:

1. Install Docker via the convenience script:

curl -fsSL https://get.docker.com | bash

2. Start the docker service:

systemctl start docker

3. Create a Docker container running Alma Linux 8.4 for testing:

docker run -dt --device /dev/fuse --cap-add SYS_ADMIN --name s3ql_test almalinux:latest bash

4. Enter the container:

docker exec -it s3ql_test bash

Download and build S3QL:

1. Install Python3.9, bzip2 and some other build dependencies:

yum -y install python39 bzip2 pkgconf pkgconf-pkg-config fuse3-devel gcc python39-devel sqlite-devel which

2. Get the source code for S3QL from Github:

cd /tmp
curl -LO https://github.com/s3ql/s3ql/releases/download/release-3.7.3/s3ql-3.7.3.tar.bz2

3. Untar the source code:

tar -xf s3ql-3.7.3.tar.bz2
cd s3ql-3.7.3

4. Install Python dependencies using pip:

python3 -mpip install -r src/s3ql.egg-info/requires.txt

5. Build S3QL:

python3 setup.py build_ext --inplace

6. Optional: Install pytest to run the tests:

python3 -mpip install pytest pytest_trio

7. Optional: Run the tests:

python3 -m pytest tests/

8. Install for all users:

python3 setup.py install

The S3QL basics - local filesystem:

Now let's create an s3ql filesystem - we'll start by creating a local one.

1. Create a directory to use as a mount point:

mkdir /opt/lampros_test01

2. Now create a filesystem there: (note the triple "/") - you will be asked for an encryption password:

mkfs.s3ql -L lampros_test01 local:///opt/lampros_test01

3. Mount the filesystem - you will be asked for the encryption password:

mount.s3ql local:///opt/lampros_test01 /mnt/

4. Check it's details:

df -hT /mnt/
Filesystem Type Size Used Avail Use% Mounted on
local:///opt/lampros_test01/ fuse.s3ql 1.0T 0 1.0T 0% /mnt

5. Copy a file in it:

cp /tmp/s3ql-3.7.3.tar.bz2 /mnt

6. Unmount it:

umount /mnt/

7. Verify the integrity of the contents:

s3ql_verify local:///opt/lampros_test01

The S3QL basics - B2 filesystem:

1. Now let's create a new filesystem on B2 Backblaze:

mkfs.s3ql -L lampros_test01 b2://lampros-s3ql-b2-test

2. You'll be asked for the B2 login info:

Enter backend login: <the Backblaze Access Key ID>
Enter backend password: <The B2 Application Key>

3. And an encryption password:

Enter backend login: <your encryption password>
Enter backend password: <your encryption password>

4. Mount it:

mount.s3ql b2://lampros-s3ql-b2-test /mnt/

5. Check it's details:

df -hT /mnt/
Filesystem Type Size Used Avail Use% Mounted on
b2://lampros-s3ql-b2-test fuse.s3ql 1.0T 0 1.0T 0% /mnt

6. Copy a file in it:

cp /tmp/s3ql-3.7.3.tar.bz2 /mnt

7. Get some statistics:

s3qlstat /mnt/
Directory entries: 2
Inodes: 4
Data blocks: 1
Total data size: 878 KiB
After de-duplication: 878 KiB (100.00% of total)
After compression: 878 KiB (100.01% of total, 100.01% of de-duplicated)
Database size: 60.0 KiB (uncompressed)
Cache size: 878 KiB, 1 entries
Cache size (dirty): 0 bytes, 0 entries
Queued object removals: 0

8. Optional: Run a fio test:

yum -y install fio
cd /mnt
fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=random_read_write.fio --bs=4k --iodepth=64 --size=1G --readwrite=randrw | grep IOPS
read: IOPS=1290, BW=5161KiB/s (5285kB/s)(512MiB/101558msec)
write: IOPS=1290, BW=5164KiB/s (5288kB/s)(512MiB/101558msec); 0 zone resets

Sources: