Lampros - Weird Bricks

MooseFS 1.6.27 on FreeBSD 10

15 June, 2014 | MooseFS

MooseFS is a distributed file system that works on multiple different platforms including FreeBSD. It has three parts one of which is optional:

1. The MooseFS Master

2. The MooseFS ChunkServers

3. The MooseFS Metalogger (Optional)

The MooseFS master keeps track of all the files and changes on the ChunkServers.

The ChunkServers is where the actual data is kept.

The Metalogger is a backup server in case the master goes down - it's not necessary to use it but it's recommended else the master becomes a single point of failure. In this post I'm not going to include the Metalogger.

Network topology for this example, yup it's going to be simple!

The MooseFS master is going to be the server 192.168.2.101

ChunkServer 1 will be 192.168.2.102

ChunkServer 2 will be 192.168.2.103

Installing the MooseFS Master

Install dependencies

pkg install python

Symlink python to /usr/bin/python as the mfscgiserv is looking for it there

ln -s /usr/local/bin/python /usr/bin/python

Compile it from ports:

cd /usr/ports/sysutils/moosefs-master; time make install clean

Enable the MFS Master service, so that it starts on boot

echo 'mfsmaster_enable="YES"' >> /etc/rc.conf

Do the same for the Moose FS CGI service:

echo 'mfscgiserv_enable="YES"' >> /etc/rc.conf

Copy the empty metadata file:

cp /var/mfs/metadata.mfs.empty /var/mfs/metadata.mfs

Set the default user by editing /usr/local/etc/mfsmaster.cfg change the lines:

#WORKING_USER =
# WORKING_GROUP =

to:

WORKING_USER = mfs
WORKING_GROUP = mfs

Set the correct permissions for the directory /var/mfs to the mfs user and group:

chown -R mfs:mfs /var/mfs/

Try starting the master:

service mfsmaster start

Check if the master is running:

service mfsmaster status

You should get:

mfsmaster is running as pid 5796.

Check that the ports of the master are listening:

sockstat -4 | grep mfsmaster

You should get:

nobody   mfsmaster  5796  8  tcp4   192.168.2.101:9419    *:*
nobody   mfsmaster  5796  9  tcp4   192.168.2.101:9420    *:*
nobody   mfsmaster  5796  10 tcp4   192.168.2.101:9421    *:*

Also start the CGI interface:

service mfscgiserv start

Check if the CGI interface is running, it's listening on port 9425:

sockstat -4 | grep 9425

You should get:

root     python2.7  2128  3  tcp4   192.168.2.101:9425    *:*
?        ?          ?     ?  tcp4   192.168.2.101:9425    192.168.2.3:50240
?        ?          ?     ?  tcp4   192.168.2.101:9425    192.168.2.3:50241

You can take a look at the interface by pointing your browser to the MooseFS master host, in my case that's at: http://192.168.2.101:9425

This is what it looks like:

Installing the MooseFS ChunkServer - do this for each one

Install dependencies

pkg install python

Compile it from ports:

cd /usr/ports/sysutils/moosefs-chunkserver/; time make install clean

Set a default user by editing /usr/local/etc/mfschunkserver.cfg change the lines:

#WORKING_USER =
# WORKING_GROUP =

to:

WORKING_USER = mfs
WORKING_GROUP = mfs

also edit the line:

# MASTER_HOST = mfsmaster

to:

MASTER_HOME = 192.168.2.101

In the above 192.168.2.101 is where the MFS master lives :D

Enable the MFS ChunkServer service, so that it starts on boot

echo 'mfschunkserver_enable="YES"' >> /etc/rc.conf

Since we don't have a dedicated filesystem to assign, we'll create one in a file - this will create a file with 1000 1MB parts - i.e. a 1GB file at /root/mfschunks1

dd if=/dev/zero of=/root/mfschunks1 bs=1m count=1k

Now enable this as a memory disk (loopback file):

mdconfig -a -t vnode -f /root/mfschunks1

Check that it's loaded:

mdconfig -l

You should get:

md0

Now create a filesystem in it:

newfs md0

The output should look like this:

/dev/md0: 1024.0MB (2097152 sectors) block size 32768, fragment size 4096
        using 4 cylinder groups of 256.03MB, 8193 blks, 32896 inodes.
super-block backups (for fsck -b #) at:
 192, 524544, 1048896, 1573248

Create a directory as a mountpoint, set the permissions and mount this there:

mkdir -p /mnt/mfschunks1
chown -R mfs:mfs /mnt/mfschunks1
mount /dev/md0 /mnt/mfschunks1/

If you're running in a Jail and get this error:

mount: /dev/md0: Operation not permitted

You'll have to do this from the host :( - go on the host and do:

dd if=/dev/zero of=/root/mfschunks1 bs=1m count=1k
mdconfig -a -t vnode -f /root/mfschunks1
newfs /dev/md0
mkdir /usr/jails/moosefschunk1/mnt/mfschunk1
mount /dev/md0 /usr/jails/moosefschunk1/mnt/mfschunk1

Then inside the jail do:

chown -R mfs:mfs /mnt/mfschunk1

Now tell the ChunkServer where the drives are that it can use - we do this by editing the file mfshdd.cfg.

This file is found under /usr/local/etc/mfshdd.cfg but needs to be moved to /usr/local/etc/mfs/mfshdd.cfg, so we'll move the file and then edit it:

First move the file:

mv /usr/local/etc/mfshdd.cfg /usr/local/etc/mfs/

Then edit /usr/local/etc/mfs/mfshdd.cfg and add the line:

/mnt/mfschunk1

Set the correct permissions for the directory /var/mfs to the mfs user and group:

chown -R mfs:mfs /var/mfs/

Now try starting the chunkserver:

service mfschunkserver start

Check if the chunkserver is listening:

sockstat -4 | grep mfs

You should get:

mfs      mfschunkse 6247  8  tcp4   192.168.2.103:9422    *:*
mfs      mfschunkse 6247  12 tcp4   192.168.2.103:21374   192.168.2.101:9420

Once the chunkserver has started the master should automatically add it to the cluster - go to the MooseFS interface and click on 'Disks', you should see something like this:

Configuring the client!

Note: A requirement for the client is that you can use FUSE - if you cannot use FUSE you won't be able to mount MooseFS - an example of this is a FreeBSD Jail :(

Install the MooseFS client - FUSE is a dependency of this too:

cd /usr/ports/sysutils/moosefs-client

Create a directory to use as mountpoint:

mkdir /mnt/moosefs

Load the fuse module

kldload fuse

Mount the moosefs export - we point to the master:

mfsmount /mnt/moosefs/ -H 192.168.2.101

Check if it's mounted:

df -h -T /mnt/moosefs

You should get:

Filesystem  Type      Size    Used   Avail Capacity  Mounted on
/dev/fuse   fusefs    1.3G      0B    1.3G     0%    /mnt/moosefs

Write some files:

dd if=/dev/random of=/mnt/moosefs/test1-100mb bs=1m count=100
dd if=/dev/random of=/mnt/moosefs/test2-200mb bs=1m count=200

Basic replication

All the test files we have created so far only have a single copy in the filesystem. That means that there's no redundancy and if one of the chunkservers go down you lose your file. Create a directory and set the goal to have at least 2 replicas:

mkdir /mnt/moosefs/replicadir

Now set the goal to 2:

mfssetgoal -r 2 /mnt/moosefs/replicadir

Check that it's set correctly:

mfsgetgoal /mnt/moosefs/replicadir/

You should get output like this - note the '2':

/mnt/moosefs/replicadir/: 2

Now add a file:

dd if=/dev/random of=/mnt/moosefs/replicadir/10meg bs=1m count=10

Check how many copies it has:

mfscheckfile /mnt/moosefs/replicadir/10meg

You should get:

/mnt/moosefs/replicadir/10meg:
 chunks with 2 copies:            1

Also in the MooseFS web interface you should now be seeing files under the column '2' which is under 'Valid Copies', like this: