MooseFS 1.6.27 on FreeBSD 10
MooseFS is a distributed file system that works on multiple different platforms including FreeBSD. It has three parts one of which is optional:
1. The MooseFS Master
2. The MooseFS ChunkServers
3. The MooseFS Metalogger (Optional)
The MooseFS master keeps track of all the files and changes on the ChunkServers.
The ChunkServers is where the actual data is kept.
The Metalogger is a backup server in case the master goes down - it's not necessary to use it but it's recommended else the master becomes a single point of failure. In this post I'm not going to include the Metalogger.
Network topology for this example, yup it's going to be simple!
The MooseFS master is going to be the server 192.168.2.101
ChunkServer 1 will be 192.168.2.102
ChunkServer 2 will be 192.168.2.103
Installing the MooseFS Master
Install dependencies
pkg install python
Symlink python to /usr/bin/python as the mfscgiserv is looking for it there
ln -s /usr/local/bin/python /usr/bin/python
Compile it from ports:
cd /usr/ports/sysutils/moosefs-master; time make install clean
Enable the MFS Master service, so that it starts on boot
echo 'mfsmaster_enable="YES"' >> /etc/rc.conf
Do the same for the Moose FS CGI service:
echo 'mfscgiserv_enable="YES"' >> /etc/rc.conf
Copy the empty metadata file:
cp /var/mfs/metadata.mfs.empty /var/mfs/metadata.mfs
Set the default user by editing /usr/local/etc/mfsmaster.cfg change the lines:
#WORKING_USER =
# WORKING_GROUP =
to:
WORKING_USER = mfs
WORKING_GROUP = mfs
Set the correct permissions for the directory /var/mfs to the mfs user and group:
chown -R mfs:mfs /var/mfs/
Try starting the master:
service mfsmaster start
Check if the master is running:
service mfsmaster status
You should get:
mfsmaster is running as pid 5796.
Check that the ports of the master are listening:
sockstat -4 | grep mfsmaster
You should get:
nobody mfsmaster 5796 8 tcp4 192.168.2.101:9419 *:*
nobody mfsmaster 5796 9 tcp4 192.168.2.101:9420 *:*
nobody mfsmaster 5796 10 tcp4 192.168.2.101:9421 *:*
Also start the CGI interface:
service mfscgiserv start
Check if the CGI interface is running, it's listening on port 9425:
sockstat -4 | grep 9425
You should get:
root python2.7 2128 3 tcp4 192.168.2.101:9425 *:*
? ? ? ? tcp4 192.168.2.101:9425 192.168.2.3:50240
? ? ? ? tcp4 192.168.2.101:9425 192.168.2.3:50241
You can take a look at the interface by pointing your browser to the MooseFS master host, in my case that's at: http://192.168.2.101:9425
This is what it looks like: 
Installing the MooseFS ChunkServer - do this for each one
Install dependencies
pkg install python
Compile it from ports:
cd /usr/ports/sysutils/moosefs-chunkserver/; time make install clean
Set a default user by editing /usr/local/etc/mfschunkserver.cfg change the lines:
#WORKING_USER =
# WORKING_GROUP =
to:
WORKING_USER = mfs
WORKING_GROUP = mfs
also edit the line:
# MASTER_HOST = mfsmaster
to:
MASTER_HOME = 192.168.2.101
In the above 192.168.2.101 is where the MFS master lives :D
Enable the MFS ChunkServer service, so that it starts on boot
echo 'mfschunkserver_enable="YES"' >> /etc/rc.conf
Since we don't have a dedicated filesystem to assign, we'll create one in a file - this will create a file with 1000 1MB parts - i.e. a 1GB file at /root/mfschunks1
dd if=/dev/zero of=/root/mfschunks1 bs=1m count=1k
Now enable this as a memory disk (loopback file):
mdconfig -a -t vnode -f /root/mfschunks1
Check that it's loaded:
mdconfig -l
You should get:
md0
Now create a filesystem in it:
newfs md0
The output should look like this:
/dev/md0: 1024.0MB (2097152 sectors) block size 32768, fragment size 4096
using 4 cylinder groups of 256.03MB, 8193 blks, 32896 inodes.
super-block backups (for fsck -b #) at:
192, 524544, 1048896, 1573248
Create a directory as a mountpoint, set the permissions and mount this there:
mkdir -p /mnt/mfschunks1
chown -R mfs:mfs /mnt/mfschunks1
mount /dev/md0 /mnt/mfschunks1/
If you're running in a Jail and get this error:
mount: /dev/md0: Operation not permitted
You'll have to do this from the host :( - go on the host and do:
dd if=/dev/zero of=/root/mfschunks1 bs=1m count=1k
mdconfig -a -t vnode -f /root/mfschunks1
newfs /dev/md0
mkdir /usr/jails/moosefschunk1/mnt/mfschunk1
mount /dev/md0 /usr/jails/moosefschunk1/mnt/mfschunk1
Then inside the jail do:
chown -R mfs:mfs /mnt/mfschunk1
Now tell the ChunkServer where the drives are that it can use - we do this by editing the file mfshdd.cfg.
This file is found under /usr/local/etc/mfshdd.cfg but needs to be moved to /usr/local/etc/mfs/mfshdd.cfg, so we'll move the file and then edit it:
First move the file:
mv /usr/local/etc/mfshdd.cfg /usr/local/etc/mfs/
Then edit /usr/local/etc/mfs/mfshdd.cfg and add the line:
/mnt/mfschunk1
Set the correct permissions for the directory /var/mfs to the mfs user and group:
chown -R mfs:mfs /var/mfs/
Now try starting the chunkserver:
service mfschunkserver start
Check if the chunkserver is listening:
sockstat -4 | grep mfs
You should get:
mfs mfschunkse 6247 8 tcp4 192.168.2.103:9422 *:*
mfs mfschunkse 6247 12 tcp4 192.168.2.103:21374 192.168.2.101:9420
Once the chunkserver has started the master should automatically add it to the cluster - go to the MooseFS interface and click on 'Disks', you should see something like this: 
Configuring the client!
Note: A requirement for the client is that you can use FUSE - if you cannot use FUSE you won't be able to mount MooseFS - an example of this is a FreeBSD Jail :(
Install the MooseFS client - FUSE is a dependency of this too:
cd /usr/ports/sysutils/moosefs-client
Create a directory to use as mountpoint:
mkdir /mnt/moosefs
Load the fuse module
kldload fuse
Mount the moosefs export - we point to the master:
mfsmount /mnt/moosefs/ -H 192.168.2.101
Check if it's mounted:
df -h -T /mnt/moosefs
You should get:
Filesystem Type Size Used Avail Capacity Mounted on
/dev/fuse fusefs 1.3G 0B 1.3G 0% /mnt/moosefs
Write some files:
dd if=/dev/random of=/mnt/moosefs/test1-100mb bs=1m count=100
dd if=/dev/random of=/mnt/moosefs/test2-200mb bs=1m count=200
Basic replication
All the test files we have created so far only have a single copy in the filesystem. That means that there's no redundancy and if one of the chunkservers go down you lose your file. Create a directory and set the goal to have at least 2 replicas:
mkdir /mnt/moosefs/replicadir
Now set the goal to 2:
mfssetgoal -r 2 /mnt/moosefs/replicadir
Check that it's set correctly:
mfsgetgoal /mnt/moosefs/replicadir/
You should get output like this - note the '2':
/mnt/moosefs/replicadir/: 2
Now add a file:
dd if=/dev/random of=/mnt/moosefs/replicadir/10meg bs=1m count=10
Check how many copies it has:
mfscheckfile /mnt/moosefs/replicadir/10meg
You should get:
/mnt/moosefs/replicadir/10meg:
chunks with 2 copies: 1
Also in the MooseFS web interface you should now be seeing files under the column '2' which is under 'Valid Copies', like this: 