NBD - Network Block Device - an alternative to iSCSI

Most folks are familiar with iSCSI. In a nutshell, iSCSI is a way to share a block device over the network. You can format the device and use it as if it were a local disk. But there’s also a way less popular alternative to iSCSI called NBD. Network Block Device.

Important note - you will either need to recompile your current kernel to add NBD support, or install a kernel version newer than 3.6.

In this example I’ll demonstrate NBD by upgrading the kernel on a CentOS 6.7 machine. To do that we’ll install the ELRepo (Enterprise Linux Repository) first.

On the NBD server:

1. First, let’s see what our current kernel is using the `uname -r` command:

uname -r
2.6.32-573.18.1.el6.x86_64

2. Let’s check if nbd is already there by trying to load the module - to do this we use the modprobe command:

[root@nbdserver ~]# modprobe nbd
FATAL: Module nbd not found.

3. OK, so we have confirmed that the NBD module does not exist. Let’s install a newer kernel - first import the ELRepo key:

rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org

Note: the above command has no output.

4. Now install the repository:

yum -y install http://www.elrepo.org/elrepo-release-6-6.el6.elrepo.noarch.rpm

5. Check the available version of the kernel-lt package:

yum -y --enablerepo=elrepo-kernel list available kernel-lt

Output:

yum -y --enablerepo=elrepo-kernel info kernel-lt
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirrors.usc.edu
 * elrepo: repos.lax-noc.com
 * elrepo-kernel: repos.lax-noc.com
 * extras: mirrors.evowise.com
 * updates: mirror.math.princeton.edu
Available Packages
Name : kernel-lt
Arch : x86_64
Version : 3.10.104
Release : 1.el6.elrepo
Size : 33 M
Repo : elrepo-kernel
Summary : The Linux kernel. (The core of any Linux-based operating system.)
URL : https://www.kernel.org/
License : GPLv2
Description : This package provides the Linux kernel (vmlinuz), the core of any
 : Linux-based operating system. The kernel handles the basic functions
 : of the OS: memory allocation, process allocation, device I/O, etc.

6. Now install the latest kernel-lt (lt = long term support)

yum -y --enablerepo=elrepo-kernel install kernel-lt

7. Check what kernel grub is now set to use by grepping for the words “default” and “title” in /etc/grub.conf:

egrep "default|title" /boot/grub/grub.conf
default=1
title CentOS (3.10.104-1.el6.elrepo.x86_64)
title CentOS (2.6.32-573.18.1.el6.x86_64)

See how "default=1" ? Since grub starts counting from 0, this means that it will use the old kernel: title CentOS (2.6.32-573.18.1.el6.x86_64).

Let’s change it to use the first one (0) by using a sed one-liner:

sed -i -- 's/default=1/default=0/g' /boot/grub/grub.conf

Confirm:

egrep "default|title" /boot/grub/grub.conf
default=0
title CentOS (3.10.104-1.el6.elrepo.x86_64)
title CentOS (2.6.32-573.18.1.el6.x86_64)

Excellent.

8 Now reboot:

reboot

9. Once the server is back up, login and check the kernel version - we do this using the `uname` command:

uname -r
3.10.104-1.el6.elrepo.x86_64

Great!

10. Now attempt to load the NBD kernel module:

modprobe nbd && lsmod | grep -i nbd
nbd 17819 0

11. Now install epel so that we can get the NBD tools:

yum -y install epel-release
yum -y install nbd

12. Now, the system I’m on does not have a secondary drive to dedicate to NBD. I’m going to create a loopback device and export it with NBD - this one will only be 10GB:

truncate nbd-disk-1.img --size 10G
nbd-server 1043 /root/nbd-disk-1.img

I also got the following output, which can be ignored:

nbd-server 1043 /root/nbd-disk-1.img

** (process:1346): WARNING **: Specifying an export on the command line is deprecated.

** (process:1346): WARNING **: Please use a configuration file instead.

Let’s also see what netstat thinks about this:

netstat -ntulp | grep nbd
tcp 0 0 0.0.0.0:1043 0.0.0.0:* LISTEN 1347/nbd-server 
tcp 0 0 :::47485 :::* LISTEN 1347/nbd-server

Only TCP ports are used, cool.

Now on the client server:

1. Install a newer kernel as well (same steps as above, summarized) :

rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
yum -y install http://www.elrepo.org/elrepo-release-6-6.el6.elrepo.noarch.rpm
yum -y --enablerepo=elrepo-kernel install kernel-lt
yum -y install epel-release
yum -y install nbd
sed -i -- 's/default=1/default=0/g' /boot/grub/grub.conf
reboot

2. Once you relogin, load the `nbd` module:

modprobe nbd && lsmod | grep -i nbd
nbd 17819 0

3. This will give you a bunch of NBD devices:

ls -al /dev/nbd*
brw-rw---- 1 root disk 43, 0 Jan 2 05:46 /dev/nbd0
brw-rw---- 1 root disk 43, 1 Jan 2 05:46 /dev/nbd1
brw-rw---- 1 root disk 43, 10 Jan 2 05:46 /dev/nbd10
brw-rw---- 1 root disk 43, 11 Jan 2 05:46 /dev/nbd11
brw-rw---- 1 root disk 43, 12 Jan 2 05:46 /dev/nbd12
brw-rw---- 1 root disk 43, 13 Jan 2 05:46 /dev/nbd13
brw-rw---- 1 root disk 43, 14 Jan 2 05:46 /dev/nbd14
brw-rw---- 1 root disk 43, 15 Jan 2 05:46 /dev/nbd15
brw-rw---- 1 root disk 43, 2 Jan 2 05:46 /dev/nbd2
brw-rw---- 1 root disk 43, 3 Jan 2 05:46 /dev/nbd3
brw-rw---- 1 root disk 43, 4 Jan 2 05:46 /dev/nbd4
brw-rw---- 1 root disk 43, 5 Jan 2 05:46 /dev/nbd5
brw-rw---- 1 root disk 43, 6 Jan 2 05:46 /dev/nbd6
brw-rw---- 1 root disk 43, 7 Jan 2 05:46 /dev/nbd7
brw-rw---- 1 root disk 43, 8 Jan 2 05:46 /dev/nbd8
brw-rw---- 1 root disk 43, 9 Jan 2 05:46 /dev/nbd9

4. Let’s try to connect to the server’s NBD drive we exported earlier:

nbd-client 10.42.226.1 1043 /dev/nbd0

The output I got was:

nbd-client 10.42.226.1 1043 /dev/nbd0
Negotiation: ..size = 10240MB
bs=1024, sz=10737418240 bytes

Explanation of the above:

10.42.226.1 = the ip of the NBD server

1043 = the port of the NBD server

/dev/nbd0 = where we want to attach the device locally

5. Now let’s check what lsblk shows:

lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 80G 0 disk 
└─sda1 8:1 0 80G 0 part /
nbd0 43:0 0 10G 0 disk

And there’s the device!

6. Try creating a filesystem on it: mkfs.ext4 /dev/nbd0 :

mkfs.ext4 /dev/nbd0 
mke2fs 1.41.12 (17-May-2010)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
655360 inodes, 2621440 blocks
131072 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=2684354560
80 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks: 
 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632

Writing inode tables: done 
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 29 mounts or
180 days, whichever comes first. Use tune2fs -c or -i to override

7. Now mount it using `mount /dev/nbd0 /media -v` (the -v is so we can get some verbose output)

mount /dev/nbd0 /media -v
mount: you didn't specify a filesystem type for /dev/nbd0
 I will try type ext4
/dev/nbd0 on /media type ext4 (rw)

8. Check `df -hT`:

df -hT
Filesystem Type Size Used Avail Use% Mounted on
/dev/sda1 ext4 79G 916M 74G 2% /
tmpfs tmpfs 1004M 0 1004M 0% /dev/shm
/dev/nbd0 ext4 9.8G 23M 9.2G 1% /media

Done!!! You can use it as any other device.

Lampros - Weird Bricks

NBD - Network Block Device - an alternative to iSCSI

On the NBD server:

1. First, let’s see what our current kernel is using the `uname -r` command:

2. Let’s check if nbd is already there by trying to load the module - to do this we use the modprobe command:

3. OK, so we have confirmed that the NBD module does not exist. Let’s install a newer kernel - first import the ELRepo key:

4. Now install the repository:

5. Check the available version of the kernel-lt package:

6. Now install the latest kernel-lt (lt = long term support)

7. Check what kernel grub is now set to use by grepping for the words “default” and “title” in /etc/grub.conf:

8 Now reboot:

9. Once the server is back up, login and check the kernel version - we do this using the `uname` command:

10. Now attempt to load the NBD kernel module:

11. Now install epel so that we can get the NBD tools:

12. Now, the system I’m on does not have a secondary drive to dedicate to NBD. I’m going to create a loopback device and export it with NBD - this one will only be 10GB:

Now on the client server:

1. Install a newer kernel as well (same steps as above, summarized) :

2. Once you relogin, load the `nbd` module:

3. This will give you a bunch of NBD devices:

4. Let’s try to connect to the server’s NBD drive we exported earlier:

5. Now let’s check what lsblk shows:

6. Try creating a filesystem on it: mkfs.ext4 /dev/nbd0 :

7. Now mount it using `mount /dev/nbd0 /media -v` (the -v is so we can get some verbose output)

8. Check `df -hT`:

Sources:

On the NBD server:

1. First, let’s see what our current kernel is using the uname -r command:

2. Let’s check if nbd is already there by trying to load the module - to do this we use the modprobe command:

3. OK, so we have confirmed that the NBD module does not exist. Let’s install a newer kernel - first import the ELRepo key:

4. Now install the repository:

5. Check the available version of the kernel-lt package:

6. Now install the latest kernel-lt (lt = long term support)

7. Check what kernel grub is now set to use by grepping for the words “default” and “title” in /etc/grub.conf:

8 Now reboot:

9. Once the server is back up, login and check the kernel version - we do this using the uname command:

10. Now attempt to load the NBD kernel module:

11. Now install epel so that we can get the NBD tools:

12. Now, the system I’m on does not have a secondary drive to dedicate to NBD. I’m going to create a loopback device and export it with NBD - this one will only be 10GB:

Now on the client server:

1. Install a newer kernel as well (same steps as above, summarized) :

2. Once you relogin, load the nbd module:

3. This will give you a bunch of NBD devices:

4. Let’s try to connect to the server’s NBD drive we exported earlier:

5. Now let’s check what lsblk shows:

6. Try creating a filesystem on it: mkfs.ext4 /dev/nbd0 :

7. Now mount it using mount /dev/nbd0 /media -v (the -v is so we can get some verbose output)

8. Check df -hT:

Sources:

1. First, let’s see what our current kernel is using the `uname -r` command:

9. Once the server is back up, login and check the kernel version - we do this using the `uname` command:

2. Once you relogin, load the `nbd` module:

7. Now mount it using `mount /dev/nbd0 /media -v` (the -v is so we can get some verbose output)

8. Check `df -hT`: