Lampros - Weird Bricks

Creating a KVM template, a DHCP server for KVM and automating the process

10 October, 2014 | KVM

Creating our FreeBSD template

This is not so much of a how to but more of a quick checklist on the minimum things we need to do to create an instance that we'll use as a template.

0. FreeBSD installed in a virt

It's expected that the FreeBSD installation is already completed and you can access the machine over KVM.

1. Enable root access

Edit /etc/ssh/sshd_config Change the line

#PermitRootLogin no

to

PermitRootLogin yes

For help look here

2. Copy your SSH key to the template

The same key will be used by all clones of the template - for help look here. To do this you'll obviously need to have an SSH key pair on your host. If you haven't, follow the instructions here.

Once you have the key copy it to the host with by following these instructions. To summarize you need these on the host:

ssh-keygen
exec ssh-agent bash
ssh-add
ssh-copy-id -o StrictHostKeyChecking=no root@192.168.2.13

In the above replace '192.168.2.13' with the IP of your FreeBSD machine.

3. Set your defaultrouter and remove the hostname from the /etc/rc.conf

Make sure to edit /etc/rc.conf and make the following changes: Add:

ifconfig_em0="DHCP"

Delete:

defaultrouter="192.168.2.1"
dnsmasq_enable="YES"
hostname="dhcp.weirdbricks.com" 

Why are we deleting the hostname line? because when the hostname isn't set in /etc/rc.conf FreeBSD will try to pick one from DHCP upon reboot :)

4. Modify your /etc/dhclient.conf so it rejects your home router DHCP (if applicable)

Also make sure your file /etc/dhclient.conf has the lines:

interface "em0" {
  reject 192.168.2.1;
}

5. Reduce the countdown time on boot

Edit the file: /boot/defaults/loader.conf Find the line:

#autoboot_delay="10"

Uncomment and change it to a much lower number, for example:

autoboot_delay="1"

Then shutdown with:

poweroff

Set a FreeBSD virt as the DHCP/DNS server

It'd be awesome to set one of the KVM virts to as a FreeBSD DHCP/DNS server for our local network, but on the host it seems that dnsmasq is already running? what gives?

ps aux | grep -i dnsmasq | grep -v grep

Output:

nobody    3512  0.0  0.0  15524   836 ?        S    19:15   0:00 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf

And listening on port 53:

netstat -ntlp | grep dnsmasq

Output:

tcp        0      0 192.168.122.1:53        0.0.0.0:*               LISTEN      3512/dnsmasq

The IP 192.168.122.1 is the one bound to the virbr0 interface:

ifconfig virbr0

Output:

virbr0: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 192.168.122.1  netmask 255.255.255.0  broadcast 192.168.122.255
        ether 72:a7:73:43:f1:1d  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

The answer to this is here. It looks like the libvirt daemon (libvirtd) is using it's own version of dnsmasq. To make things interesting I'm going to create a clone, set a static IP address and configure dnsmasq on it. To create the clone, first we'll need to shutdown the 'freebsd' virt (if it's running - if not, you'll just get an error)

sudo virsh destroy freebsd

Now clone the original 'freebsd' virt to a new one called 'dhcpserver':

virt-clone -o freebsd -n dhcpserver --auto-clone

The --auto-clone will figure the disk paths automatically Note: the clone process took about 2 1/2 minutes - this will vary depending on your system specs and of course the size of the VM Now we got the 2 following virts:

virsh list --all

Output:

 Id    Name                           State
----------------------------------------------------
 -     dhcpserver                     shut off
 -     freebsd                        shut off

Now start the 'dhcpserver' then connect to it over KVM. Since this is a clone of 'freebsd' it might be trying to get a DHCP address - it will fail since we set it to reject router DHCP offers and since there's no other DHCP server around :) In that case just type CTRL+C to stop trying to get a DHCP address. Proceed with logging in and making the following modifications: Follow the instructions from here to set it up so FreeBSD has a static IP. So I'm adding those lines in /etc/rc.conf:

hostname="dhcp.weirdbricks.com"
ifconfig_em0="inet 192.168.2.99 netmask 255.255.255.0"
defaultrouter="192.168.2.1"
dnsmasq_enable="YES"

Restart networking and routing (source: here)

service netif restart; service routing restart

Bootstrap PKG:

env ASSUME_ALWAYS_YES=YES pkg bootstrap

Install the dnsmasq package:

pkg install dnsmasq

Add this configuration to /usr/local/etc/dnsmasq.conf - make sure to edit the dhcp-host line with the MAC address,IP address and hostname you want that machine to have, you'll need one line per host. This is based off my post here

interface=em0
domain=weirdbricks.com
dhcp-range=192.168.2.101,192.168.2.165,12h
dhcp-option=option:router,192.168.2.1

#dhcp reservations
#this one is just an example
#dhcp-host=52:54:00:da:25:59,192.168.2.90,example,infinite

#log DNS queries
log-queries
#log DHCP details
log-dhcp
#log file to use
log-facility=/var/log/dnsmasq.log

#dns stuff
domain-needed
bogus-priv
bind-interfaces
#option 6 = DNS server to use - this is important!
dhcp-option=6,192.168.2.99
#forward internet DNS requests to the router
server=192.168.2.1

Restart dnsmasq for changes to take effect:

service dnsmasq restart

Creating some clones

Now let's create a clone of the 'freebsd' virt - first we'll need to shutdown the 'freebsd' virt (if it's running):

sudo virsh destroy freebsd

Now clone the original 'freebsd' virt to a new one called 'freebsd-1':

virt-clone -o freebsd -n freebsd-1 --auto-clone

The --auto-clone will figure the disk paths automatically Note: the clone process took about 2 1/2 minutes - this will vary depending on your system specs and of course the size of the VM Now we got the following virts:

virsh list --all

Output:

 Id    Name                           State
----------------------------------------------------
 5     dhcpserver                     running
 -     freebsd                        shut off
 -     freebsd-1                      shut off

Now let's get the MAC address of the 'freebsd-1' machine so we can add it to our DHCP reservations on dhcpserver's dnsmasq configuration:

virsh domiflist freebsd-1 | grep bridge0 | awk '{print $5}'

Output:

52:54:00:df:b5:4e

SSH into 'freebsd' - remember now it has the static IP 192.168.2.99: (if you can't SSH make sure you've edited the /etc/ssh/sshd_config file to allow root logins)

ssh root@192.168.2.99

Now edit the dnsmasq configuration (/usr/local/etc/dnsmasq.conf) and add this line:

dhcp-host=52:54:00:df:b5:4e,192.168.2.200,freebsd-1,infinite

Exit and save and restart dnsmasq:

service dnsmasq restart

Now, back to the host start freebsd-1

sudo virsh start freebsd-1

You can start pinging 'freebsd-1''s IP address (192.168.2.200) if all went well it should start responding in a couple of minutes.

ping 192.168.2.200

Output:

PING 192.168.2.200 (192.168.2.200) 56(84) bytes of data.
64 bytes from 192.168.2.200: icmp_seq=1 ttl=64 time=0.464 ms
64 bytes from 192.168.2.200: icmp_seq=2 ttl=64 time=0.137 ms
64 bytes from 192.168.2.200: icmp_seq=3 ttl=64 time=0.127 ms

Very nice! OK, that's cool, but pinging the hostname from the host doesn't work :(

ping freebsd-1

Output:

ping: unknown host freebsd-1

On the CentOS 7 host simply edit /etc/resolv.conf - right now I got the following:

; generated by /usr/sbin/dhclient-script
nameserver 192.168.2.1

Change it to:

; generated by /usr/sbin/dhclient-script
nameserver 192.168.2.99
nameserver 192.168.2.1

And try again:

ping freebsd-1

Output:

PING freebsd-1 (192.168.2.200) 56(84) bytes of data.
64 bytes from freebsd-1.weirdbricks.com (192.168.2.200): icmp_seq=1 ttl=64 time=0.216 ms
64 bytes from freebsd-1.weirdbricks.com (192.168.2.200): icmp_seq=2 ttl=64 time=0.176 ms

Much better! Why can't I ping the hostname from the DNS/DHCP server? The same reason! You need to edit /etc/resolv.conf - this is what I currently got:

# Generated by resolvconf
nameserver 192.168.2.1

Change it to:

# Generated by resolvconf
nameserver 192.168.2.99
nameserver 192.168.2.1

Now that works too:

ping 192.168.2.200

Output:

PING 192.168.2.200 (192.168.2.200): 56 data bytes
64 bytes from 192.168.2.200: icmp_seq=0 ttl=64 time=0.269 ms
64 bytes from 192.168.2.200: icmp_seq=1 ttl=64 time=0.254 ms

How come this is already setup on the new hosts that get their options from DHCP? Remember this line in /usr/local/etc/dnsmasq.conf?

#option 6 = DNS server to use - this is important!
dhcp-option=6,192.168.2.99

That's what /etc/resolv.conf uses!

A short script

Here's a short script to automate the cloning and DHCP addition:

#!/bin/bash

NEW=$1
#replace ORIGINAL with the virt you're using as a template
ORIGINAL=freebsd
NEWIP=$2

virt-clone -o $ORIGINAL -n $NEW --auto-clone
MAC_ADDRESS=`virsh domiflist $NEW | grep bridge0 | awk '{print $5}'`
ssh -o StrictHostKeyChecking=no root@dhcp.weirdbricks.com "echo dhcp-host=$MAC_ADDRESS,$NEWIP,$NEW,infinite >> /usr/local/etc/dnsmasq.conf; service dnsmasq restart"

Make the script executable:

chmod +x clone-kvm.sh

use it like this:

 ./clone-kvm.sh freebsd-3 192.168.2.202

Where freebsd3 is the new hostname and 192.168.2.202 is the new IP Now make a bunch of clones and start hitting them with Ansible.