Creating a KVM template, a DHCP server for KVM and automating the process
Creating our FreeBSD template
This is not so much of a how to but more of a quick checklist on the minimum things we need to do to create an instance that we'll use as a template.
0. FreeBSD installed in a virt
It's expected that the FreeBSD installation is already completed and you can access the machine over KVM.
1. Enable root access
Edit /etc/ssh/sshd_config Change the line
#PermitRootLogin no
to
PermitRootLogin yes
For help look here
2. Copy your SSH key to the template
The same key will be used by all clones of the template - for help look here. To do this you'll obviously need to have an SSH key pair on your host. If you haven't, follow the instructions here.
Once you have the key copy it to the host with by following these instructions. To summarize you need these on the host:
ssh-keygen
exec ssh-agent bash
ssh-add
ssh-copy-id -o StrictHostKeyChecking=no root@192.168.2.13
In the above replace '192.168.2.13' with the IP of your FreeBSD machine.
3. Set your defaultrouter and remove the hostname from the /etc/rc.conf
Make sure to edit /etc/rc.conf and make the following changes:
Add:
ifconfig_em0="DHCP"
Delete:
defaultrouter="192.168.2.1"
dnsmasq_enable="YES"
hostname="dhcp.weirdbricks.com"
Why are we deleting the hostname line? because when the hostname isn't set in /etc/rc.conf FreeBSD will try to pick one from DHCP upon reboot :)
4. Modify your /etc/dhclient.conf so it rejects your home router DHCP (if applicable)
Also make sure your file /etc/dhclient.conf has the lines:
interface "em0" {
reject 192.168.2.1;
}
5. Reduce the countdown time on boot
Edit the file: /boot/defaults/loader.conf - Find the line:
#autoboot_delay="10"
Uncomment and change it to a much lower number, for example:
autoboot_delay="1"
Then shutdown with:
poweroff
Set a FreeBSD virt as the DHCP/DNS server
It'd be awesome to set one of the KVM virts to as a FreeBSD DHCP/DNS server for our local network, but on the host it seems that dnsmasq is already running? what gives?
ps aux | grep -i dnsmasq | grep -v grep
Output:
nobody 3512 0.0 0.0 15524 836 ? S 19:15 0:00 /sbin/dnsmasq --conf-file=/var/lib/libvirt/dnsmasq/default.conf
And listening on port 53:
netstat -ntlp | grep dnsmasq
Output:
tcp 0 0 192.168.122.1:53 0.0.0.0:* LISTEN 3512/dnsmasq
The IP 192.168.122.1 is the one bound to the virbr0 interface:
ifconfig virbr0
Output:
virbr0: flags=4099<UP,BROADCAST,MULTICAST> mtu 1500
inet 192.168.122.1 netmask 255.255.255.0 broadcast 192.168.122.255
ether 72:a7:73:43:f1:1d txqueuelen 0 (Ethernet)
RX packets 0 bytes 0 (0.0 B)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
The answer to this is here. It looks like the libvirt daemon (libvirtd) is using it's own version of dnsmasq. To make things interesting I'm going to create a clone, set a static IP address and configure dnsmasq on it. To create the clone, first we'll need to shutdown the 'freebsd' virt (if it's running - if not, you'll just get an error)
sudo virsh destroy freebsd
Now clone the original 'freebsd' virt to a new one called 'dhcpserver':
virt-clone -o freebsd -n dhcpserver --auto-clone
The --auto-clone will figure the disk paths automatically Note: the clone process took about 2 1/2 minutes - this will vary depending on your system specs and of course the size of the VM.
Now we got the 2 following virts:
virsh list --all
Output:
Id Name State
----------------------------------------------------
- dhcpserver shut off
- freebsd shut off
Now start the 'dhcpserver' then connect to it over KVM. Since this is a clone of 'freebsd' it might be trying to get a DHCP address - it will fail since we set it to reject router DHCP offers and since there's no other DHCP server around :) In that case just type CTRL+C to stop trying to get a DHCP address.
Proceed with logging in and making the following modifications:
Follow the instructions from here to set it up so FreeBSD has a static IP.
So I'm adding those lines in /etc/rc.conf:
hostname="dhcp.weirdbricks.com"
ifconfig_em0="inet 192.168.2.99 netmask 255.255.255.0"
defaultrouter="192.168.2.1"
dnsmasq_enable="YES"
Restart networking and routing (source: here)
service netif restart; service routing restart
Bootstrap PKG:
env ASSUME_ALWAYS_YES=YES pkg bootstrap
Install the dnsmasq package:
pkg install dnsmasq
Add this configuration to /usr/local/etc/dnsmasq.conf - make sure to edit the dhcp-host line with the MAC address,IP address and hostname you want that machine to have, you'll need one line per host. This is based off my post here
interface=em0
domain=weirdbricks.com
dhcp-range=192.168.2.101,192.168.2.165,12h
dhcp-option=option:router,192.168.2.1
#dhcp reservations
#this one is just an example
#dhcp-host=52:54:00:da:25:59,192.168.2.90,example,infinite
#log DNS queries
log-queries
#log DHCP details
log-dhcp
#log file to use
log-facility=/var/log/dnsmasq.log
#dns stuff
domain-needed
bogus-priv
bind-interfaces
#option 6 = DNS server to use - this is important!
dhcp-option=6,192.168.2.99
#forward internet DNS requests to the router
server=192.168.2.1
Restart dnsmasq for changes to take effect:
service dnsmasq restart
Creating some clones
Now let's create a clone of the 'freebsd' virt - first we'll need to shutdown the 'freebsd' virt (if it's running):
sudo virsh destroy freebsd
Now clone the original 'freebsd' virt to a new one called 'freebsd-1':
virt-clone -o freebsd -n freebsd-1 --auto-clone
The --auto-clone will figure the disk paths automatically Note: the clone process took about 2 1/2 minutes - this will vary depending on your system specs and of course the size of the VM.
Now we got the following virts:
virsh list --all
Output:
Id Name State
----------------------------------------------------
5 dhcpserver running
- freebsd shut off
- freebsd-1 shut off
Now let's get the MAC address of the 'freebsd-1' machine so we can add it to our DHCP reservations on dhcpserver's dnsmasq configuration:
virsh domiflist freebsd-1 | grep bridge0 | awk '{print $5}'
Output:
52:54:00:df:b5:4e
SSH into 'freebsd' - remember now it has the static IP 192.168.2.99: (if you can't SSH make sure you've edited the /etc/ssh/sshd_config file to allow root logins)
ssh root@192.168.2.99
Now edit the dnsmasq configuration (/usr/local/etc/dnsmasq.conf) and add this line:
dhcp-host=52:54:00:df:b5:4e,192.168.2.200,freebsd-1,infinite
Exit and save and restart dnsmasq:
service dnsmasq restart
Now, back to the host start freebsd-1
sudo virsh start freebsd-1
You can start pinging 'freebsd-1''s IP address (192.168.2.200) if all went well it should start responding in a couple of minutes.
ping 192.168.2.200
Output:
PING 192.168.2.200 (192.168.2.200) 56(84) bytes of data.
64 bytes from 192.168.2.200: icmp_seq=1 ttl=64 time=0.464 ms
64 bytes from 192.168.2.200: icmp_seq=2 ttl=64 time=0.137 ms
64 bytes from 192.168.2.200: icmp_seq=3 ttl=64 time=0.127 ms
Very nice! OK, that's cool, but pinging the hostname from the host doesn't work :(
ping freebsd-1
Output:
ping: unknown host freebsd-1
On the CentOS 7 host simply edit /etc/resolv.conf - right now I got the following:
; generated by /usr/sbin/dhclient-script
nameserver 192.168.2.1
Change it to:
; generated by /usr/sbin/dhclient-script
nameserver 192.168.2.99
nameserver 192.168.2.1
And try again:
ping freebsd-1
Output:
PING freebsd-1 (192.168.2.200) 56(84) bytes of data.
64 bytes from freebsd-1.weirdbricks.com (192.168.2.200): icmp_seq=1 ttl=64 time=0.216 ms
64 bytes from freebsd-1.weirdbricks.com (192.168.2.200): icmp_seq=2 ttl=64 time=0.176 ms
Much better! Why can't I ping the hostname from the DNS/DHCP server? The same reason! You need to edit /etc/resolv.conf - this is what I currently got:
# Generated by resolvconf
nameserver 192.168.2.1
Change it to:
# Generated by resolvconf
nameserver 192.168.2.99
nameserver 192.168.2.1
Now that works too:
ping 192.168.2.200
Output:
PING 192.168.2.200 (192.168.2.200): 56 data bytes
64 bytes from 192.168.2.200: icmp_seq=0 ttl=64 time=0.269 ms
64 bytes from 192.168.2.200: icmp_seq=1 ttl=64 time=0.254 ms
How come this is already setup on the new hosts that get their options from DHCP? Remember this line in /usr/local/etc/dnsmasq.conf?
#option 6 = DNS server to use - this is important!
dhcp-option=6,192.168.2.99
That's what /etc/resolv.conf uses!
A short script
Here's a short script to automate the cloning and DHCP addition:
#!/bin/bash
NEW=$1
#replace ORIGINAL with the virt you're using as a template
ORIGINAL=freebsd
NEWIP=$2
virt-clone -o $ORIGINAL -n $NEW --auto-clone
ssh -o StrictHostKeyChecking=no root@dhcp.weirdbricks.com "echo dhcp-host=$MAC_ADDRESS,$NEWIP,$NEW,infinite >> /usr/local/etc/dnsmasq.conf; service dnsmasq restart"
Make the script executable:
chmod +x clone-kvm.sh
use it like this:
./clone-kvm.sh freebsd-3 192.168.2.202
Where freebsd3 is the new hostname and 192.168.2.202 is the new IP Now make a bunch of clones and start hitting them with Ansible.