AfNOG 2012 Setup

By Chris Wilson on 05 May 2012

I'm at the AfNOG Workshop 2012 in Serrekunda, Gambia as one of the instructors in the Scalable Services track (SS-E).

A local TV company has reported about us on the local news, and you can watch the video. (Thanks to Nadejda Loumbeva for finding it on YouTube!)

These are my notes from our setup for 2012, building on the virtualisation setup from last year.

We are hosting on CentOS 6 this year. I had a lot of problems installing the VirtualBox RPM, with this error message:

error: unpacking of archive failed: cpio: read failed - No such file or directory

That almost had me abandon CentOS, but it turned out to be a corrupt download file.

CentOS 6 seems to install headless, which saves us memory, but makes the configuration of VirtualBox more challenging. Luckily we had most of the VBoxManage commands from last year.

Remote headless installation

VirtualBox for CentOS doesn't appear to have VNC support at all. You need to install the extension pack, to enable RDP support, to get nice remote access to the machines to do the installation:

sudo VBoxManage extpack install Oracle_VM_VirtualBox_Extension_Pack-4.1.14-77440.vbox-extpack

Configure the Master virtual machine, ensuring that its first network interface is bridged to the host's physical Ethernet interface and promiscuous mode is enabled:

VBoxManage modifyvm Master \
	--memory 256 \
	--nic1 bridged --bridgeadapter1 p4p1 \
	--nicpromisc1 allow-all --vram 4 \
	--pae off --audio none --usb on --uart1 0x3f8 4 \
	--uartmode1 server /home/inst/.VirtualBox/Master-console.pipe

To attach a 20GB virtual hard disk image to the Master virtual machine for installation:

VBoxManage createhd --filename VirtualBox\ VMs/Master/FreeBSD.vdi \
	--size 20480 --format VDI
VBoxManage storageattach Master --storagectl "IDE Controller" --port 0 \
	--device 0 --type hdd --medium VirtualBox\ VMs/Master/FreeBSD.vdi

To attach a FreeBSD ISO image to the Master virtual machine for installation:

VBoxManage storagectl Master --name "IDE Controller" --add ide
VBoxManage storageattach Master --type dvddrive --storagectl "IDE Controller" \
	--port 1 --device 0 --medium FreeBSD-8.3-RELEASE-i386-dvd1.iso

You can remove the virtual disk from the virtual CD-ROM drive like this:

VBoxManage storageattach Master --type dvddrive --storagectl "IDE Controller" \
	--port 1 --device 0 --medium emptydrive

This seems to trigger a serious bug in FreeBSD that makes the system unbootable. So instead, you can leave it connected but change the boot order so that the VM boots from disk instead of CD:

VBoxManage modifyvm Master --boot1 disk 

To enable an RDP remote display on port 5900, for headless operation:

VBoxManage modifyvm Master --vrde=on --vrdeport=3380

To start the Master virtual machine:

VBoxManage startvm Master --type headless

If your virtual machine refuses to start, with an error like this:

[inst@host1 ~]$ VBoxManage startvm Master
Waiting for VM "Master" to power on...
VBoxManage: error: The virtual machine 'Master' has terminated unexpectedly during startup with exit code 0
VBoxManage: error: Details: code NS_ERROR_FAILURE (0x80004005), component Machine, interface IMachine, callee 

It probably means that you're running the command remotely without a GUI, and you forgot to add --type headless, so the graphical display can't start. Thanks VirtualBox!

Install a caching nameserver such as BIND on the host, and when configuring the Master machine, set the DNS server to 10.0.3.2 to use the host's DNS. This avoids problems using a DNS server on the network while all the clones are using the same IP address, resulting in long delays for SSH logins.

Remote virtual serial console

ConMan does indeed work for host to guest communication over the serial pipe, as an alternative to RDP. The following line enables the UNIX socket for the serial console, if you haven't done this already (requires the virtual machine to be stopped):

VBoxManage modifyvm Master --uartmode1 server /home/inst/.VirtualBox/Master-console.pipe

The following line in /etc/conman.conf on the host enables the conman server:

console name="Master" dev="unix:/home/inst/.VirtualBox/Master-console.pipe"

To enable a serial terminal in a FreeBSD guest, add the following line to /etc/ttys:

ttyu0   "/usr/libexec/getty std.9600"   vt100   on secure

Then, after you start (or restart) conmand, you should be able to connect to the Master's serial console like this:

conman Master

The escape sequence to terminate the conman session is &. But this still isn't a good way to mass-control multiple machines because conman doesn't like commands being piped into it.

Remote port-forwarded SSH management

You can set up NAT port forwarding to a second virtual network interface in the guest, configured by DHCP, like this:

VBoxManage modifyvm Master --nic2 nat --natpf2 ssh,tcp,,4400,,22

This is really helpful when all your virtual machines start off with the same IP address. As long as you have SSH keys, you can iterate over all of them, using the host IP addresses, and update their network configurations to give them unique, preassigned IP addresses.

FreeBSD proxy server configuration

To save bandwidth and speed up package installation, I added the following to /etc/profile:

export FTP_PROXY=http://196.200.223.1:3128
export HTTP_PROXY=http://196.200.223.1:3128
export http_proxy=http://196.200.223.1:3128

Bulk virtual machine management

This is a shorter and more efficient script to create the guest virtual machines by snapshotting and cloning the master image, resulting in less disk usage (faster host and guest cloning) and more efficient disk cache usage (faster guest operation):

create-vms.sh

#!/bin/bash


set -e
set -x

master=Master
snapshot="Clone Source `date --rfc-3339=seconds`"
VBoxManage snapshot "$master" take "$snapshot" --pause

cd ~/"VirtualBox VMs"
for i in {1..7}; do
	# echo $i
	vmname=VM0$i
	if [ "$1" = "-f" ]; then
		VBoxManage unregistervm "$vmname" --delete || true
	fi
	VBoxManage clonevm "$master" --name "$vmname" --options link \
		--options keepdisknames --snapshot "$snapshot" --register
	VBoxManage modifyvm "$vmname" --memory 256 \
		--natpf2 "ssh-$vmname",tcp,,$[4400+$i],,22 \
		--uartmode1 server /home/inst/.VirtualBox/"$vmname"-console.pipe \
		--vrde=on --vrdeport=$[3380+$i]
done

Note that you have to use a unique snapshot name for each new snapshot of the master image. VirtualBox doesn't stop you from reusing an old name, but when you clone the master, you get a copy of the oldest snapshot with that name, not the latest one.

This script logs into each VM using the NAT port forwarding from the host, which works even when their IP address is not configured yet, sets their hostname and IP address and reboots them:

configure-vms.sh

#!/bin/bash


# set -e
set -x

guests_per_host=7

for host in 1; do
	hostname="196.200.219.$[200+$host]"
	for guest in `seq 1 $guests_per_host`; do
		nat_ssh_port=$[4400+$guest]
		guest_ip_suffix="$[($host-1)*$guests_per_host + $guest]"
		guest_ip="196.200.219.$guest_ip_suffix"
		guest_name="vm0$guest-host$host-ip$guest_ip_suffix.sse.ws.afnog.org"
		ssh root@$hostname -p $nat_ssh_port \
			-o "StrictHostKeyChecking no" << EOF
set -x
sed 	-e 's/hostname=.*/hostname="$guest_name"/' \
	-e 's/ifconfig_em0=.*/ifconfig_em0="inet $guest_ip netmask 255.255.255.0"/' \
	-i '' /etc/rc.conf
reboot
EOF
	done
done

Starting virtual machines automatically

We want all the virtual machines to start at boot time. My usual sudo trick doesn't work in /etc/rc.local, but thanks to this blog post I found a way:

/etc/rc.local

su --session-command "VBoxManage startvm VM01 --type headless" inst
su --session-command "VBoxManage startvm VM02 --type headless" inst
su --session-command "VBoxManage startvm VM03 --type headless" inst
su --session-command "VBoxManage startvm VM04 --type headless" inst
su --session-command "VBoxManage startvm VM05 --type headless" inst
su --session-command "VBoxManage startvm VM06 --type headless" inst
su --session-command "VBoxManage startvm VM07 --type headless" inst

We also want the host to shut them down cleanly:

/etc/default/virtualbox-ose

SHUTDOWN_USERS=inst
SHUTDOWN=acpibutton

Assign unique MAC addresses

After creating the remaining virtual machine hosts, by cloning the first one, and if you update and copy out the virtual machine disk images again, it's important to randomise all the MAC addresses of all the hosts. If two virtual machines have the same MAC address, routers will have problems learning their addresses. You can do that in bulk with this command:

for i in `seq 201 208`; do
ssh -t inst@196.200.219.$i '
./manage-vms.sh controlvm acpipowerbutton
./manage-vms.sh modifyvm --macaddress1 auto
./manage-vms.sh startvm --type headless'
done

Automatic Ubuntu desktop installation

We received 12 desktops with Windows in various states of repair, and decided to reformat them all and install Ubuntu. Since there were several hardware variations, I did this using Debian/Ubuntu installation preseeding instead of system imaging.

Preseed options which need to be decided before networking is initialised cannot be stored in the preseed file, so they must be passed to the kernel on the command line:

• debian-installer/locale=en_US
• netcfg/get_hostname=desktop.sse.ws.afnog.org
• console-setup/ask_detect=false
• console-setup/layoutcode=us

This is the pxelinux configuration that we used:

/var/lib/tftpboot/pxelinux.cfg/default

label ubuntu
  menu label Install ^Ubuntu (10.04.4 alternate i386)
  kernel ubuntu-10.04.4-alternate-i386/linux
  append initrd=ubuntu-10.04.4-alternate-i386/initrd.gz url=http://196.200.219.200/tftp/ubuntu-10.04.4-alternate-i386/preseed.txt netcfg/get_hostname=desktop.sse.ws.afnog.org debian-installer/locale=en_US console-setup/ask_detect=false console-setup/layoutcode=us
  menu default

Using preseeding, we were able to automate almost all of the installation process, providing a nearly unattended install. I wasn't able to preseed the partition selection for some reason; the instructions did not appear to work.

This is the preseed configuration file we used:

preseed.txt

d-i netcfg/wireless_wep string
d-i mirror/country string enter information manually
d-i mirror/http/hostname string 196.200.219.200
d-i mirror/http/directory string /ubuntu-10.04.4-alternate-i386/ubuntu/
d-i mirror/http/proxy string http://196.200.219.200:3128/
d-i partman-auto/init_automatically_partition select Guided - use entire disk
d-i partman-auto/disk string /dev/discs/disc0/disc
d-i partman-auto/method string regular
d-i partman-auto/purge_lvm_from_device boolean true
d-i partman-lvm/confirm boolean true
d-i partman-auto/choose_recipe \
       select All files in one partition (recommended for new users)
d-i partman/confirm_write_new_label boolean true
d-i partman/choose_partition \
       select Finish partitioning and write changes to disk
d-i partman/confirm boolean true
d-i clock-setup/utc boolean true
d-i clock-setup/ntp boolean true
d-i time/zone string Africa/Banjul
d-i passwd/user-fullname string AfNOG User
d-i passwd/username string afnog
d-i passwd/user-password password afnog
d-i passwd/user-password-again password afnog
d-i user-setup/allow-password-weak boolean true
d-i user-setup/encrypt-home boolean false
d-i grub-installer/only_debian boolean true
d-i grub-installer/with_other_os boolean true
tasksel tasksel/first multiselect standard, ubuntu-desktop
popularity-contest popularity-contest/participate boolean true
d-i pkgsel/update-policy select unattended-upgrades
d-i finish-install/reboot_in_progress note
xserver-xorg xserver-xorg/autodetect_monitor boolean true
xserver-xorg xserver-xorg/config/monitor/lcd boolean true

I ran into this issue while preseeding the Lucid desktop install: the installer reports an error and refuses to proceed:

http://server/ubuntu/lucid/dists/lucid/restricted/binary-i386/Packages was corrupt

The solution presented there basically worked. The steps I used were:

sudo cp -a ubuntu-10.04.4-alternate-i386/dists/lucid/restricted/binary-i386/ lucid-restricted-binary-i386.bugfix
cd lucid-restricted-binary-i386.bugfix/
sudo gunzip Packages.gz 
cd ..
sudo mount --bind lucid-restricted-binary-i386.bugfix ubuntu-10.04.4-alternate-i386/dists/lucid/restricted/binary-i386/

If you use a Squid cache in front of your Ubuntu repository web server, for caching security updates, then it will cache the corrupted Packages file that you deleted in the above step, and the error will persist. I recommend you exclude the local web server from your Squid cache like this:

/etc/squid/squid.conf

acl localweb dstdomain 196.200.219.200
cache deny localweb