View low bandwidth version

Archive for July, 2010

System Imaging for Free using G4L

Thursday, July 22nd, 2010

This is a copy of the notes that I wrote at AfNOG 2010 as a guide to using system imaging at future workshops. Unfortunately that wiki is not accessible without signing up for an account, so I’m posting the information here too.

How to Install Computer Labs

If you ever need to set up a large number of computers in identical configurations, you have a few options:

  • Install each one individually by hand
  • Automate the standard install process, for example using:
  • Configure one machine exactly how you like it, and then exactly duplicate the hard disk to the others (disk imaging)

The first option (manual installation) is extremely slow, tedious, error-prone, unlikely to result in identical machines, and does not speed up future installations or reinstallations.

The second option requires using rarely-used and less tested parts of the installer, slows down badly with multiple simultaneous installations (due to limited network bandwidth and bugs in the inetd TFTP server), and places limits on what you can customise. For example, it seems impossible to customise /etc/rc.conf using the installer on FreeBSD, and pre-installing SSH keys is tricky. I spent days writing a sysinstall script to automate the process. It would have taken just half an hour to set one machine up perfectly by hand, and then copy the system image onto all the other PCs in a few unattended hours.

Therefore I prefer the third option, system imaging.

What is System Imaging

Imaging is the process of making exact copies of one machine’s hard disk, including all partitions, onto another. This only works when the second hard disk is at least as large as the first. It works best when all the PCs are identical.

Imaging is independent of the operating system. You can image Windows, FreeBSD, any version of Linux, dual-boot and triple-boot installations, whatever you like.

We successfully used imaging to set up the PCs for these workshops:

How to Image

Many systems administrators have heard of Norton Ghost and Acronis True Image, two of the most popular commercial applications.

However, open source alternatives such as G4L (Linux-based) and its ancestor G4U (FreeBSD-based) are pretty good, and completely free. G4L however lacks a website, and it’s not obvious how best to use it, hence this post.

G4L is quite similar to G4U, and I could have used G4U instead. But I find the Linux kernel’s hardware support a bit better than FreeBSD’s, and G4L supports multicasting, which enables it to install many machines at the same time with good performance.

Using Ghost for Linux (G4L)

I’ve successfully used Ghost 4 Linux (G4L) versions 0.27 and 0.33 for this process. 0.33 has multicast support, which allows setting up an entire room in one go, without wasting network bandwidth copying the same 4 GB disk image to each of 50 machines independently.

Set up an FTP server on your network with an account that supports downloads and uploads (e.g. on a local server on your network). Make sure it has plenty of disk space free, perhaps 40 GB. Create an “img” directory under the FTP user’s home directory for the images.

Download G4L and burn some CDs, maybe about five copies, or set up network booting (this conflicts with FreeBSD PXE installation and may require BIOS setup changes to enable PXE).

It’s a good idea to explore G4L and get used to the options, but please be very careful, as it has the potential to wipe your hard disk! So please use a machine with a fresh hard disk or which you don’t mind wiping.

To boot into G4L (you will need to do this several times below, but not yet, unless you just want to explore):

  • Reboot or power up the machine
  • Press the key to choose boot device
  • If CD-ROM is not on the list, reboot, go into the BIOS and enable booting from CD-ROM
  • Choose to boot from the CD
  • Choose the default kernel at the GRUB screen (just press Enter)
    • If for some reason the default kernel doesn’t work, the machine hangs or crashes or doesn’t detect the network interface, then try one or two other kernels
  • Wait for the kernel and initrd to be loaded (two long lines of dots)
  • Then you can remove the CD, about one minute from cold boot, and start booting another PC
  • Press space to skip each of the information/advertising screens (about 8 of them)
  • Enter g4l at the prompt (if you go past this and get a shell, just type g4l at the shell prompt)
  • You can access other consoles with Ctrl-Alt-F1 to F4, log in as g4l with no password, and run g4l, ifconfig, ping or whatever
  • Choose Network Use (default)
  • Choose Raw Mode (default)
  • Check that you have an IP address (option B) or try again to acquire one by DHCP
  • If you can’t get an IP address by DHCP, check your cabling and DHCP server

Create a Restore Image (optional)

Back up one of your PCs if necessary (if you plan to restore the PCs later) by:

  • Follow the procedure above to get into Ghost for Linux
  • Enter the FTP server’s IP address, username and password
  • Choose an image name, e.g. backup_original_2010_07_22.img
  • Choose the back up option
  • Press Space to select the entire disk (mark it with an asterisk [*])
  • Start backing up the image

This process can take 1-2 hours. In the mean time…

Set up the Master PC

If you don’t already have a master computer set up, it’s a good idea to WIPE THE DISK first. This makes the image much smaller, and transfer much faster. Please DO NOT do this if you have anything valuable on the master computer, for example an existing operating system installation that you want to keep.

Boot G4L on the PC that you will use as the master. Use DD to wipe the entire disk with zeroes:

dd if=/dev/zero of=/dev/sda bs=1M

Install FreeBSD or whatever operating system(s) on the master PC, and set it up exactly the way you want all of the PCs to be. Examples include:

  • Install Gnome (gnome/gnome2)
  • Install Xorg (x11/xorg)
  • Install Firefox (www/firefox35)
  • Install Xpdf (print/xpdf)
  • Enable gnome and sshd in /etc/rc.conf, and add templates for the IP address configuration (this saves typing when setting all the machines to static IPs): 
    hostname="pc01.sse.ws.afnog.org"
ifconfig_bge0="dhcp"
# ifconfig_bge0="196.200.219.101/24"
defaultrouter="196.200.219.254"
gnome_enable="YES"
sshd_enable="YES"
  • Create a user account (e.g. username afnog, password afnog)
  • Log into Gnome, add firefox, terminal and the Downloads folder to your toolbar, and remove epiphany and evolution
  • Edit /etc/fstab and add the proc filesystem: 
    proc /proc procfs rw 0 0

    (this allows GDM to display the user list and shut down and restart the machine)

  • Edit /etc/profile and set the default pager to less by adding: 
    PAGER=less; export PAGER
  • Set the timezone by softlinking /etc/localtime to something like /usr/share/zoneinfo/Africa/Kigali
  • Create /etc/rc.local and have it run /usr/sbin/ntpd -qg to set the time once at boot

I recommend using DHCP on this machine. Otherwise all the imaged machines will boot up with the same IP address, causing IP address conflicts, and you will have to reconfigure them before you can access the Internet at all, or reconfigure them automatically.

Create some SSH keys for use in administering the machines. You may wish to set up the local server already and generate the keys there for security. I recommend adding the keys to /root/.ssh/authorized_keys. Please test that they work, and that sshd comes up automatically after boot!

Imaging the other PCs

On all the PCs (master and clones):

  • Boot G4L as above
  • Check that it has an IP address (option B)

Once a master is online, all the PCs will show “press any key to start”. Pressing any key on any computer will start all the machines imaging. If any PCs are not ready yet, you will have to cancel the imaging process on all of them and start again, or image those PCs later. So:

Start the master last! (when all the other PCs are ready)

Start the clones first, by following these steps on each one:

  • Choose UDP Multicast Client (option U)
  • Select the entire disk, /dev/sda with the space key
  • Say yes, you’re sure
  • When it says “Compressed UDP receiver”, it’s ready and waiting for a master to appear on the network

Then start the master:

  • Get ALL the clones ready, as above, before doing this!
  • On the master, choose UDP Multicast Server (option W)
  • Select the entire disk, /dev/sda, with the space key
  • Leave the options blank
  • Say yes, you’re sure
  • The master start accepting connections from clients, which will happen automatically. The screens on the clients will also change.
  • Please check that every client says “Press any key to start”.
  • If not, please check it for network problems, etc.
  • DO NOT stop or kill the server now, unless you want to visit every client again!
  • You can press Ctrl+C on the client and run g4l again to check the IP address, retry DHCP, and try the UDP Multicast Client option again.
  • This is your last chance to join any remaining clients to the group for this imaging session!
  • When all the clients are ready, press a key on the master to start transfer.

The master will show progress of the transfer, and an error line if any clients fail to respond. Clients that cause too many errors will be kicked out of the group and appear to “finish” early.

It’s difficult to tell if the imaging process finished successfully or failed on the clients. However it appears that FreeBSD is very good at detecting filesystem corruption, and will fail to boot if the image was not completely transferred. So you can test them by trying to boot FreeBSD and seeing if it boots completely or stops with a filesystem error. Ideally this would be improved in future versions of G4L.

Tags: , , , , , ,
Posted in Engineer's Log, Infrastructure, Linux, Open Source, System Administration | 2 Comments »

Mobiles for Scientific Research

Friday, July 9th, 2010

We know mobiles are very useful in areas where desktop computer and communications infrastructure is not easily available or affordable. And we’re very interested in mobile applications and scientific research in exactly these regions.

So I was very interested to see a new training workshop being run by the Science Dissemination Unit (SDU) of the Abdus Salam International Centre for Theoretical Physics (ICTP). The workshop is on Mobile Science: Sensing, Computing and Dissemination and the deadline for applications is tomorrow, July 10th.

Quoting from the announcement:

The Science Dissemination Unit (SDU) of the Abdus Salam International Centre for Theoretical Physics (ICTP), with the assistance of the University of Washington (USA) and of the UCLA Centerfor Embedded Networked Sensing (USA) will hold a Workshop on “Mobile Science: Sensing, Computing and Dissemination” in Trieste (Italy) from the 2 to the 5 of November 2010.

Mobile applications offer tremendous benefits to academic research and
education, and to society as a whole throughout the world. This is an
opportunity that deserves attention and promotion, especially in less
developed areas where mobile phones are the first telecommunications
technology in history to have more users than in the developed world.

The specific things that interested me were:

The Mobile Science workshop aims to engage the scientific community in developing countries in the design, development, and deployment of the newest mobile scientific applications;
i.e. advocating appropriate mobile applications in scientific
research/academia;
Participants will learn how to apply mobile technology tools to retrieve scientific data
I.e. designing mobile apps for science data collection;
how to apply appropriate web-based analysis to assimilate mobile data into scientific studies
I.e. web-based statistical analysis and presentation, like a free online version of SPSS? As far as I know this doesn’t exist yet. The closest that I can think of is the Google Docs spreadsheet, which is of course just a spreadsheet, requires an internet connection and doesn’t allow plugins for additional scientific analysis functionality. But there could be a very interesting app to develop here.
and how to share their scientific findings with a potentially large mobile audience.
I.e. low bandwidth design with an emphasis on web standards for cross-platform compatibility, so that it works on the largest number of mobile devices.

If you want to apply, better get on your bike (or modem?) because the deadline is tomorrow. If you want to do mobile scientific research applications, please get in touch, we’d like to help you.

Tags: , , , , ,
Posted in Mobiles, Offline, Open Source, PDAs, Web, agile, bandwidth, data collection | No Comments »

The reality of Internet access in Africa

Thursday, July 8th, 2010

When we talk about making the Internet accessible over slow and unreliable connections, we speak from experience.  Our recent work in Zambia, for example,  taught us a lot about the realities of Internet in the field, in rural Africa.

We were helping to deliver hands-on computer training for 150 young Zambian women as part of our ongoing work with Camfed. The training was framed as an introduction to email; for many it was a first introduction to computers.

We did not expect great Internet connectivity. We set up  a local email system  for the training to avoid dependency on a good connection. While practical, this approach wasn’t ideal as we wanted to offer as much real experience of the Internet as possible.

The rural high school where we were working  has a small IT lab that is connected to the Internet through a directional WIFI link to a near by youth skills centre where there is a VSAT link to the rest of the world.

The environment was harsh for network equipment. The mains power supply was was poor, with regular cuts and brown outs. The school and skills centre both had back-up generators but these  were started manually and not always at the same time. The route to the Internet could not have been called reliable. Neither building normally had UPSs to condition the power to their network equipment, though fortunately we had brought some with us.

Adjusting the antenna to improve signal-to-noise ratio

When conditions conspired to supply power to all the right places, Internet access at the school worked, but it wasn’t great. A VSAT connection has an unavoidably high latency and costs prohibit high bandwidth connections. The  Internet connection at the school was a nominal 1Mbit/s with a 1:10 contention ratio which means we were sharing our satellite bandwidth with nine other user sites. This works well in countries with abundant Internet connectivity as network use is effectively intermittent and sharing doesn’t cause much degradation of service. In Zambia, and much of Africa, a connection like his is often shared between a number  schools, hospitals and telecentres, etc., meaning network use is almost constant. Our experience of bandwidth was often that of the worst-case: 12Kbyte/s.  Though we soon found there were more serious problems with the connection.

Corruption

The web is surprisingly forgiving over a lossy connection experiencing data corruption, and at first it was easy to dismiss slow page load times as a limitation of the connection. Though it soon became obvious that something else was wrong. The image above shows corruption to image data downloaded from a popular mapping site.

A web page with corrupted content can be simply reloaded, but when downloading Ubuntu packages to update a machine we found the md5sums weren’t matching.

The following tests helped us uncover network faults:

  • Pinging hosts on the Internet route using a larger packet (-s packet size).
  • Downloading a large file with a known hash and comparing the computed value. If the hash value didn’t match, then repeating the download and checking if the computed value has changed.
  • Using MTR to see at what router packet loss is occurring.
  • Using Network Diagnostic Tool from a JAVA enabled browser to measure network performance.

Having identified the problem we wanted to establish that the corruption wasn’t happening on our own network: we repeated  the above tests while directly connected to the school’s router, and using more than one computer. The corruption was happening somewhere upstream of our connection, so we called our local ISP (by mobile telephone):

“Can we get an engineer to look into this?”

“Our engineers only visit Samfya district as part of a regular inspection”

“How often do your engineers come out?”…

“Once a month”.

No such visit was likely before we left Zambia. We needed  to find a workaround.

When running a shell over SSH on the trouble connection,  every so often the session would close with a “Corrupted MAC on input” error. This indicates that the last SSH packet failed verification, presumably because the underlying transport layers failed to guarantee the integrity of the data. It seemed like this characteristic could be useful in building a system to help download large files.

Being able to drop connections when ever packets are corrupted isn’t in itself going to help download complete files. But it does mean that a file transfer application running over SSH will never unknowingly write incorrect data to the file (up to the point that SSH MAC checks can detect the problem). This lead to the idea of a brute force solution by persistently running a file transfer application over SSH until the whole file is successfully downloaded. The file transfer application must be able to resume part-way through a file if the link is dropped. A suitable application is rsync.

If files are available on a remote host which can be accessed using public key authentication so that it can be left running unattended, the described work around becomes in shell script form;

return 1
until [ $? -eq 0 ] ; do
 rsync -axrv --partial --inplace <remote-host>:directory .
done

With this method we were able to retrieve the Ubuntu packages we needed up update our servers.

Is there a better way?

Having to manually transfer data through a third machine is inconvenient, a better solution would be transparent to the existing network. Although we can’t easily solve the underlying problem, which is the failure of the relatively small size of the TCP checksum to guarantee the data integrity (in this instance), it is possible to create the conditions where it’s no longer a problem. To do this we might look at reliable transport protocols implemented on top of UDP, which could then be used to tunnel the troubled TCP connections.

Network Diagnostic Tool

Posted in Uncategorized | No Comments »