• A User Guide for the system that we developed for UNICEF, IDS and RuralNet Zambia
• A Developers’ Guide for anyone wishing to build something similar
• Notes on lessons learned that may assist future implementers

Project goals

Automate the data entry part of a long paper-based survey, by replacing the paper forms with electronic devices.

Hardware and application selection

The survey has several long and complex questions, and long sets of multiple-choice answers. The data collection needs to be done in dusty rural Zambia, and the devices might need to be used for a full day without power. Collected data should be sent wirelessly to a secure data repository at some time after collection.

Text entry is required for many fields. That means either a real keyboard with keys, or a sufficiently large touch screen to type comfortably on. Use of the device camera, and presentation of reports and graphs on the same device, might be required in future.

Two possible hardware platforms were identified:

• Tablet laptops with touch screens
• Tablet mobile devices (iPad or Android tablet)

We selected the latter for this project due to lower cost, lighter weight, better usability and longer battery life.

The available software options that we identified were:

• EpiSurveyor (Java J2ME, partly closed source, we have used before and fixed bugs)
• OpenXdata (Java J2ME, open source, developed and supported by an Aptivate alumnus among others)
• Open Data Kit (ODK) (Android, open source, active community)
• Bespoke online/offline survey in HTML5

Of these, we eliminated EpiSurveyor and OpenXdata due to lack of compatibility with the hardware platform(s) we had chosen.

We chose ODK over a bespoke system due to limited time available for development, and ability to easily take photos and record GPS coordinates using the device’s hardware.

Of the available Android tablet devices, we chose the Samsung Galaxy Tab for the pilot project, due to its high quality construction. For future projects we would probably use a lower cost device; see the lessons learned for details.

Form creation

Since the survey is quite long (about 230 questions) we wanted an easy way to enter the questions. The ODK application requires the form to be in XForms format. We identified the following tools for creating XForms:

• ODK Build
• PurcForms
• Kobo Form Builder
• XLS2XForm (actually the pyxform fork)

We decided to use XLS2XForm, which enabled us to enter the large number of questions easily in Excel. The others all have graphical builders, which have advantages and disadvantages for less technical users:

• More visually appealing
• All available options presented visually (types of controls, groups, etc.)
• Less likely to make a mistake and produce an invalid form
• Cumbersome user interface slows down data entry

Unfortunately, none of these designers were able to import an existing form in XForms format, which means that the modifiable “source code” of the form must be maintained in a “proprietary” format in each case, and it’s difficult to switch between tools.

You can download the conversion tools, and the Excel spreadsheet with the completed questionnaire as we delivered it to RuralNet, here. RuralNet staff, please use the latest version of the spreadsheet that you can find locally. To use the tools, you will need to download and install Python 2.7 and Java (JRE). Then download the tools as a ZIP file and extract it somewhere. I recommend that you keep the master copy of the spreadsheet in Dropbox to ensure that it’s backed up, and it’s always clear what the latest version is.

For help in building surveys using XLS2XForm, please see the documentation. In addition to the question types listed there, we have used the following shortcuts, which also work in this customised version of XLS2XForm:

• text is short for add text prompt (a text field, such as a person’s name)
• note is short for add note prompt (a read-only field, providing additional information for the user)
• time is a time field without a date (for example, survey start and end times)

To compile the spreadsheet into an XForms form, run the build_and_validate.py script by double-clicking on it. If it works, it will show the message “Success!”, otherwise it will show an error message, usually caused by a mistake in the Excel spreadsheet. If it works, it will create (replace) the file called zambia-ranq-round3.xml in the same directory. If your spreadsheet has a different name, you can create a shortcut to call build_and_validate_custom.py with the name of the spreadsheet on the command line.

Software components

ODK Aggregate is the software that powers the Internet server. It is a repository for blank forms (designs) and completed forms (data). Our server is located at http://partimob.appspot.com/. This server is currently paid for by us, and will need to transfer to RuralNet at some point.

ODK Collect is the application runs on the device, and users interact with it to complete the survey. It’s essentially a user interface for XForms. It can download blank forms (designs) from an ODK Aggregate server, and upload completed forms (data) to the Aggregate server as well.

ODK Briefcase is the software that downloads completed forms (data) from the Aggregate server and convert them into CSV (spreadsheet) format, which can be loaded into

Customised ODK Collect

We are using a custom version of ODK Collect. You can download the source code for it here, or the compiled application here. You can also find it in the ZIP file download. If you prefer, you can use the latest official version of ODK Collect. The two are compatible, but our version adds the following useful features:

• Use supplied login and password by default to save a round trip and a prompt.
• Add keyboard navigation, useful for form filling on android-x86 because the mouse interface is pretty clunky.
• Restore ability to modify completed and submitted forms on the device, which was removed from the official version in 1.1.7.
• Improved error messages and progress indication during form uploads.
• Allow setting the instance name on the first page of the survey.
• Allow saving incomplete surveys on required questions (in case a survey is interrupted; almost all of our questions are required).

There are several ways to install ODK Collect on a device:

• Download it from the Android Market (official version only, not our customised version)
• Copy the APK file onto a microSD card, insert the card into the device, and use the My Files application find and open it from the SD card.
• Attach the USB cable from the device to a computer, enable mass storage mode on the device, and on the computer, drag and drop the APK file onto the device’s internal memory, then use the My Files application to find and open it.
• Attach the USB cable from the device to a computer, and use ADB‘s install command to install the APK file.

It’s useful to put the application onto the device’s desktop. To do that, open the Applications list, find ODK Collect, and press and hold it with your finger for a few seconds. The background will change to the desktop; release your finger to drop the application there.

It’s also useful to remove all the other junk from the desktop. For each icon and widget on the desktop, press and hold it with your finger for a few seconds, until the trashcan icon appears, then drag your finger to the trashcan and release it there.

Form management on the device

There are several ways to put blank forms (designs) onto the tablets:

• Download them from the ODK Aggregate server using ODK Collect.
• Copy them onto a microSD card, insert the card into the device, and use the My Files application to copy them from the SD card to the /sdcard/odk/forms directory.
• Attach the USB cable from the device to a computer, enable mass storage mode on the device, and on the computer, drag and drop the form into the /sdcard/odk/forms directory.
• Attach the USB cable from the device to a computer, and use ADB or DDMS to push the file onto the device, into the /sdcard/odk/forms directory.

Of these methods, ADB or DDMS is recommended for rapid development, and using the Aggregate server is recommended for production use, since the form must be installed on the Aggregate server for it to be able to accept submissions.

Similarly there are several ways to copy completed forms (data) off the device:

• Upload them to the ODK Aggregate server using ODK Collect.
• Use the My Files application to copy them from /sdcard/odk/instances to a microSD card, then remove the card and connect it to the computer, and drop the files into the ODK Briefcase data directory.
• Attach the USB cable from the device to a computer, enable mass storage mode on the device, and on the computer, drag and drop the files from the /sdcard/odk/instances directory to the ODK Briefcase data directory.
• Attach the USB cable from the device to a computer, and use ADB or DDMS to pull the file from the device’s /sdcard/odk/instances directory to the ODK Briefcase data directory.

Of these methods, using ODK Aggregate is recommended for development and production use.

Since the Aggregate server is on the Internet, this method requires that the device have Internet access. So it either needs a valid SIM card installed with credit and a data bundle, or a WiFi network connected. We had many problems with using SIM cards for data, so WiFi is preferred if possible.

The directories mentioned above will not exist until ODK Collect is installed on the device and run for the first time. Forms downloaded from the Aggregate server will also be placed in the /sdcard/odk/forms directory. Forms completed on the device will be placed in the /sdcard/odk/instances directory.

Configuring ODK Collect

Collect needs to know the details of the ODK Aggregate server to log into it, download blank forms and upload completed forms.

Open the ODK Collect application, press the Settings button and click on Change Settings. Click on URL and enter https://partimob.appspot.com. Similarly, complete the Username and Password using the details that you’ve been given by the Aggregate server operator, or the account that you’ve created on the Aggregate server. This account should only have Data Collector permissions, no more. Press the Back key to get back to the main menu of ODK Collect.

Downloading forms using ODK Collect

Open ODK Collect on the device, and click on the Get Blank Form button. Collect will try to log into the Aggregate server using the details that you’ve provided, and get a list of forms on the server that have the Downloadable box ticked. This is on by default for newly uploaded forms.

Tick the box next to all the forms that you want to download, and click on the Get Selected button.

Filling forms on the device

Open ODK Collect on the device, and click on the Fill Blank Form button. All the forms in the device’s /sdcard/odk/forms directory should be listed. Choose the form that you want to complete.

You will see an introductory screen showing how to move between questions by swiping your finger across the screen, from right to left or left to right. This screen has a text box at the bottom, which you can use to name the form that you’re completing. Naming forms is useful if your data collection is interrupted and you need to resume it later. It’s much easier to identify the form using its name, rather than opening it and flicking through to find some identifying information. You might name the form based on the household code that you’re surveying.

Depending on your answers to some questions, others may be hidden, or their text might change.

At the end of the form there is another chance to Name this form, and a tickbox to Mark form as finalized. Before you can upload the form to the Aggregate server, this box must be ticked, and you must press the Save Form and Exit button. Otherwise Collect will consider that the form is incomplete.

Sending completed forms to Aggregate

Open ODK Collect on the device, and click on the Send Finalized Form button on the main menu. Tick the box next to all the forms that you want to upload to Aggregate, and click on Send Selected. After the upload is complete, you should see the Upload Results message. Every form should have “Success” next to it, otherwise it was not sent successfully.

Downloading forms using Briefcase

We are using a customised version of ODK Briefcase with the following changes:

• Fix the export of repeated groups, which before only worked for the first row (issue 461).
• Shorten exported column names, to allow the CSV file to be imported into Access.
• Allow the server name, username and password to be provided on the command line (or via a shortcut).

You can find the source code here and the pre-compiled version here, as an executable JAR file. You can also find it in the ZIP file download. If you make changes to the source and want to build the executable JAR again, install Maven and use the mvn package command.

To download the completed forms, open Briefcase by double-clicking on the briefcase-1.0-jar-with-dependencies.jar file. On the Transfer tab, click on the Connect button. For the URL, enter https://partimob.appspot.com, and for the user name and password, give the details of an ODK Aggregate account with Data Viewer permissions.

Then you should see a list of forms appear under the heading Forms to Transfer. Tick the box next to the one that your users have been completing, and then click on the Transfer button. If you do this after all the completed forms (data) have been submitted to the ODK Aggregate server, you will not need to do it again for that form template (design).

Now switch to the Transform tab and see if the form appears in the Form list. If it doesn’t, then exit and restart the Briefcase application (issue 464).

For Output Type, choose .csv and media files. For Output Directory, choose the directory where you’d like to save the CSV files. Note that any previous files exported to that directory from the same form will be overwritten without warning, even if they have been modified (cleaned). Click on the Output button to write the CSV files.

Cleaning data in Excel

You can find the Excel spreadsheet that we use for data storage and cleaning here. Note that Excel is a long way from the best way to store and manipulate data like this. Microsoft Access would be far more appropriate. Yet again I wish there was a sufficiently powerful open source alternative.

Because the spreadsheet contains cleaned data, which is “better” than the raw data which is included in the CSV export, we don’t want to overwrite existing rows. For the main section of the questionnaire (the so-called Single Responses) you can include only the new data like this:

• Open the main spreadsheet and switch to the Single Responses tab
• Highlight all rows from 3 down to the bottom, and Sort them by the SubmissionDate column.
• Note the last submission date on this spreadsheet.
• Open the newly exported CSV file for the single responses (something like RANQ-2011-Round-4-v5.csv).
• Sort this file by the SubmissionDate column as well.
• Highlight and copy all the rows whose submission date is later (more recent) than the last one in the main spreadsheet.
• Paste them at the bottom of the Single Responses tab of the main spreadsheet, below the other data.

For the other tables, this process needs to be done completely manually at present.

You can then check and clean the data by viewing and modifying it in Excel. Note that each sheet has one or two columns at the end, which are filled by formulae that look up values from the Single Responses sheet, such as the Household Code.

Using the Android x86 Emulator

To be written.

Lessons learned

To be written.

I must admit to being skeptical about any such innovation, and I’ve been listening to both sides of the debate on the BytesForAll mailing list. Despite my skepticism, Osama’s questions have some answers, and I’d like to present them for comment.

• India has one of the lowest ratio of teachers—just 456 teachers per million people.
• Seventy-two percent of our primary schools have only three teachers or less.
• 25% of teachers were absent from school, and only about half were teaching, during unannounced visits to a nationally representative sample of government primary schools.

How is the $35 tablet going to solve any of these problems?

Of course technology on its own is not going to solve these problems. It is just a valuable weapon in the armoury of those who would launch an all-out war on poverty (and other abstract nouns).

Kentaro Toyama, an ex-Microsoft guru turned ICT4D researcher, says that “technology is [just] an amplifier of human intent and capacity.” And when faced with a task that’s possible but simply too large, an amplifier is exactly what we need. It doesn’t need to be high tech. Tanzania did just fine with radio, one of the oldest, simplest and most inclusive ICTs:

About ten years after independence, Tanzania decided to move towards universal primary education, almost doubling the number of children in school. The government estimated that it needed an extra 40,000 teachers. As the existing training colleges were producing only 5,000 new teachers a year, it was decided to recruit secondary school leavers and train them on an apprenticeship model, partly on the job and partly through distance education. Over a period of three years, they were posted in schools where they had a reduced teaching load. They then followed correspondence courses backed by radio programmes; they were supervised and tested on their classroom practices, and passed their examinations. Two evaluations found that they ended up reasonably competent in the classroom (Chale, 1993; quoted by Perranton, 2000; retrieved from UNU)

If India were to launch a massive teacher education programme, they would find it cheaper to implement that programme using technology. For example, they might distribute radios, TVs, portable audio players or even (heaven forbid!) computers to trainee teachers. It might take longer for those teachers to reach high standards, and more might drop out, without the personal connection and feedback of face-to-face training. Even so, one could train more teachers for more time and achieve a similar number of fully trained teachers at a lower cost.

In the business sector, more than 70% micro, small and medium enterprises (MSMEs) are not connected to information society to leverage opportunities of business and efficiency. How will the $35 tablet help in the financial inclusion of MSMEs, which are largely situated in small towns and remote areas?

It’s unfortunate that the tablet doesn’t include a long-range wireless network (such as GPRS), which must surely cover most of India as it does Africa. Even without an Internet connection, it can still provide useful services such as record keeping, business accounting and stock tracking to small enterprises. The tablet is based on Android, but the marketplace has been disabled, and this is a serious limitation. I think it’s likely to be overcome soon. When that happens, India’s many skilled software developers will be free to create localised applications for a potentially huge local market.

Most of India’s 3.3 million non-governmental organisations (NGOs) are also located in remote areas—70% of them lack any sort of information and communication technology (ICT) infrastructure or connectivity, and have no websites.

How can the $35 tablet help these NGOs’ global outreach efforts or aid the millions of people working with them in rural areas?

You probably know the answer to this question as well as I do: The same way as computer and phones can, only more so. Helping people to communicate and to do their work is exactly what ICTs do. All of them. With the possible exception of Angry Birds. A computer can help us to make leaflets, track visits to patients and beneficiaries, diagnose illnesses, improve farming techniques, or learn about anything we wish to know in the whole world of knowledge.

Can it bring transparency in governance at this level?

Good question. Not by itself, sure. Transparency comes from open data. The people might get together to publish what the government would rather hide, or pressure the government to release the data, but a $35 tablet won’t help them much.

When they do release that data, however, the usual problem is how to make use of it. Government data tends to be massive and unwieldy, and answering difficult questions takes much time and significant skill even with the best of data. I think that free, open, widecast media provide the biggest opportunity to make real use of transparency, and our use of the Internet as an enabler of democracy is the best example of that. Potentially, a simple but powerful Internet device could help bring people together to investigate and answer those difficult questions. But by the sound of it, this tablet is not quite there yet. Hopefully it will be soon.

Since a large population of our country communicate verbally, and cannot read and write with ease, their preferred medium of content consumption and content production is audio-visual… But to make use of good multimedia content, you need powerful machines, not cheap and underperforming ones.

I disagree with that. I grew up with “multimedia content” on BBC Micros: simple games, moving blocks around a screen, simple word processors and spreadsheets and databases and graphics. A picture is worth a thousand words, and a simple, clear diagram can be worth far more than a complex, confusing one. Advanced graphics are no substitute for a visual designer’s ingenuity and skill. Wikipedia is “multimedia content” that is perfectly suited to a $35 tablet.

If the $35 tablet can do anything good to education in India, the only way is by handing them to each and every teacher and school management staff to monitor the workings and functioning of the school and its teachers…

Monitoring is an interesting application, and a double-edged sword. Robert Chambers, the inventor of participatory rural appraisal, told us a story at the recent ICT4D Finale event in Cambridge of a hospital in India where the nurses were given mobile phones “to collect data at the source.” But the director of the hospital used it to monitor what they were doing, effectively spying on them. The nurses went on strike and eventually the director was fired. I think that for monitoring to have a positive benefit, it must be done with consent and a shared vision to use the data to improve performance, not to criticise and control.

rather than assuming that each student will buy Aakash and India will become digitally literate overnight.

I have to agree with that sentiment, although I’m not sure who raised it. Kapil Sibal, who takes the credit for inventing the $35 tablet, merely said:

This low cost device is part of our national mission on education through information and communication technology (NME-ICT) which will connect over 1,000 institutions across the country, enabling tonnes of web-based course content for free.

Now that doesn’t sound so far-fetched, does it?

Dummynet can be used to create arbitrary delays and packet loss, which is very useful for simulating poor connections, but not for sharing bandwidth and prioritising packets between different traffic classes on a real traffic shaper.

However, I’ve just been testing PF (the new standard packet filter) and ALTQ (the alternative queueing system) on FreeBSD, and I’m impressed by the capabilities. I prefer this combination (PF+ALTQ) over Linux TC because:

• PF and ALTQ are fully integrated and configured using the same file, whereas TC has its own (very hard to use) classifier. I normally use the iptables CLASSIFY target to classify traffic instead, but this is not integrated.
• TC is very hard to use generally. The authors seem more concerned with functionality than usability.
• ALTQ has named queues which helps usability enormously compared to TC’s hex numbered classes.
• ALTQ gives very low delay when the interface is not 100% saturated, which seems impossible to achieve with TC.

It does annoy me that ALTQ is not enabled in the default kernel, so you have to compile your own kernel. I used the following commands to copy the default GENERIC configuration to a custom one, which I called ALTQ:

cd /boot
cp -p kernel GENERIC # backup the current kernel
cd /usr/src/sys/i386/conf
cp GENERIC ~/ALTQ
ln -s ~/ALTQ .
vi ALTQ

and added the following lines to the new kernel configuration file, ALTQ:

options ALTQ
options ALTQ_RED
options ALTQ_RIO
options ALTQ_HFSC
options ALTQ_PRIQ

and then compiled and installed the new kernel:

cd /usr/src
make buildkernel KERNCONF=ALTQ
make installkernel KERNCONF=ALTQ

and then reboot to load the new kernel. After that, we need to create a pf configuration. Some example configurations use CBQ queues, but I prefer HFSC because:

• HFSC is guaranteed accurate, whereas CBQ is approximate
• CBQ requires you to guess the average packet size and its accuracy depends entirely on this
• HFSC has service curves which allow you to deliver small files quickly and drop the priority of large connections (e.g. file downloads) with great ease.

Here is a sample configuration of PF+ALTQ+HFSC that I used for testing on a transparent bridging firewall (bridge0 connecting em0 and em1):

altq on em1 hfsc bandwidth 1Mb queue { ftp, ssh, icmp, other }
queue ftp bandwidth 30% priority 0 hfsc (upperlimit 99%)
queue ssh bandwidth 30% priority 2 hfsc (upperlimit 99%)
queue icmp bandwidth 10% priority 2 hfsc (upperlimit 99%)
queue other bandwidth 30% priority 1 hfsc (default upperlimit 99%)
pass out quick on bridge0 inet proto tcp from any port 21 to any queue ftp
pass out quick on bridge0 inet proto tcp from any port 22 to any queue ssh
pass out quick on bridge0 inet proto icmp from any to any queue icmp
pass out quick on bridge0 all

We are only queueing on em1 here, which is the downstream, so we are only limiting downloads. We deliberately limit them to 1 Mbps for testing. The limit should always be lower than your actual download bandwidth, to ensure that the queue is on the FreeBSD firewall and not any other device.

We create four named queues under the root, which is implicitly named root_em1. We reserve 30% of bandwidth each for FTP, SSH and other traffic, and 10% for ICMP. However, any class can exceed its reserved bandwidth, up to the upperlimit, which defaults to 100%, which means that one class can potentially cause delays to traffic in other classes, so we override this to 99%.

Note that even though we create the queues on the em1 device, we must filter packets on bridge0, as otherwise our traffic does not match our pf rules.

Update: I found some more information about traffic shaping and advanced usage of HFSC, including realtime guaranteed classes for VoIP.

Update 2: For a simpler setup with ALTQ, try this guide.

Alan Barrett laying cable

I’m in Dar es Salaam, Tanzania for AfNOG 2011. I arrived on Wednesday morning at 7am (on the red-eye flight from London Heathrow) and I’m here until Tuesday 7th June.

Until now we’ve been setting up the venue. We’ve been super busy, working until midnight every night so far. We had to run our own cables, quite a lot of them (over 600 metres).

Running them through the windows was tricky, since we needed to be able to close them for security, and to allow the air conditioning to work. Someone (Alan?) came up with the genius idea of using tough palm leaves wrapped around them to protect them as they pass through the narrow gap between window panes. Bio-degradable trunking!

To cope with the power failures, Geert Jan built a monster Power-over-Ethernet injector to power the wireless access points in each room and keep the wireless network running.

The training workshops start tomorrow, Sunday 29th May, with the Unix Boot Camp, an introduction to Unix and the command line. We expect that many of the participants will have little experience with Unix, as has been the case in previous years. These free tools have immense benefits, both for us running the workshops and for the participants when they return home. But they are very different to the Windows environments that the participants are most familiar with. Without basic skills, they would struggle and hold back the group during the rest of the workshops.

Feeding the cable monster

I’m not involved in the boot camp, but after it finishes, we move straight into the main tracks, which last for five days. This year we have some new tracks: Network Monitoring & Management, Advanced Routing Techniques and Computer Emergency Response Team training.

We have also cancelled the basic Unix System Administration track (SA-E) this year, as it has finally been localised to most African countries, and therefore people have the opportunity to attend it locally at lower cost and build local communities. However, this leaves us with nowhere to cover more advanced systems administration techniques, which are some of my favourite topics, including:

Geert Jan and the Monster Injector

• virtualisation (desktops, servers and thin clients, VirtualBox, Xen, KVM, jails, lxc)
• system imaging (ghost, snapshots)
• backups (snapshots, Rsync, Rdiff-backup, Duplicity)
• file servers (NFS, Samba, sshfs, AFS, ZFS)
• virtualised storage (iSCSI, ATAoE, Fibre Channel, DRBD)
• cloud computing (Amazon and Linode virtual servers, scripting and APIs)
• cluster computing (Mosix, virtual machine host clusters)
• DHCP (for network management and booting)
• network security (firewalls, port locking, 802.1x)
• wireless networks (planning, monitoring, troubleshooting, WEP and WPA, 802.1x authentication)
• Windows domains and security (including Samba 4)

If participants show enough interest in these topics, they could be added in future. I think it’s unfortunate that the course is arranged into week-long tracks rather than half-day or one-day sessions from which people could pick and choose, Bar Camp style. That would make it much easier for people to run sessions on many new topics.

Some of our 80 desktop computers

In the past this would have been difficult, because we provided desktop computers for participants. It used to take us 3-4 days to set up 80-odd desktop PCs with customised FreeBSD installations. We’ve noticed that more and more people are coming to the workshops with their laptops, and this time we’ve made a big effort to shift from dedicated to virtual platforms, to reduce setup time and costs in future.

The hardest track to do this for, in my opinion, was Scalable Services English (SS-E), the one I’m working on. We were all pretty much agreed to stay with desktop PCs this year, making us the only track to do so. But when we arrived, we discovered that the mains power situation here is pretty awful. On Wednesday we had four power failures. We only have five UPS, not nearly enough to protect every desktop.

For participants with laptops, they effectively have their own built-in UPS. If we give them virtual machines to work with, then we only have to protect the hosts. We can keep those in the NOC (Network Operations Centre), where the UPS are, so they’ll be protected for around 15 minutes of any power outage, which we have to hope will be enough for the hotel to start their generator.

Cannibalising RAM

Some participants will probably forget their laptops, so we’ll provide them with desktops, but we have no way to UPS them. These desktops will be set up with FreeBSD, as in previous years.

We rented 80 machines from a local company. Some had Windows, in varying states of repair, some had no operating system installed. We decided to use some of these desktops as hosts for the participants’ virtual machines.They only had 2 GB of RAM each, but since we had plenty, we cannibalised eight others for their RAM to upgrade our machines to 4 GB each.

We decided to use VirtualBox for the virtual machines. It’s free, open source, can host on all major platforms (Windows, Mac, Linux and even FreeBSD), has a nice GUI and a command-line automation tool, supports bridged networking easily, and is relatively fast and efficient.

Imaging backend

We configured the master (that we’ll copy onto the other machines) starting with the setup from last year. We then had to install VirtualBox and build our first virtual machine inside it. Then we discovered that the virtual machine was unable to access the network in bridged mode. Packets sent by the virtual machine we simply never sent by the host. We needed to use bridged mode so that participants could run services on their machines simply by installing them. without requiring extra configuration on the host.

We had no Internet access for most of that day, because all three of our redundant providers were down for different reasons. Eventually we managed to use Geert Jan’s 3G dongle to get online and research the problem. We found that bridged networking doesn’t work in the binary package of VirtualBox 3.2.12 that comes with FreeBSD 8.2, so we had to wait until Internet access was fixed to download 120 MB of software (ports updates and VirtualBox 4.0.8) like this:

Imaging frontend

pkg_add -r portupgrade
portsnap fetch extract update
portupgrade virtualbox-ose virtualbox-ose-kmod

This took a long time, as VirtualBox is a large piece of software which also required us to download and build a new version of QT, but eventually it succeeded and the problem was solved.

We decided to put only five virtual machines on each host. Sometimes we would have the whole class compiling software from ports, which would slow down all of them significantly. We will use six or seven servers to host 30-35 virtual machines. On the master host, we created five copies of our master virtual machine by copying its hard disk like this:

cd .VirtualBox/HardDisks
for i in 1 2 3 4 5; do
	cp AfNOG\ SSE\ Master.vdi vm0$i.vdi
	VBoxManage internalcommands sethduuid vm0$i.vdi
done

Relocation

Then we created the virtual machines in the VirtualBox GUI and attached them to these new images. We needed to generate a new UUID for each disk image copy, using the undocumented sethduuid command above, otherwise VirtualBox would refuse to add the copies because it had a disk image already registered with the same UUID.

We could have created the virtual machines using the VBoxManage command as well, but it would have taken longer to work out how to use it than simply to create the five machines by hand. I later worked out the commands that we could have used:

cd ~/"VirtualBox VMs"
for i in {1..5}; do
	echo $i
	vmname=VM0$i
	diskimage="$vmname/FreeBSD.vdi"
	VBoxManage createvm --name "$vmname" --ostype FreeBSD
	VBoxManage modifyvm "$vmname" --memory 256 \
		--nic1 bridged --bridgeadapter1 bge0.219 \
		--nic2 bridged --bridgeadapter2 bge0.$[50+$i] \
		--vram 4 --pae off --audio none --usb on \
		--uart1 0x3f8 4 --uartmode1 server /home/chris/"$vmname"-console.pipe
	VBoxManage storagectl "$vmname" --name "IDE Controller" --add ide
	cp VM01/FreeBSD.vdi "$diskimage"
	VBoxManage internalcommands sethduuid "$diskimage"
	VBoxManage storageattach "$vmname" --storagectl "IDE Controller" \
		--port 0 --device 0 --type hdd --medium "$diskimage"
done

We named the images VM01 to VM05, which was important for running automated scripts on them. Then we configured VirtualBox to start them automatically at boot time, in headless mode, by adding the following lines to /etc/rc.conf:

vboxheadless_enable="YES"
vboxheadless_machines="VM01 VM02 VM03 VM04 VM05"
vboxheadless_user="inst"

We wrote a short script to help us apply the same command to all five virtual machines:

#!/bin/sh
# script to control all five virtual machines

command=$1
shift

for i in 1 2 3 4 5; do
	VBoxManage $command VM0$i "$@"
done

This allows us to log into a machine and do things like:

• ./manage acpipowerbutton to initiate a controlled shutdown of all five virtual machines
• ./manage modifyvm --macaddress1 auto to generate new, random MAC addresses after cloning the host
• ./manage startvm --type headless to get the virtual machines running again (headlessly, independent of the GUI)

The NOC

We knew that we wouldn’t have space to attach monitors and keyboards to all the hosts, and we’d have to fiddle about with cables in the hot NOC room (without working aircon) if we needed access to their consoles, so we added the ability to log into them remotely using VNC and GDM. To do this, we had to install the VNC server:

pkg_add -r vnc

Which unfortunately doesn’t come with the nice xorg loadable module that adds a built-in VNC server to the X server, making a fast and stateless remote control session possible. So we had to hack about with inetd, first by adding a service name with a port number to /etc/services:

vnc		5900/tcp

And then a service line in /etc/inetd.conf:

vnc	stream	tcp	nowait		root	/usr/local/bin/Xvnc Xvnc -inetd :1 -query localhost -geometry 1024x768 -depth 24 -once -fp /usr/local/lib/X11/fonts/misc/ -securitytypes=none

This requires us to enable the XDMCP protocol in GDM, in order for VNC to communicate with it to present a GDM login screen. So we replaced the contents of /usr/local/etc/gdm/custom.conf with the following:

[xdmcp]
Enable=true

[security]
DisallowTCP=false

And then restarted GDM:

sudo /usr/local/etc/rc.d/gdm restart

And checked that we could connect from another machine and got a login prompt:

vncviewer 196.200.217.128

Which did indeed give us a working login screen:

VNC graphical login on a FreeBSD virtual machine host

This method is very slow. I wanted to find a better way to access the guests, especially if their network configuration was broken. I tried to connect a host serial port to a pipe and then access that pipe from a shell command, eventually over ssh, in a similar way to the text-only console offered by Xen (xm console). The above VBoxManage commands set up a pipe in my home directory, and I wrote the following short script to access it:

#!/bin/sh
set -x
echo "Console for $USER"
nc -U /home/chris/$USER-console.pipe

I created user accounts for each virtual machine, with the same name, and set their shells to this script, so that when they log in, they would automatically be connected to the pipe. However I was unable to make it work well. In particular, because of incompatible terminal emulations, I was unable to run vi to edit configuration files in the guest. If you find a way around this, please let me know. I haven’t tried it yet, but conman looks like it might be a good bet.

I spent a long time searching for the hidden VNC support in VirtualBox 4. It’s undocumented (the manual only talks about RDP) and people on the IRC channel say that it doesn’t exist, but it does, at least when starting the guests in headless mode. I added the following lines to /etc/rc.conf:

vboxheadless_VM01_flags="-n -m 5901"
vboxheadless_VM02_flags="-n -m 5902"
vboxheadless_VM03_flags="-n -m 5903"
vboxheadless_VM04_flags="-n -m 5904"
vboxheadless_VM05_flags="-n -m 5905"

And then, after starting the guests in headless mode, I could connect to these ports and access the virtual displays, much more conveniently and much faster than by shutting down the guests using VBoxManage and starting them again using the VirtualBox GUI.

We used multicast to image the six virtual machine hosts from the master. This took about three hours, so we left it running overnight.

In the morning we checked that the hosts had been imaged successfully by booting them with their newly installed images, and gave them unique hostnames (host1.sse.ws.afnog.org etc.) and IP addresses.

We used the manage script to reset the MAC addresses of the network cards of each virtual machine on each host:

for i in 128 129 130 131 132 133 134; do ssh 196.200.217.$i ./manage acpipowerbutton; done
for i in 128 129 130 131 132 133 134; do ssh 196.200.217.$i ./manage modifyvm --macaddress1 auto; done
for i in 128 129 130 131 132 133 134; do ssh 196.200.217.$i ./manage startvm; done

Astral projection

Since they were all configured for DHCP, we could have got their IP addresses from the DHCP server, but we wanted to give them a nice naming scheme, so we logged in to each one (using the console and the VirtualBox GUI) and assigned it a unique hostname and a static IP address.

We checked that we could log into each virtual machine remotely using the SSH keys that we’d installed, and then we shut down the hosts and moved them to the NOC.

Boot camp starts tomorrow, next door, but we still have to arrange our room.

Classroom

We may have up to 37 people, our biggest class ever, in a room that’s about eight metres on a side, so layout of the room is a real challenge.

I wanted to do something to facilitate working in groups, such as each table having four places (two each side) and with its long axis front-to-back. This was vetoed because participants would have to turn their heads to see the projected screen, and it might be hard for them to take notes as a result.

So we’re going to have long, cramped benches instead. I think this is unfortunate, and I hope I can persuade people to try something more imaginative in future.

Jeff continues to try and solve the low bandwidth/high latency problems that aid workers face in the field every day and that we encountered in Indonesia. We all know the joy of VSAT networks that slow to a crawl because either some folks on the team are downloading stuff they shouldn’t be downloading or all the computers are infected with bandwidth sucking viruses. It appears Jeff has moved one step closer to sorting out some of the problems surrounding bandwidth optimization by utilizing the Go programming language.

Rather than try and explain to you what Jeff has done I’ll let you read ‘A rate-limiting HTTP proxy in Go‘ and ‘How to control your HTTP transactions in Go‘ and sort out what he is talking about. Hopefully, this post will bait Jeff into leaving a lengthy comment that explains exactly what the hell he is up to.

My understanding is that Jeff is developing two useful tools:

• A web proxy that simulates low bandwidth connections, similar to the Loband Simulator. Jeff’s version is probably more accurate than ours because it doesn’t need to modify the web page, but ours might be easier for non-developers to try out, because you don’t have to install any software.
• A web proxy that can be fed prepackaged content repositories to serve up, so that you can take pre-prepared content (offline websites) with you into the field, and browse them through the proxy as though you were online.

People have been trying to make offlineable websites for a long time. Some of the best examples so far are using entirely client-side (in-browser) technology, such as the Logistics Operational Guide, developed by the World Food Programme for the Logistics Cluster, which can run entirely offline using Google Gears.

Gears had a lot of potential for developers to create offlineable websites, but Google has abandoned its future development in favour of the open standard HTML5, which is not ready yet. So there’s no obvious and future-proof way to develop offlineable websites at the moment. Jeff’s proxy, combined with a spidering system, could be one way to download an entire site, even if it wasn’t designed to be downloaded by the developers.

Another important potential comes from content management systems (CMS) such as WordPress, Drupal and Joomla. More and more websites are developed using such systems, rather than coded from scratch. The systems know all of the pages on the site, and the links between them, and could easily build an offlineable version of the site for download into Gears, HTML5 or Jeff’s proxy. And one plugin could potentially enable thousands of sites to be offlineable, especially if it was included in the CMS distribution and enabled by default.

A few wikis such as MediaWiki, MoinMoin, DocuWiki and JSPWiki have a programming interface (XML-RPC or WebDAV) that allows a smart client to download pages in their original text format, which could make them more efficient to store offline and also potentially editable offline. Jeff’s proxy could be extended to support sites built in such wikis automatically. There are still some limitations to this approach:

• The pages would not look the same as the online versions, since the styling wouldn’t be downloaded and the effects of CMS plugins would not be visible;
• It would probably still be quite slow to download an entire site this way, by spidering, without server-side support for downloading multiple pages at once;
• Few websites are built out of Wikis, so the potential maximum reach is limited compared to better support for WordPress, Drupal or Joomla.

Anyway, I wish I knew Go, and had time to hack on Jeff’s proxy tools.

I’ve been wanting to write some thoughts about Digital Resilience for a while and reading Janet Gunter’s blog post about Mobile Blackouts and the Poor has prompted me into finally doing it.

I think we’re going to see the subject of Digital Resilience rising in our consciousness over the coming years… at least we should. What do I mean by it? We regularly allow ourselves to rely on information and communication technologies but these tools can be  fragile and have many modes of failure. Often this fragility is not taken into account. I see this all the time.

Here’s a typically example. I’m on the phone to a friend organising an evening out. How many times have I heard the phrase “I’ll call you when I get to town”? This relies on us both having signal, on our batteries not running out and on our phones not breaking, being lost or stolen. Modern “smart” phones eat through their batteries at a tremendous rate and sometimes crash or reboot so these failure scenarios are not that unusual. If any of the things that our phones rely upon fail what is our back-up plan? Are we going to wander around the city, aimlessly, hoping to bump in to each other? Or go home?

A failed night out is not the end of the world. The point I’m wanting to make is that most of the time people seem oblivious to the failure modes of the technology they use or the need for back-up plans. And yet when you’re in the habit of considering technological risks it becomes second nature, like finishing a sentence with a full-stop
.

So when I hear “I’ll call you when I get to town” I usually can’t help myself say “And if I don’t hear from you, I’ll meet you at the station…”

The situation gets a little more serious in a business context. Take the example of a conference call. Scheduling a call between several people can take a lot of planning trying to find a time when everyone is free. If this time-slot is missed it can represent a significant opportunity cost to the organisations involved. The duration of the call itself represents a cost when you consider the time of the people involved. There is growing use of Skype for conference calls in the professional community. Sometimes Skype works very well but in my experience it frequently doesn’t work at all. Many times I have been asked to participate in a Skype conference call with no back-up even when many of the participants are in developing countries. When the Skype call fails much time is wasted, sometimes several participants are excluded and sometimes the call is abandoned all together. My automatic response to a Skype invitation is usually to send round the number of a phone conference service just in case.

OK, failed conference calls are still not the end of the world. For someone poor or vulnerable a technology failure could be much worse. At the recent ICTD2010 conference I was very interested to hear for the first time a few people talking about vulnerabilities to the poor caused by reliance on ICTs. For years the ICT4D community has been promoting the use of ICTs in poverty reduction. There has been a fair amount of work in adapting technologies for harsh environments. However these technologies still have failure modes and it was very encouraging to hear the resulting vulnerabilities being discussed.

Digital resilience is less about building robust technologies and more about building an understanding of its failure… and the habit of back-up plans.

All images are embedded SVG in the XHMTL. No bitmaps at all. Notice how fluid it is, how it scales with the browser’s zoom in and zoom out controls (Control + and Control – in Firefox) and as you resize the browser window.

Screenshot of Sam Ruby's Blog

The page is small, just 14.5k of HTML plus 6.6k of CSS. There’s 21k of JavaScript that isn’t required for the design. Even the drop-down menu at the top works with Javascript disabled. Finally there a WOFF web font that adds 40k (another nice technique). It would be nice to have web fonts hosted for cross-site caching.

One disadvantage of designing sites this way is that the page must be valid XHTML for inline SVG to work. This makes it difficult to support older browsers properly, because the server must send the content-type as text/xhtml+xml, not text/html. This will cause older browsers to download the page instead of rendering it. You could work around that with user agent sniffing. I think that Internet Explorer might need that in any case.

Another disadvantage is that very few CMSs currently support generating valid XHTML, so it’s difficult to know what tool we could recommend to help you to build and manage a website with inline SVG. Massimiliano of the Habari Project says:

I don’t have any examples of blog software constructed this way… the only way to find out how many people would like having SVG on their blog is to provide a blogging tool which allows them to do it.

Both issues could be worked around by using external SVG (in separate files) instead of inline (embedded in XHTML). External SVG files are more cacheable but require additional HTTP requests to fetch from the server the first time.

Most older browsers do not support SVG images, so although the site degrades gracefully, it looks very plain without any graphics. You could work around this with a server-side renderer that converts the SVG to PNG for older browsers.

I think this is an excellent example of a great technique that we could be using for many more sites.

This is something that ought to be really simple. It’s probably the most common use case of VPNs, Windows has a built-in VPN client, and Cisco routers can be used as VPN servers. EWB want it to be simple, because they have non-technical remote workers. It turned out to be much harder and take much longer than I expected.

Information Overload

One of the biggest problems was the lack of useful information, and the profusion of useless. The information fell mainly into four categories:

• Cisco marketing materials touting the benefits of VPNs and their expensive Concentrator and WebVPN products;
• Cisco knowledge base articles describing the setup of complex VPN scenarios;
• Cisco command references with little or no details on what each command actually does, or how to use them together;
• Cisco exam study sites with inaccurate, out-of-date or cookie-cutter command sequences, with even less explanation of what the commands actually do.

Because I couldn’t find what I was looking for, and had to work it out the hard way, I’ve written it up in the hope that it will help others.

I would recommend any organisations that simply want to share files to seriously consider a file-sharing service like DropBox or raw Amazon S3 instead of a local file server and VPN. In many cases the low upload bandwidth of ADSL connections, combined with internal office use of the connection. will make a VPN impractically slow, especially compared to Amazon’s unlimited upload and download bandwidth. But EWB already had the file server and they just wanted to access it remotely, not to change how they work.

Our scenario is simple: an internal office network with private IP addresses, a Cisco 1800 router providing ADSL connectivity for the office, and remote field workers running Windows desktops.

Getting the Client

For simplicity, we and EWB had hoped to use the built-in VPN client on Windows, which would remove the need to download and install software on the remote workers’ machines. But unfortunately the Cisco 1800 does not support this. Windows uses L2TP over IPSEC for modern, secure VPNs, as a replacement for the old insecure PPTP protocol. But Cisco has crippled the L2TP support in this router, and it only supports raw IPSEC. Only their more expensive routers support serving L2TP over IPSEC, allowing simple direct connections from Windows.

Raw IPSEC is the only remaining option on this router, but it’s difficult to configure due to its complexity, and the number of choices that need to be made. The standard requires both sides to have the same settings configured, but provides no way to do this automatically. Manual configuration would make life very hard for the remote workers. To solve this problem, Cisco has a non-standard protocol for auto-configuration of the clients:

Establishing a VPN connection between two routers can be complicated, and it typically requires tedious coordination between network administrators to configure the two routers’ VPN parameters.

The Cisco Easy VPN Client feature eliminates much of this tedious work by implementing Cisco’s Unity Client protocol, which allows most VPN parameters to be defined at [the] IPSec server.

Cisco Easy VPN Client for the Cisco 1700 Series Routers

So we needed to find a replacement client that was easy to use and could talk to the Cisco. Preferably a free one.

Then we discovered that although Cisco’s own VPN client is technically free, you can’t actually download it without a support contract, which neither we nor EWB have.

In the end we found that if you go to Cisco’s VPN client software page, find the filename of the latest version of the client, and Google it, you’ll find that several people have had enough of this nonsense and posted the client online, so it can be downloaded.

Of course it’s important to be aware of the potential for viruses in copies that you download from random sites on the Internet, as well as fake download sites that lead you around in circles of free registrations, credit card details and pop-up porn adverts. This site worked fine for me, but it may have been taken down by Cisco’s attack dogs by the time you read this.

Security with Obscurity

We decided to choose a configuration that trades some security for ease of use. So instead of authenticating with certificates, we used pre-shared keys. The VPN server has its own login system anyway, which provides an additional layer of security once the remote user is connected to the VPN.

Names and Addresses

Connecting clients need to be allocated an IP address to use over the VPN. We could have used public IPs, or private IPs in the same subnet (with proxy ARP), but we chose to use private IPs in a different subnet. This makes the routing easier, as clients and local network servers will know that they have to route the traffic via the router anyway, and it allows EWB to implement stricter network access policies for VPN clients, if they wish.

We needed to create a local pool (not a DHCP pool) to draw these addresses from:

ip local pool vpnpool 192.168.2.100 192.168.2.200

Keys to the Kingdom

We created an ISAKMP (IKE) policy to specify the authentication method and the level of encryption to be used for negotiation of IPSEC Security Associations (SAs). We chose to make this the first, highest priority policy, and to use AES-256 encryption (strong and fast), Group 2 (1024-bit) Diffie-Hellman key exchange, and pre-shared keys for client authentication as noted above:

crypto isakmp policy 1
 encr aes 256
 authentication pre-share
 group 2

Then we specified the pre-shared key itself. This is the only thing that stops random clients on the Internet from connecting to your local network, so it’s even more important than a strong wireless network key. Of course this is not the real key:

crypto isakmp key ThisKeyMustBeKeptSecret address 0.0.0.0 0.0.0.0

We specify that any IP address can use it by using the wildcard address, 0.0.0.0 0.0.0.0.

At the End of the Tunnel

It seems to be common in corporate environments that, when a user is connected to a VPN, all of their Internet traffic is routed through the VPN. It certainly makes it easier for the network administrators, as they don’t have to define specific routes for the tunnel, but it wastes their bandwidth and makes Internet access much slower for the remote workers, so we decided not to do this.

Just routing a single subnet through a tunnel is called a split tunnel. I couldn’t find simple documentation on setting it up, so I used the Cisco Easy VPN Remote example, extracting just the bits we needed to route only the 192.168.1.0/24 subnet through the tunnel.

First we have to create an access control list (ACL) that defines, on the local (source address) side, what traffic clients should route into the tunnel:

ip access-list extended ewb_office_split_tunnel
 remark Defines which local (office) networks a remote VPN client will route to
 permit ip 192.168.1.0 0.0.0.255 192.168.2.0 0.0.0.255

I’m not sure if the second half of the ACL is actually necessary. It doesn’t appear to make any difference if I specify any instead of 192.168.2.0 0.0.0.255.

Client Configuration

We use Cisco’s EzVPN (Unity) protocol, as described earlier, to configure connecting clients automatically. To do this, we have to tell the server what configuration should be sent to clients when they connect:

crypto isakmp client configuration group EWB
 key ThisKeyMustBeKeptSecret
 dns 192.168.1.1
 wins 192.168.1.2
 pool vpnpool
 acl ewb_office_split_tunnel
 netmask 255.255.255.0

A little explanation about what these options do:

crypto isakmp client configuration group [name]

The name must match the group name that the client uses when it connects. This is how the server decides which configuration to send to the client.

key

For some reason the client needs to be told what key to use, even though it’s already been entered by the user, and the client knows it because it wouldn’t be able to get this far in the negotiation without it!

dns

Tells the client which DNS server to use, for resolving local (private) hostnames, or resolving inside the split horizon. You can specify a second DNS server after the primary one. You probably only need this if you’re running a Windows domain, in which case it should point to the domain controller, or if you have split horizon DNS.

wins

Tells the client which WINS server to use, for resolving local SMB server names. Again, you probably only need this if you’re running a Windows domain, in which case it should also point to the domain controller.

pool

Tells the server which local pool (not DHCP pool) to assign the client’s address from. You can specify any name here, even a pool that doesn’t exist, but clients won’t be able to connect unless the pool name is a valid local pool.

acl

This ACL, which we defined earlier, is used to tell the clients which subnets are reachable through the connection (split tunnel mode). If no acl statement is used, the tunnel is not split, and a default route is set through the VPN tunnel instead.

netmask

Defines the network mask that the client will apply to its client interface, in combination with the IP address assigned from the pool.

Profiling

Next, we create an ISAKMP profile on the server which tells the server to assign IP addresses automatically, and which virtual template to use when creating the virtual-access interfaces for the server side of the tunnel. We haven’t defined the virtual template yet, but we will in a second.

crypto isakmp profile ewb_isakmp_profile
   match identity group EWB
   isakmp authorization list sdm_vpn_group_ml_4
   client configuration address respond
   virtual-template 1

When a client connects using the group name EWB, it will check for network authorization using the AAA list name sdm_vpn_group_ml_4 (or default if that list doesn’t exist), respond to IP address requests from the client (using the pool defined in the client configuration above), and create a local virtual-access interface based on virtual template number 1.

You should use the same group name that you used for the client configuration above, instead of EWB, unless you’re EWB of course.

Strong Encryption

Now we define the level of encryption used for data communications with hosts on the internal network, as opposed to securing the negotiation process. We start by defining a transform set which uses 256-bit AES encryption, the SHA hash algorithm and LZS compression for data packets:

crypto ipsec transform-set ewb_encryption esp-aes 256 esp-sha-hmac comp-lzs

Then we create an IPsec profile that links these settings to the ISAKMP profile that we defined above:

crypto ipsec profile ewb_ipsec_profile
 set transform-set ewb_encryption
 set isakmp-profile ewb_isakmp_profile

Virtual Template

Now we define the template for the virtual interfaces, that we referenced above in the ISAKMP policy:

interface Virtual-Template1 type tunnel
 ip unnumbered Vlan1
 zone-member security in-zone
 tunnel mode ipsec ipv4
 tunnel protection ipsec profile ewb_ipsec_profile

We use ip unnumbered Vlan1 to set the IP address of the virtual-access interfaces to the address of the router on the local LAN (in this case it’s a VLAN bridge), which allows you to ping the router using its internal IP address (192.168.1.1 in our case) when you’re connected to the VPN, which is a useful connectivity test.

We place the virtual interfaces into the in-zone (internal zone) which means that they have full access to the local network, which is not very secure, but simplifies things. We also specify that this interface accepts only traffic encrypted with IPsec and bound to the profile that we created earlier. I’m not sure why it needs to be bound in both directions, as the IPsec profile is connected to the ISAKMP profile which is connected to this virtual interface already.

Client Setup

That should be it for the server-side setup. To configure a client, install the VPN software you downloaded earlier, start it, create a new IPsec configuration, and enter the following details:

Server

The public IP address of the VPN server

Group Name

The same group name that you used on the server earlier

Pre-Shared Key

The same key that you entered on the server earlier

Now click on the Connect button, and after a few seconds the window should minimize to the system tray, and you should be connected to the VPN. You can check this by pinging the internal IP address of the router (e.g. 192.168.1.1) and if that works, the IP addresses of whatever internal servers you want to connect to.

If it doesn’t work, use the Log menu to enable logging, try to connect again, and check the results on the Logging tab. You can also try enabling IPsec debugging on the router, in run mode (not configuration mode):

debug crypto engine packet
debug crypto ipsec error
debug crypto isakmp error
debug crypto verbose
terminal monitor

When the configuration works, write it to the router’s non-volatile memory to ensure that you don’t lose it when you next reboot the router:

write

And that’s it!

References

Here are some random unsorted links to pages that I found useful while figuring out how to do this:

• Configuring a Cisco Router to Accept VPN Connections (even simpler example, without split tunnels)
• Configuring Internet Key Exchange for IPsec VPNs (good general overview of how Cisco’s IPsec works)
• Cisco Easy VPN Remote configuration guide
• Cisco VPN client downloads
• Cisco ARP Commands
• Cisco ISAKMP command reference
• Configuring Zone Policy Firewalls

The average size of web pages has been growing much faster than the average speed of connections in developing countries, and journal websites are no exception, as you can see in Alan’s blog post:

Average Page Size vs Bandwidth

As Alan points out, the average journal’s home page in his sample would take over 90 seconds to load on average, for researchers at universities in developing countries. Usability research has shown that people expect a computer to respond within 30 seconds. Making them wait longer interrupts their concentration, causes dissatisfaction and annoyance, and they often abandon the process. The biggest factor in user satisfaction is speed of response.

While this research probably did not include users who are accustomed to slow and unreliable computers, I think it’s safe to say that most people would find it annoying and difficult to use the Internet on a dial-up modem. And even a modem would have been preferable to some of the Internet connections that I’ve experienced (and paid for) in some countries in the last few years.

Academics have little ability to persuade their universities to upgrade their internet connections, at a cost of several peoples’ salaries (several thousand dollars a month). The only people who can change this are the publishers of the journals, by optimising their journals’ websites for users with slower connections.

But how to persuade the publishers that this is important? We built a low bandwidth simulator ourselves, and took it to Oxford, to INASP and the ACU‘s Publishers for Development conference.

What We Did

We set up spare machine as a bandwidth management box, and used it as a network filter for the participants. They could come and plug their laptops into the box, and browse the Internet and their own websites at a simulated slow speed.

Exercise Table

We configured the box for transparent bridging. This allowed us to insert and remove it from the network easily, just by switching over a network cable, to demonstrate the difference between fast and slow loading of pages.

We gave the participants at the meeting tasks to perform on various publishers’ websites, for example finding and downloading an academic paper by topic or researcher.

Playing the Game

I think they found the activities enlightening, because we had some very good comments from some of the participants:

• “We’re so pleased that Alan was able to work his magic at the recent PfD session – his delivery is innovative, dynamic and fact-packed so it really sparks enthusiasm from the audience… [which] is demonstrably channelled into action once people return to their places of work.“
  Publishers for Development Team
• “It was really useful to try the low bandwidth! [Our site] is already considered fast but it made us think even more around this issue, what else can we do etc.“
  Anonymous Participant
• “Alan Jackson’s information about bandwidth was kind of shocking even if I knew it before, but to really experience it was very valuable. We are going to redesign DOAJ’s home page and this must be the starting point.“
  Sonja Brage, DOAJ
• “Site speed is a major consideration for us, and I really enjoyed Alan/Aptivate’s session, experiencing the exasperation of trying (and failing) to connect via low-bandwidth… I have a feeling that there is ‘excess baggage’ on a number of the pages…“
  James Kitchen, OECD

How We Did It

We used FreeBSD as the operating system for the software bridge, because its dummynet traffic shaper is relatively easy to use, and very good at simulating slow connections.

We wanted to use a laptop instead of a desktop machine, so that we could carry it to the conference easily, but we had hardware compatibility issues with FreeBSD on all the laptops we had available to us (mostly IBM Thinkpads). We ended up using a compact Fujitsu desktop box.

We installed FreeBSD 8 on it, and configured it to transparently bridge between two interfaces. Our internet access at the conference would be wireless, but we had issues with bridging wired and wireless interfaces together. So instead we used a Linksys WRT-54GL router with the Tomato firmware, which enables wireless client mode, to connect to the network:

Throttler Network Diagram

And this is what it looked like in the room. Notice the essential coffee and cupcake, without which the system mysteriously failed to work:

Network Close Up

We configured the FreeBSD box to bring up the bridge automatically at boot time, and to load a set of ipfw firewall rules to enable dummynet, the traffic shaper. On this box, the ethernet interfaces are called em0 and rl0, so we added the following lines to /etc/rc.conf:

ifconfig_em0="up"
ifconfig_rl0="up"
cloned_interfaces="bridge0"
ifconfig_bridge0="addm em0 addm rl0 up dhcp"

firewall_enable="YES"
firewall_type="/etc/ipfw.rules"
dummynet_enable="YES"

Then we created /etc/ipfw.rules with the following contents:

# with bridge mode, two nics. em0 is wan
add pipe 1 all from any to any out recv em0
add pipe 2 all from any to any out xmit em0
add allow all from any to any
pipe 1 config delay 700ms bw 40Kbit/s mask dst-ip 0x000000ff
pipe 2 config delay 700ms bw 40Kbit/s mask src-ip 0x000000ff

This configuration creates two dummynet pipes. Pipe 1 is for traffic received on the external interface (downloads), and pipe 2 is for traffic being sent out of the external interface (uploads). We have to follow this by a rule which allows all other traffic, otherwise local traffic (on the box itself) is denied by default when the firewall is enabled, which breaks local DNS and inbound SSH and makes the box pretty unusable on the console.

Then we configure both pipes to allocate 40 Kbps (kilobits per second) for each individual IP address in the private subnet (allocated by the DHCP server on the Tomato router) and a 700 ms delay in each direction, which gives a 1400 ms round trip time. This is somewhat higher than the expected 600 ms round trip for a connection by geostationary satellite.

The end result is that each user connects a laptop to the switch behind the box, gets an IP address from the DHCP server on the router, is NATted by the router onto the public network, and is able to browse the Internet with a connection of 40 kbps upload and download. If you remove the FreeBSD box, by connecting the switch directly to the router, you can access the public network at full speed.

One issue was that the public network used a captive portal, which we had to log into. We didn’t want each client on our network to have to log in separately, so we enabled NAT on the router, and in wireless client mode, all the NATted clients get the MAC address of the router, so the public network thinks that they’re all the same PC and doesn’t ask them to log in again.

Why We Did it

We think that members of universities and research institutions need to be able to join and participate in the global research community as equals, in order to play their part in assisting development in their home countries.

Programmes such as PERii, HINARI and AGORA negotiate free or discounted online access to these journals for universities in developing countries. But the users still need to get online and access the content.

Online publishing for Western markets is usually designed for users with fast Internet connections, which Western universities have. But in other regions, universities often can’t afford fast connections, and this makes it very difficult for them to access these journals online.

Publishers for Development is bringing international publishers together who are interested in finding out how they might contribute to discourse and action around developing country access, encourage publication from developing country researchers and understand the diversity within research cultures/communities and the challenges these present.

I think that censorship is useful when it serves a higher purpose, for example to save lives, or to save vital money for underfunded universities in countries where bandwidth is expensive and there are alternative ways for students to access the uncensored Internet for private browsing purposes. I’m opposed to censorship that requires leaving the country or changing your ISP to get around it.

Walubengo wrote on the BMO Training mailing list:

Am just from the student labs and came across this sneaky little [software]:

http://www.ninjacloak.com/

It basically allows my students to get behind the good old
dansguardian/squid proxy_firewall; essentially allowing them to visit
and download all and sundry (read porn, warez, torrents et al)

[H]ave been wondering why the clamour to “open-up” the internet “for
research” had gone down (now I know).

Any quick counters? (beyond just blocking ninjacloak.com, since they are likely to get an equivalent sooner rather than later)

I have never used ninjacloak and I don’t intend to, but I’m sure that if you post some logs of its use from your proxy server, we can figure out how to block it.

However, no security is perfect. There will always be ways around any security measure that we implement. However, no workaround is perfect either. Once we understand how it works, e.g. what the requests that it makes look like, we can block it.

This quickly turns into an arms race between the user and the administrator. The winner is usually the one with the most time, patience and determination. This may be a fight that you don’t want to take on.

In my view, if users really really want to access some blocked content, they will find a way. However, a good security system will make it possible to at least trace that they did so, if not exactly what they accessed. So my approach would be two-fold:

1. Tackle the biggest problems first, and when they make sense. If someone uses ninjacloak to view a porn site once, it is hardly going to bring down your network, so you don’t need to care. If all your students are using TOR, AND it is bringing down your network, THEN it’s time to do something about it. If you don’t know what the biggest problem is, find out.
2. Don’t forget that social measures are far more effective than technical ones. If students know that they are being watched, they are much less likely to try things like this. Make REALLY sure that everyone knows and understands your policy. When you find students bypassing your security, go and talk to them. If necessary, consider the use of formal sanctions, which are likely to have a stronger deterrent effect.

If users think they are being treated unfairly or harshly, it can increase their determination to fight the system. If you have a good reason for censoring, because you can show them how much damage their actions are causing to legitimate or intended uses (such as academic research), they are much more likely to understand and comply with your requests, hopefully avoiding the need for sanctions.

nb: but again, someone may ask, why not just open up the internet any way?

Because (and only when) it wastes your precious bandwidth that’s better used for your core purpose (e.g. academic research), which is why you pay for the connection in the first place.