• PDF files
• Microsoft Office DOC, XLS and PPT files
• and the new XML equivalents, DOCX, XLSX and PPTX.

I found various tools online to help extract this text, largely thanks to Stack Overflow here and here. This ended up with a hodgepodge of tools:

• PDF Miner for PDF files
• python-docx for DOCX files
• DocToText for PPTX, XLSX and PPT files
• antiword for DOC files

There were a number of problems with this hodgepodge:

• I was unable to find any Python or command-line solution for old Excel (XLS) files;
• These solutions did not extract metadata, only document text;
• The choice of which tool to use depends on the MIME type returned by the file(1) command, which varies depending on the OS (Debian/Ubuntu or CentOS) and which version of the library is installed

Another Stack Overflow post recommended Apache Tika for metadata extraction. It appears to support all the document formats that we need, and to have auto-detection of the document format, which solves all the MIME type problems as well. However, it introduces a new problem: it’s written in Java, which is hard to access from Python.

Luckily I found some instructions for building a Python wrapper around Tika, using some tools that I’d never heard of, and this seemed like a good approach. Unfortunately the installation process is very non-standard, which would not fit in with our fabric-based automated deployment process, and would make it harder for users to install the Intranet themselves.

The instructions are somewhat outdated at the time of writing, as they refer to Tika version 0.7, while 1.0 has been released. I was unable to register for an account to update that page, so I wrote to the author with the details that I discovered, and will also document here that the following command works for me:

python ../jcc/jcc/__main__.py \
        --include /usr/share/java/org.eclipse.osgi.jar \
        --jar tika-parsers-1.0.jar \
        --jar tika-core-1.0.jar \
        java.io.File java.io.FileInputStream \
        java.io.StringBufferInputStream \
        --package org.xml.sax \
        --include tika-app-1.0.jar \
        --python tika --version 1.0 --reserved asm

I was able to go further than this, and package Tika in a way that makes it easy to install with Pip, and thus integrate with our deployment process.

The wrapper is written using JCC, which works by generating and compiling C++ code that links to the Java classes, and then a Python wrapper around that C++. This means that it needs to be recompiled for each platform, so I couldn’t just distribute a binary blob with the Intranet (I had the same problem with DocToText above).

The version of setuptools on our servers doesn’t support JCC’s shared library mode. JCC dies with an error if it’s not explicitly disabled or the patch applies. I couldn’t do either of these as part of our standard deployment process. So I patched JCC to disabled shared mode, since we don’t need it anyway. I also added some patches to allow various setup.py commands used by pip to be forwarded through JCC to the setup function call.

This seems to be enough to allow you to install JCC like this:

pip install git+git://github.com/aptivate/jcc.git

I also wrote a setup.py file that handles pip’s command line invokations and passes the necessary options to JCC, and JCC’s invocation of the setup function. This seems to be enough to install the package using pip:

pip install git+git://github.com/aptivate/python-tika.git

and you can use the last parameter as a package specification in pip_packages.txt, or whatever you pass to pip -r.

You can find the pip-installable Tika package, complete with Tika 1.0 JAR files, in our python-tika repository on Github. This will save you the work of downloading and compiling Tika and all of its dependencies. I have started a discussion with the JCC developers about merging these changes into the upstream project.

This is an area in which Aptivate is also very interested, and one in which I’ve done some research and been following developments. I still managed to learn quite a bit from three very interesting presentations:

Information Economy Report 2010 (UNCTAD)

The informational dimension of poverty, i.e. where information can help to alleviate or reduce poverty:

• Market price information
• Income-earning opportunities (e.g. jobs)
• Weather information and warnings
• Correct use of pesticides and fertilisers
• Health information and education
• Disaster risk reduction

Communication up and down the supply chain, and with peers and advisors, also helps.

There is an increasing trend to direct involvement of the beneficiaries in the production of ICTs:

• As ICT workers
• Manufacturing of ICTs (as an alternative occupation to subsistence farming)
• Providing IT and ICT-enabled services (answering questions, finding information, running telecentres)

Mobile phone penetration has exceeded all other ICTs in growth in developing countries. On average in the least developed countries, it has increased from 2% to 26% of the population (1000% growth) from 2000 to 2009. Possibly the fastest-spreading technology ever in the history of the world.

Growth is uneven. There are still some LDCs where less than 10% of the population have a mobile phone. In Ethiopia for example, only 5% have a phone. This was largely attributed to lack of liberalisation of telecomms markets.

Half of rural population in LDCs have no access to a mobile phone signal, which will limit the further growth of mobile usage. Many Universal Service Funds are sitting unused. In some cases this is because they are mandated only to be used on the fixed line network, which is nearly obsolete.

Mobile micro-insurance has become a big topic. For example:

• Kilimo Salama in Kenya
• Burkina Faso, Mali (index-based crop insurance)
• Alliance Afrique

Kilimo Salama recently made their first payouts to farmers because weather conditions exceeded their thresholds. The payouts are automatic and don’t have to be claimed by the farmers. The largest was about $30.

Even those who don’t have access to ICTs themselves can benefit from more transparent markets when enough participants use ICTs.

Download the full report (PDF, 171 Pages, 1240Kb).

Enabling role of ICTs to transform smallholder farmers to entrepreneurs (IFAD)

IFAD offers grants and loans to governments for argicultural development programmes. They are starting to offer grants (but not loans) to the private sector as well.

Grameen and BRAC had limited success with mobile banking (so far), because most of their customers are groups, not individuals, and mobile phones tend to be personal devices.

IFAD and WFP are running a joint project called the Weather Risk Management Facility (WRMF), a micro-insurance project. Half of the insurance premiums are paid by the farmers, and half by the sellers of inputs (seeds, fertilizer, pesticides) as they benefit from farmers being willing to buy more of their products due to reduced risk of crop failure.

ICTs enhancing plant production at the field level (FAO)

e-Locust2 uses vehicles with GPS, laptops and HF radio modems to send real-time information on locust swarms to governments, which can help to warn and prepare neighbouring villages and allow the targeted use of pesticides to control the pests. Time is critical to achieve this.

Digital Pens are being used to capture information entered on forms. The pen recognises what is being written, and where on the form, and captures the data for later upload. This makes it possible to have electronic filing with minimal training, minimal unreliable ICTs, an inherent fallback to paper-based methods, and hard copies of the forms that can be given to farmers or stored in local offices.

There are problems getting pest monitoring officials to enter high quality data when there is no incentive (reward) for accurate data, e.g. in one-way monitoring systems. If governments used this data to target their interventions, villagers would have a much more obvious incentive to ensure that the data was entered accurately and on time.

Thanks

Thanks to FAO for hosting this excellent seminar, and to the World Food Programme for allowing me time off to attend it.

Several of us expressed an interest in continuing the discussion online, we have been heard, and Michael Riggs, lead facilitator of the e-Agriculture Community, is working on enabling this to happen. There will also be a follow-on discussion at the ICTD 2010 Conference in London.

This is something that ought to be really simple. It’s probably the most common use case of VPNs, Windows has a built-in VPN client, and Cisco routers can be used as VPN servers. EWB want it to be simple, because they have non-technical remote workers. It turned out to be much harder and take much longer than I expected.

Information Overload

One of the biggest problems was the lack of useful information, and the profusion of useless. The information fell mainly into four categories:

• Cisco marketing materials touting the benefits of VPNs and their expensive Concentrator and WebVPN products;
• Cisco knowledge base articles describing the setup of complex VPN scenarios;
• Cisco command references with little or no details on what each command actually does, or how to use them together;
• Cisco exam study sites with inaccurate, out-of-date or cookie-cutter command sequences, with even less explanation of what the commands actually do.

Because I couldn’t find what I was looking for, and had to work it out the hard way, I’ve written it up in the hope that it will help others.

I would recommend any organisations that simply want to share files to seriously consider a file-sharing service like DropBox or raw Amazon S3 instead of a local file server and VPN. In many cases the low upload bandwidth of ADSL connections, combined with internal office use of the connection. will make a VPN impractically slow, especially compared to Amazon’s unlimited upload and download bandwidth. But EWB already had the file server and they just wanted to access it remotely, not to change how they work.

Our scenario is simple: an internal office network with private IP addresses, a Cisco 1800 router providing ADSL connectivity for the office, and remote field workers running Windows desktops.

Getting the Client

For simplicity, we and EWB had hoped to use the built-in VPN client on Windows, which would remove the need to download and install software on the remote workers’ machines. But unfortunately the Cisco 1800 does not support this. Windows uses L2TP over IPSEC for modern, secure VPNs, as a replacement for the old insecure PPTP protocol. But Cisco has crippled the L2TP support in this router, and it only supports raw IPSEC. Only their more expensive routers support serving L2TP over IPSEC, allowing simple direct connections from Windows.

Raw IPSEC is the only remaining option on this router, but it’s difficult to configure due to its complexity, and the number of choices that need to be made. The standard requires both sides to have the same settings configured, but provides no way to do this automatically. Manual configuration would make life very hard for the remote workers. To solve this problem, Cisco has a non-standard protocol for auto-configuration of the clients:

Establishing a VPN connection between two routers can be complicated, and it typically requires tedious coordination between network administrators to configure the two routers’ VPN parameters.

The Cisco Easy VPN Client feature eliminates much of this tedious work by implementing Cisco’s Unity Client protocol, which allows most VPN parameters to be defined at [the] IPSec server.

Cisco Easy VPN Client for the Cisco 1700 Series Routers

So we needed to find a replacement client that was easy to use and could talk to the Cisco. Preferably a free one.

Then we discovered that although Cisco’s own VPN client is technically free, you can’t actually download it without a support contract, which neither we nor EWB have.

In the end we found that if you go to Cisco’s VPN client software page, find the filename of the latest version of the client, and Google it, you’ll find that several people have had enough of this nonsense and posted the client online, so it can be downloaded.

Of course it’s important to be aware of the potential for viruses in copies that you download from random sites on the Internet, as well as fake download sites that lead you around in circles of free registrations, credit card details and pop-up porn adverts. This site worked fine for me, but it may have been taken down by Cisco’s attack dogs by the time you read this.

Security with Obscurity

We decided to choose a configuration that trades some security for ease of use. So instead of authenticating with certificates, we used pre-shared keys. The VPN server has its own login system anyway, which provides an additional layer of security once the remote user is connected to the VPN.

Names and Addresses

Connecting clients need to be allocated an IP address to use over the VPN. We could have used public IPs, or private IPs in the same subnet (with proxy ARP), but we chose to use private IPs in a different subnet. This makes the routing easier, as clients and local network servers will know that they have to route the traffic via the router anyway, and it allows EWB to implement stricter network access policies for VPN clients, if they wish.

We needed to create a local pool (not a DHCP pool) to draw these addresses from:

ip local pool vpnpool 192.168.2.100 192.168.2.200

Keys to the Kingdom

We created an ISAKMP (IKE) policy to specify the authentication method and the level of encryption to be used for negotiation of IPSEC Security Associations (SAs). We chose to make this the first, highest priority policy, and to use AES-256 encryption (strong and fast), Group 2 (1024-bit) Diffie-Hellman key exchange, and pre-shared keys for client authentication as noted above:

crypto isakmp policy 1
 encr aes 256
 authentication pre-share
 group 2

Then we specified the pre-shared key itself. This is the only thing that stops random clients on the Internet from connecting to your local network, so it’s even more important than a strong wireless network key. Of course this is not the real key:

crypto isakmp key ThisKeyMustBeKeptSecret address 0.0.0.0 0.0.0.0

We specify that any IP address can use it by using the wildcard address, 0.0.0.0 0.0.0.0.

At the End of the Tunnel

It seems to be common in corporate environments that, when a user is connected to a VPN, all of their Internet traffic is routed through the VPN. It certainly makes it easier for the network administrators, as they don’t have to define specific routes for the tunnel, but it wastes their bandwidth and makes Internet access much slower for the remote workers, so we decided not to do this.

Just routing a single subnet through a tunnel is called a split tunnel. I couldn’t find simple documentation on setting it up, so I used the Cisco Easy VPN Remote example, extracting just the bits we needed to route only the 192.168.1.0/24 subnet through the tunnel.

First we have to create an access control list (ACL) that defines, on the local (source address) side, what traffic clients should route into the tunnel:

ip access-list extended ewb_office_split_tunnel
 remark Defines which local (office) networks a remote VPN client will route to
 permit ip 192.168.1.0 0.0.0.255 192.168.2.0 0.0.0.255

I’m not sure if the second half of the ACL is actually necessary. It doesn’t appear to make any difference if I specify any instead of 192.168.2.0 0.0.0.255.

Client Configuration

We use Cisco’s EzVPN (Unity) protocol, as described earlier, to configure connecting clients automatically. To do this, we have to tell the server what configuration should be sent to clients when they connect:

crypto isakmp client configuration group EWB
 key ThisKeyMustBeKeptSecret
 dns 192.168.1.1
 wins 192.168.1.2
 pool vpnpool
 acl ewb_office_split_tunnel
 netmask 255.255.255.0

A little explanation about what these options do:

crypto isakmp client configuration group [name]

The name must match the group name that the client uses when it connects. This is how the server decides which configuration to send to the client.

key

For some reason the client needs to be told what key to use, even though it’s already been entered by the user, and the client knows it because it wouldn’t be able to get this far in the negotiation without it!

dns

Tells the client which DNS server to use, for resolving local (private) hostnames, or resolving inside the split horizon. You can specify a second DNS server after the primary one. You probably only need this if you’re running a Windows domain, in which case it should point to the domain controller, or if you have split horizon DNS.

wins

Tells the client which WINS server to use, for resolving local SMB server names. Again, you probably only need this if you’re running a Windows domain, in which case it should also point to the domain controller.

pool

Tells the server which local pool (not DHCP pool) to assign the client’s address from. You can specify any name here, even a pool that doesn’t exist, but clients won’t be able to connect unless the pool name is a valid local pool.

acl

This ACL, which we defined earlier, is used to tell the clients which subnets are reachable through the connection (split tunnel mode). If no acl statement is used, the tunnel is not split, and a default route is set through the VPN tunnel instead.

netmask

Defines the network mask that the client will apply to its client interface, in combination with the IP address assigned from the pool.

Profiling

Next, we create an ISAKMP profile on the server which tells the server to assign IP addresses automatically, and which virtual template to use when creating the virtual-access interfaces for the server side of the tunnel. We haven’t defined the virtual template yet, but we will in a second.

crypto isakmp profile ewb_isakmp_profile
   match identity group EWB
   isakmp authorization list sdm_vpn_group_ml_4
   client configuration address respond
   virtual-template 1

When a client connects using the group name EWB, it will check for network authorization using the AAA list name sdm_vpn_group_ml_4 (or default if that list doesn’t exist), respond to IP address requests from the client (using the pool defined in the client configuration above), and create a local virtual-access interface based on virtual template number 1.

You should use the same group name that you used for the client configuration above, instead of EWB, unless you’re EWB of course.

Strong Encryption

Now we define the level of encryption used for data communications with hosts on the internal network, as opposed to securing the negotiation process. We start by defining a transform set which uses 256-bit AES encryption, the SHA hash algorithm and LZS compression for data packets:

crypto ipsec transform-set ewb_encryption esp-aes 256 esp-sha-hmac comp-lzs

Then we create an IPsec profile that links these settings to the ISAKMP profile that we defined above:

crypto ipsec profile ewb_ipsec_profile
 set transform-set ewb_encryption
 set isakmp-profile ewb_isakmp_profile

Virtual Template

Now we define the template for the virtual interfaces, that we referenced above in the ISAKMP policy:

interface Virtual-Template1 type tunnel
 ip unnumbered Vlan1
 zone-member security in-zone
 tunnel mode ipsec ipv4
 tunnel protection ipsec profile ewb_ipsec_profile

We use ip unnumbered Vlan1 to set the IP address of the virtual-access interfaces to the address of the router on the local LAN (in this case it’s a VLAN bridge), which allows you to ping the router using its internal IP address (192.168.1.1 in our case) when you’re connected to the VPN, which is a useful connectivity test.

We place the virtual interfaces into the in-zone (internal zone) which means that they have full access to the local network, which is not very secure, but simplifies things. We also specify that this interface accepts only traffic encrypted with IPsec and bound to the profile that we created earlier. I’m not sure why it needs to be bound in both directions, as the IPsec profile is connected to the ISAKMP profile which is connected to this virtual interface already.

Client Setup

That should be it for the server-side setup. To configure a client, install the VPN software you downloaded earlier, start it, create a new IPsec configuration, and enter the following details:

Server

The public IP address of the VPN server

Group Name

The same group name that you used on the server earlier

Pre-Shared Key

The same key that you entered on the server earlier

Now click on the Connect button, and after a few seconds the window should minimize to the system tray, and you should be connected to the VPN. You can check this by pinging the internal IP address of the router (e.g. 192.168.1.1) and if that works, the IP addresses of whatever internal servers you want to connect to.

If it doesn’t work, use the Log menu to enable logging, try to connect again, and check the results on the Logging tab. You can also try enabling IPsec debugging on the router, in run mode (not configuration mode):

debug crypto engine packet
debug crypto ipsec error
debug crypto isakmp error
debug crypto verbose
terminal monitor

When the configuration works, write it to the router’s non-volatile memory to ensure that you don’t lose it when you next reboot the router:

write

And that’s it!

References

Here are some random unsorted links to pages that I found useful while figuring out how to do this:

• Configuring a Cisco Router to Accept VPN Connections (even simpler example, without split tunnels)
• Configuring Internet Key Exchange for IPsec VPNs (good general overview of how Cisco’s IPsec works)
• Cisco Easy VPN Remote configuration guide
• Cisco VPN client downloads
• Cisco ARP Commands
• Cisco ISAKMP command reference
• Configuring Zone Policy Firewalls

How to Install Computer Labs

If you ever need to set up a large number of computers in identical configurations, you have a few options:

• Install each one individually by hand
• Automate the standard install process, for example using:
  • AIK for Windows;
  • Kickstart for Red Hat Linux, CentOS and Ubuntu; or
  • sysinstall and mfsroot for FreeBSD.
• Configure one machine exactly how you like it, and then exactly duplicate the hard disk to the others (disk imaging)

The first option (manual installation) is extremely slow, tedious, error-prone, unlikely to result in identical machines, and does not speed up future installations or reinstallations.

The second option requires using rarely-used and less tested parts of the installer, slows down badly with multiple simultaneous installations (due to limited network bandwidth and bugs in the inetd TFTP server), and places limits on what you can customise. For example, it seems impossible to customise /etc/rc.conf using the installer on FreeBSD, and pre-installing SSH keys is tricky. I spent days writing a sysinstall script to automate the process. It would have taken just half an hour to set one machine up perfectly by hand, and then copy the system image onto all the other PCs in a few unattended hours.

Therefore I prefer the third option, system imaging.

What is System Imaging

Imaging is the process of making exact copies of one machine’s hard disk, including all partitions, onto another. This only works when the second hard disk is at least as large as the first. It works best when all the PCs are identical.

Imaging is independent of the operating system. You can image Windows, FreeBSD, any version of Linux, dual-boot and triple-boot installations, whatever you like.

We successfully used imaging to set up the PCs for these workshops:

• AfNOG 2009, Cairo, all tracks
• KENET BMO Training, Nairobi 2009
• AfNOG 2010, Kigali, for three tracks: SA-E, SS-E and SI-F

How to Image

Many systems administrators have heard of Norton Ghost and Acronis True Image, two of the most popular commercial applications.

However, open source alternatives such as G4L (Linux-based) and its ancestor G4U (FreeBSD-based) are pretty good, and completely free. G4L however lacks a website, and it’s not obvious how best to use it, hence this post.

G4L is quite similar to G4U, and I could have used G4U instead. But I find the Linux kernel’s hardware support a bit better than FreeBSD’s, and G4L supports multicasting, which enables it to install many machines at the same time with good performance.

Using Ghost for Linux (G4L)

I’ve successfully used Ghost 4 Linux (G4L) versions 0.27 and 0.33 for this process. 0.33 has multicast support, which allows setting up an entire room in one go, without wasting network bandwidth copying the same 4 GB disk image to each of 50 machines independently.

Set up an FTP server on your network with an account that supports downloads and uploads (e.g. on a local server on your network). Make sure it has plenty of disk space free, perhaps 40 GB. Create an “img” directory under the FTP user’s home directory for the images.

Download G4L and burn some CDs, maybe about five copies, or set up network booting (this conflicts with FreeBSD PXE installation and may require BIOS setup changes to enable PXE).

It’s a good idea to explore G4L and get used to the options, but please be very careful, as it has the potential to wipe your hard disk! So please use a machine with a fresh hard disk or which you don’t mind wiping.

To boot into G4L (you will need to do this several times below, but not yet, unless you just want to explore):

• Reboot or power up the machine
• Press the key to choose boot device
• If CD-ROM is not on the list, reboot, go into the BIOS and enable booting from CD-ROM
• Choose to boot from the CD
• Choose the default kernel at the GRUB screen (just press Enter)
  • If for some reason the default kernel doesn’t work, the machine hangs or crashes or doesn’t detect the network interface, then try one or two other kernels
• Wait for the kernel and initrd to be loaded (two long lines of dots)
• Then you can remove the CD, about one minute from cold boot, and start booting another PC
• Press space to skip each of the information/advertising screens (about 8 of them)
• Enter g4l at the prompt (if you go past this and get a shell, just type g4l at the shell prompt)
• You can access other consoles with Ctrl-Alt-F1 to F4, log in as g4l with no password, and run g4l, ifconfig, ping or whatever
• Choose Network Use (default)
• Choose Raw Mode (default)
• Check that you have an IP address (option B) or try again to acquire one by DHCP
• If you can’t get an IP address by DHCP, check your cabling and DHCP server

Create a Restore Image (optional)

Back up one of your PCs if necessary (if you plan to restore the PCs later) by:

• Follow the procedure above to get into Ghost for Linux
• Enter the FTP server’s IP address, username and password
• Choose an image name, e.g. backup_original_2010_07_22.img
• Choose the back up option
• Press Space to select the entire disk (mark it with an asterisk [*])
• Start backing up the image

This process can take 1-2 hours. In the mean time…

Set up the Master PC

If you don’t already have a master computer set up, it’s a good idea to WIPE THE DISK first. This makes the image much smaller, and transfer much faster. Please DO NOT do this if you have anything valuable on the master computer, for example an existing operating system installation that you want to keep.

Boot G4L on the PC that you will use as the master. Use DD to wipe the entire disk with zeroes:

dd if=/dev/zero of=/dev/sda bs=1M

Install FreeBSD or whatever operating system(s) on the master PC, and set it up exactly the way you want all of the PCs to be. Examples include:

• Install Gnome (gnome/gnome2)
• Install Xorg (x11/xorg)
• Install Firefox (www/firefox35)
• Install Xpdf (print/xpdf)
• Enable gnome and sshd in /etc/rc.conf, and add templates for the IP address configuration (this saves typing when setting all the machines to static IPs):

  hostname="pc01.sse.ws.afnog.org"
  ifconfig_bge0="dhcp"
  # ifconfig_bge0="196.200.219.101/24"
  defaultrouter="196.200.219.254"
  gnome_enable="YES"
  sshd_enable="YES"
  

• Create a user account (e.g. username afnog, password afnog)
• Log into Gnome, add firefox, terminal and the Downloads folder to your toolbar, and remove epiphany and evolution
• Edit /etc/fstab and add the proc filesystem:

  proc /proc procfs rw 0 0
  

  (this allows GDM to display the user list and shut down and restart the machine)

• Edit /etc/profile and set the default pager to less by adding:

  PAGER=less; export PAGER
  

• Set the timezone by softlinking /etc/localtime to something like /usr/share/zoneinfo/Africa/Kigali
• Create /etc/rc.local and have it run /usr/sbin/ntpd -qg to set the time once at boot

I recommend using DHCP on this machine. Otherwise all the imaged machines will boot up with the same IP address, causing IP address conflicts, and you will have to reconfigure them before you can access the Internet at all, or reconfigure them automatically.

Create some SSH keys for use in administering the machines. You may wish to set up the local server already and generate the keys there for security. I recommend adding the keys to /root/.ssh/authorized_keys. Please test that they work, and that sshd comes up automatically after boot!

Imaging the other PCs

On all the PCs (master and clones):

• Boot G4L as above
• Check that it has an IP address (option B)

Once a master is online, all the PCs will show “press any key to start”. Pressing any key on any computer will start all the machines imaging. If any PCs are not ready yet, you will have to cancel the imaging process on all of them and start again, or image those PCs later. So:

Start the master last! (when all the other PCs are ready)

Start the clones first, by following these steps on each one:

• Choose UDP Multicast Client (option U)
• Select the entire disk, /dev/sda with the space key
• Say yes, you’re sure
• When it says “Compressed UDP receiver”, it’s ready and waiting for a master to appear on the network

Then start the master:

• Get ALL the clones ready, as above, before doing this!
• On the master, choose UDP Multicast Server (option W)
• Select the entire disk, /dev/sda, with the space key
• Leave the options blank
• Say yes, you’re sure
• The master start accepting connections from clients, which will happen automatically. The screens on the clients will also change.
• Please check that every client says “Press any key to start”.
• If not, please check it for network problems, etc.
• DO NOT stop or kill the server now, unless you want to visit every client again!
• You can press Ctrl+C on the client and run g4l again to check the IP address, retry DHCP, and try the UDP Multicast Client option again.
• This is your last chance to join any remaining clients to the group for this imaging session!
• When all the clients are ready, press a key on the master to start transfer.

The master will show progress of the transfer, and an error line if any clients fail to respond. Clients that cause too many errors will be kicked out of the group and appear to “finish” early.

It’s difficult to tell if the imaging process finished successfully or failed on the clients. However it appears that FreeBSD is very good at detecting filesystem corruption, and will fail to boot if the image was not completely transferred. So you can test them by trying to boot FreeBSD and seeing if it boots completely or stops with a filesystem error. Ideally this would be improved in future versions of G4L.

I think that censorship is useful when it serves a higher purpose, for example to save lives, or to save vital money for underfunded universities in countries where bandwidth is expensive and there are alternative ways for students to access the uncensored Internet for private browsing purposes. I’m opposed to censorship that requires leaving the country or changing your ISP to get around it.

Walubengo wrote on the BMO Training mailing list:

Am just from the student labs and came across this sneaky little [software]:

http://www.ninjacloak.com/

It basically allows my students to get behind the good old
dansguardian/squid proxy_firewall; essentially allowing them to visit
and download all and sundry (read porn, warez, torrents et al)

[H]ave been wondering why the clamour to “open-up” the internet “for
research” had gone down (now I know).

Any quick counters? (beyond just blocking ninjacloak.com, since they are likely to get an equivalent sooner rather than later)

I have never used ninjacloak and I don’t intend to, but I’m sure that if you post some logs of its use from your proxy server, we can figure out how to block it.

However, no security is perfect. There will always be ways around any security measure that we implement. However, no workaround is perfect either. Once we understand how it works, e.g. what the requests that it makes look like, we can block it.

This quickly turns into an arms race between the user and the administrator. The winner is usually the one with the most time, patience and determination. This may be a fight that you don’t want to take on.

In my view, if users really really want to access some blocked content, they will find a way. However, a good security system will make it possible to at least trace that they did so, if not exactly what they accessed. So my approach would be two-fold:

1. Tackle the biggest problems first, and when they make sense. If someone uses ninjacloak to view a porn site once, it is hardly going to bring down your network, so you don’t need to care. If all your students are using TOR, AND it is bringing down your network, THEN it’s time to do something about it. If you don’t know what the biggest problem is, find out.
2. Don’t forget that social measures are far more effective than technical ones. If students know that they are being watched, they are much less likely to try things like this. Make REALLY sure that everyone knows and understands your policy. When you find students bypassing your security, go and talk to them. If necessary, consider the use of formal sanctions, which are likely to have a stronger deterrent effect.

If users think they are being treated unfairly or harshly, it can increase their determination to fight the system. If you have a good reason for censoring, because you can show them how much damage their actions are causing to legitimate or intended uses (such as academic research), they are much more likely to understand and comply with your requests, hopefully avoiding the need for sanctions.

nb: but again, someone may ask, why not just open up the internet any way?

Because (and only when) it wastes your precious bandwidth that’s better used for your core purpose (e.g. academic research), which is why you pay for the connection in the first place.

Management of Internet connections is a big subject. Whole books have been written about it, including the freely downloadable How To Accelerate Your Internet (BMO Book). However, for anyone who doesn’t have time to read it, I will briefly summarise the most important points that I can think of:

• have a clear, simple and strict Internet access policy, and enforce
  it.
• have enough bandwidth, AT LEAST 3 kbps per computer, uncontended. So if you have 1000 computers, you should have 3 MBits dedicated bandwidth, or 60 MBps if it’s shared or contended with a 20:1 contention ratio (typical ISPs).
• have competent network administrators. If you don’t have them, then hire or train them.
• implement good network management practices, e.g. by following the advice of the BMO Book.
• start by solving the problems that users complain most about, to give them the best possible service.
• monitor your network to understand how Internet bandwidth is being used.
• block misuses of Internet access that are causing problems for legitimate use of the Internet connection.
• ensure that client PCs have good, fast antivirus, perform well, are
  regularly reformatted and reimaged, and have strong local security to prevent unauthorized software installation.

Far more information on all of these topics can be found in the BMO book. I suggest starting with the Introduction if you’re interested.

She now has two computers connected via an ADSL router. Both computers run Ubuntu (8.06 and 9.10). I have set things up so that I can log into the router, and also SSH to both computers simultaneously: I use two different port numbers…

I now want to be able to see her desktops, but I haven’t figured out how to do this. Having read the Gnome help, I believe that the Gnome remote desktop is inherently insecure: I would prefer to tunnel things over SSH, probably using vncserver and vncviewer (or perhaps Vinagre).

Can anybody explain what I need to do to get this to work, please?

I get asked this kind of question so often that I thought I’d write it up somewhere so I could just point people to the post.

SSH port forwarding is not hard to do, once you get your head around how it actually works. Thanks to Alan for drawing this simple diagram:

SSH port forwarding is not like a VPN and it’s not magic. It’s quite like a proxy server:

• You tell SSH, with the -L option, to listen for connections on a port on your local side.
• SSH connects to the remote host immediately as usual, and then starts listening on this port.
• When it receives a connection on this port, it tells the other side (the SSH server that you connected to) to connect to the remote hostname and port that you specified.
• If the remote side succeeds, the two SSH processes join the two sides together, forwarding bytes from each side to the other.

(Note: it’s also possible to ask the remote SSH server to listen on a port on its side, with the -R option, and connect to a host and port on the client side, but in the interests of simplicity I will ignore that for today.)

I’ll show you the commands that I suggested to David, and then explain what they do:

ssh username@ip-address-of-ssh-server -p port1 -L 5901:localhost:5900
ssh username@ip-address-of-ssh-server -p port2 -L 5902:localhost:5900
vncviewer localhost:1 (connects to computer 1)
vncviewer localhost:2 (connects to computer 2)

This opens two SSH connections, one to each of the machines behind his firewall, which are completely independent of each other. One SSH connection would actually be enough, as we will see in a minute, but this way fit more logically with my explanation.

These commands contain some placeholders that must be adapted to your situation:

username

The user name that you want to connect as. You can omit the name and the @ sign if it’s the same as your logged-in user on the client.

ip-address-of-ssh-server

The IP address or hostname of the SSH server that you want to connect to. In David’s case, he can’t see the SSH server directly, so he needs to use the public IP address of the router here, and the router will forward the port to the SSH server on his internal network.

port1 and port2

David said that he can “SSH to both computers simultaneously [using] two different port numbers.” Presumably using port forwarding on his router. These are the two port numbers.

vncviewer localhost:1

This runs the VNC viewer on the client and tells it to connect to VNC display 1, which runs on port 5901 (by definition, VNC ports are display number plus 5900), which we already forwarded to computer 1 using SSH.

After running the two ssh commands command, the first SSH client will be listening on port 5901 on the machine that you run it on, and the second will be listening on port 5902.

After this, until you disconnect the SSH sessions or kill the clients in some way, whenever you connect to port 5901 on the client, it will tell the computer it’s connected to (computer 1) to connect to localhost port 5900 (that is, to its own VNC server) and then join the connections together, forwarding any data sent in either direction over the tunnel.

This part of the SSH command:

-L 5902:localhost:5900

tells the SSH client to Listen on port 5902 on the client, and when it receives a connection, to ask the other side (the server) to connect to (what it sees as) localhost port 5900, and SSH will forward communications between the two over the SSH tunnel.

Note first of all that we tell vncviewer to connect to localhost, not to the IP of the remote computer (internal or external). That’s because the client side of the SSH port forwarding is listening on localhost port 5901, and not any other IP address or port. If you connect to anything other than localhost port 5901, you will not end up talking to the local SSH client connected to computer 1.

Note secondly that when we created the tunnels, we told the ssh client to connect them to port 5900, also on localhost. This time, localhost is relative to the remote machine (the server), so we are telling it to connect to itself (not back to you). We could also specify any IP address and port that is reachable to the server, which is acting as our proxy in this case. However, we cannot specify an IP or port that is reachable to the client but not to the server, because the server will not be able to connect to it.

Now let’s imagine that we want to be able to VNC to both computers over a single SSH tunnel. We can do this by forwarding two different local ports, one to localhost, and one to the IP address of the other computer, like this:

ssh username@ip-address-of-ssh-server -p port1 -L 5901:localhost:5900 -L 5902:192.168.10.5:5900
vncviewer localhost:1 (connects to computer 1)
vncviewer localhost:2 (connects to computer 2)

This assumes that computer 2 has the internal (RFC1918) IP address 192.168.10.5, and allows connections from computer 1 to its port 5900.

Port forwarding is unlike a VPN in several ways. The client does not end up with routing to the ultimate destination, nor does it need it. This means that it works even if the client and server have different views of the IP space, for example if they are located in subnets that use the same IP range to refer to different machines.

The server does not try to connect to the ultimate destination until the client receives an incoming connection (e.g. from vncviewer in this case). At this point, it may discover that there is nothing listening on the port to which it was told to connect, or that the destination host is down, or the port is blocked by a firewall. The server informs the client of this, but the client has no way to pass this information onto the connection that it received, which is has already accepted. All it can do is close the connection.

This means, for example, that if you were to sit at the server and type vncviewer 192.168.10.5, and that computer was not running VNC, you might get a Connection refused error. However, if you sit at the client and type vncviewer localhost, you will see the connection is opened and immediately closed, as though the VNC process was listening but refused to talk to you for some reason. Do not be fooled into assuming that VNC is running on the destination. With SSH port forwarding, you have no idea.

You cannot forward ICMP (pings), UDP sockets (DNS) or any other protocol except TCP using port forwarding, so you will never be able to ping remote hosts using this method alone.

It is currently impossible to add new forwarded ports to an existing connection or to change the ultimate destination host and port, so you must disconnect and reconnect with a new command line instead. This is inconvenient in some cases, especially where you have a long-running process open in the shell. I recommend using ssh -N to open an ssh client that does only port forwarding and not a shell; then open a separate shell if you need one.

The ssh client cannot exit while any connection is open, so if you log out with connections open, it will appear to hang. All open connections will be closed if the ssh client is forcibly killed by a signal or escape character.

If your port forwarding doesn’t appear to be working, check that you don’t have another process listening on the same port. For example, in the VNC case, both Gnome and KDE desktop sharing create a VNC server on the standard port, 5900, so you cannot forward the local port 5900 to anywhere if you have remote desktop access enabled on the client. The easiest solution is to listen on different port numbers, like 5901 and 5902, which correspond to VNC displays 1 and 2 in the command examples above.

Finally, please note that the meaning of commands like these is very different depending on where it is run (on the client or on the server):

vncviewer localhost
vncviewer 192.168.10.5

This is because:

• The meaning of localhost is different depending on where you run it (on the client or on the server); it always means connecting to the same computer that the command is running on.
• The meaning of 192.168.10.5 (or any other IP address) similarly depends on where you run it (on the client or on the server); it is always relative to the computers that are reachable from the one running the command.
• Connections always appear to the recipient to be coming from the computer running the command, so when the client or the server connects to 192.168.10.5, even if that’s the same computer for both, it will see the connections coming from different IP addresses.

Tariq adds that you can also run:

ssh -D 9999 username@ip-address-of-ssh-server

where the -D option tells SSH to creates a SOCKS proxy server tunnel. You can then tell your web browser (and other clients with SOCKS support) to use localhost:9999 as a SOCKS proxy server. This will forward all your browsing through the SSH tunnel, which makes it look like you’re in a different location (e.g. to watch iplayer when not in the UK) and protects your unencrypted web browsing from random sniffers on public networks.

How does one determine the number of users,  a wireless network can support. I need to buy a wireless router to support 2000 users within an organization. The problem is how do I determine this capability given the specs of the wireless router.

To put it in a better way “what determines the number of users a wireless router can support”[?]

Although I’m not an expert on wireless networks, I have worked with them a bit, and I sent a reply that might be useful to others (I hope).

I’m not sure there’s an easy answer to that question. Some factors that may influence the decision are:

• The total bandwidth available to a single wireless access point (AP), e.g. 54 MBps for an 802.11g router. This also depends on the level of 802.11 that the clients support. An 802.11b client will use much more airtime per packet than an 802.11g client, so if most of your clients are 802.11b then you won’t get more than 11MBps per AP, regardless of the theoretical maximum of the AP.
• The frequency space available. There are only three non-overlapping 802.11b bands (maybe fewer for 802.11g), so no matter how many APs you have, the most bandwidth you could get in a given spot cannot be more than three times the bandwidth of one AP. Also, if they form a contiguous roaming network (same SSID and key) you have little or no control over which one a client will associate with, so you can’t evenly divide the available bandwidth between the three that you can see.
• The guard time between different transmissions and for RTS/CTS round trips. This will cut your available bandwidth at least in half from the theoretical maximum, and more if you have hidden nodes (which is close to inevitable with thousands of clients, unless they are all in the same room).
• The maximum number of clients that can associate with a given router. Most APs don’t publish this number, but Cradlepoint routers can handle between 4 and 64 clients per router. Keenan Systems reckons that “Once you have more than 25 clients associated most access points start to break down”. I’d guess that Cisco kit has the highest limit, especially the professional versions (not Linksys branded) and el cheapo generic Chinese kit has the lowest.
• If the AP is serving DHCP and running NAT (acting as a router as well as an AP) then the translation and DHCP tables of the router will be a limit. Some router DHCP servers only allow class C subnets, with a maximum of 253 usable client IP addresses per AP. It’s probably more advisable to use a real machine (with a hard disk) as a DHCP server.
• Similarly, if you don’t do NAT on the AP, then whatever handles the NAT on your Internet gateway will see the IPs of the individual machines, and will therefore need to be able to handle however many simultaneous IPs your clients have, and connections that they make.
• Whatever your DHCP server, the number of IPs available in your network subnet will limit the number of clients who can have a valid unique IP address at one time.
• The bandwidth of your Internet connection. The minimum that I’ve seen working at all is 3kbps per client, or 6 MBps with 2000 clients. That should be real bandwidth, not contended upstream by the ISP, otherwise multiply by the contention ratio. Don’t forget to include your fixed clients as well.

The best advice I can give you, never having built a wireless network this large myself, is to:

• Grit your teeth and buy the best kit you can find on the market. Be prepared to pay through the nose, e.g. $1000 per AP or more.
• Talk to the manufacturers about the maximum number of associated clients, and get assurances in writing that their kit can handle the load. Preferably get them to propose a solution for 2000 clients, also in writing.
• Use small cells with directional antennae and lots of APs in areas where you expect more than 10 clients at peak times.
• Try to scale your network up smoothly rather than buying a complete solution in one go. Don’t try to support 2000 clients in the first year, let alone the first day.
• Monitor and graph the performance of the network, particularly bandwidth, wireless contention, number of errors and number of associated clients, and identify hotspots.
• Keep one or two APs spare, and deploy them in the areas that are seeing the most activity.

Sunday Folayan wrote:

Must this network be implemented with JUST ONE wireless router? With one router … If you run 802.11bg at 2.4ghz, you have just about 2Mbps of bandwidth to play with, from one AP. If you deploy 802.11a at 5.8Ghz, you should get better than 10Mbps. If any of the clients is 802.11bg, the AP will default to 802.11bg, even if it is capable of 802.11a. With 2000 users, that is an average of 1Kbps or 5kbps at the best per subscriber! Could this be what you want?

To put it in a different way … One single AP cannot do it.

And Hervey Allen wrote:

From what I’ve experienced wireless router specifications and claims often do not match what you will experience in real-world use. I know of several large-scale installations (10,000+ users and above) who ended up using Cisco Aironet series routers with Power over Ethernet capabilities (PoE).

I will double-check, but last time I was on-site the upper limit for one of these wireless routers was around 50 concurrent users with light to moderate use. That is, a single user running a torrent can make an access point almost unusable for the other 49 potential users…

It would be interesting to hear from others on the list who have large wireless installations what their experience has been, and what hardware they have used.

Issues of giving out addresses, roaming, recapturing addresses, etc… are quite important.

Patrick Okui wrote:

Joel Ja did a pretty good presentation on what he’s learned from setting up wifi installations for the various meetings/events at NANOG27. A few things have changed in the wifi world since 2003 but the concepts are still valid.

Hamish Downer wrote in a comment to this post:

This page has some good answers. It is about tech conferences, but the basic problem of getting lots of people on wifi in a single space is covered by the solutions.

I fully agree with Hamish, the page has excellent advice from people who have actually done this, unlike me.

Finally, Mark Tinka replied:

I generally wouldn’t recommend vendors on a public mailing list in such variable matters as wireless deployments, but given the scale you’re considering, Aruba came to see me once (uninvited, as usual), and they seemed to have some rather interesting things to say re: their wireless product portfolio, with particular regard to large scale installations.

You might want to add them to your shopping list, but my guess is the price point is way-up-there, what with their controllers and all.

But be careful about “buying” everything they tell you (same goes for other vendors). As others have mentioned, binding assurances from them as well as PoC’s (proof of concept) before you sign would be great!

I hope this helps someone. Please let us know how you get on.

We didn’t have a backup MX because:

• It usually can’t check whether recipients are valid or not, and therefore must accept mail that it can’t deliver;
• It usually doesn’t have as good antispam checks as the primary, because it’s a hassle to keep it updated;
• Spammers usually abuse backup MXes to send more spam, including Joe Jobs.

I thought that this was OK because people who send us mail also have mail servers with queues, which should hold the mail until our server comes back up. It’s normal for mail servers to go down sometimes and this should not cause mail to be lost or returned.

However, we had a report that one of our users did not receive a mail addressed to them, and was told by the sender that it had bounced. I saw the bounce messsage and suspected Exchange, so I decided to check how long Exchange holds messages before bouncing them. Turns out it’s only five hours by default. Most mail servers hold mail for far longer, for example five days, sending a warning message back to the sender after one day.

Bouncing messages looks bad on us. Apart from making our main mail server more reliable we need a backup MX to accept mail when the master is down.

However I do still want to minimise the spam problem that this will cause. Therefore I configured our backup MX to only accept mail when the master is down. Otherwise it defers it, which will tell the sender to try sending it to the master (again).

How did I achieve this magic? With a little Exim configuration that took me a day and that I’m quite proud of. I set up a new virtual machine which just has Exim on it, nothing else. I configured it as an Internet host, and to relay for our most important domains. Then I created /etc/exim4/exim4.conf.localmacros with the following contents:

CHECK_RCPT_LOCAL_ACL_FILE=/etc/exim4/exim4.acl.conf
callout_positive_expire = 5m

This allows us to create a file called /etc/exim4/exim4.acl.conf which contains additional ACL (access control list) conditions. The other change, callout_positive_expire, I’ll describe in a minute.

I created /etc/exim4/exim4.acl.conf with the following contents:

# if we know that the primary MX rejects this address, we should too
deny
        ! verify = recipient/callout=30s,defer_ok
        message = Rejected by primary MX

# detect whether the callout is failing, without causing it to
# defer the message. only a warn verb can do this.
warn
        set acl_m_callout_deferred = true
        verify = recipient/callout=30s
        set acl_m_callout_deferred = false

# if the callout did not fail, and the primary mail server is not
# refusing  mail for this address, then it's accepting it, so tell
# our client to try again later
defer
        ! condition = $acl_m_callout_deferred
        message = The primary MX is working, please use it

# callout is failing, main server must be failing,
# accept everything
accept
        message = Accepting mail on behalf of primary MX

The first clause, which has a deny verb, does a callout to the recipient. A callout is an Exim feature which makes a test SMTP connection and starts the process of sending a mail, checking that the recipient would be accepted. This is designed to catch and block emails that the main server would reject. Our backup server has no idea what addresses are valid in our domains; only the primary knows that.

The callout response is cached for the default two hours if it returns a negative result (the recipient does not exist on the master) or five minutes (see callout_positive_expire above) if the address does exist. We use a defer_ok condition here so that if we fail to contact the master, we don’t defer the mail immediately, but instead assume that the address is OK and therefore continue to the next clause.

The second clause of the ACL, which has a warn verb, is what took me so long to work out. Normally, if a condition in a statement returns a result of defer, which means that it failed, the server will defer the whole message (tell the sender to come back later). In almost all cases this is the right thing to do, but it’s the exact opposite of what we want here. We want to accept mail if the callout is failing, not defer it, otherwise our backup MX is useless (it stops accepting mail if the primary goes down).

Because this is such an unusual thing to do, there is no configurable option for it in Exim. The only workaround that I found is that there is exactly one way to avoid a deferring condition causing the message to be deferred: a warn verb. The documentation for the warn verb says:

If any condition on a warn statement cannot be completed (that is, there is some sort of defer), the log line specified by log_message is not written… After a defer, no further conditions or modifiers in the warn statement are processed. The incident is logged, and the ACL continues to be processed, from the next statement onwards.

So what we do is:

1. Set the local variable
   acl_m_callout_deferred to true;
2. Try the callout. If it defers (cannot contact the primary server) then we stop processing the rest of the conditions in the warn statement, as described above;
3. If we get to this point, we know that the callout did not defer, so we set acl_m_callout_deferred to false.

The third clause  of the ACL, which has a defer verb, simply checks the variable that we set above. If we get this far then the primary server is not rejecting this address; and if it’s not deferring either, then it must be accepting mail for the address. In that case, we defer the message, telling our SMTP client to try again later, at which point it will hopefully succeed in delivering directly to the primary.

Callout result caching becomes a problem here. If the master was not reachable, but a previous callout had verified that a particular address existed, and that callout result was cached for the default 24 hours, then the backup MX would defer subsequent mail to that address for the next 24 hours, even if the master went down. This is why we changed the positive callout result caching time to 5 minutes earlier.

The fourth clause  of the ACL, which has an accept verb, is even simpler. It accepts everything that was not denied or deferred earlier. We can only get this far if the master is not accepting or rejecting mail for that address.

So far the configuration appears to work fine and has blocked 14 spam attempts (abusing the backup MX) in 14 hours.

East Africa currently depends completely on expensive satellite bandwidth. There are several projects in progress to lay submarine cables down the east and west coasts, and it can be difficult to keep track of them all.

Luckily, Steve Song has drawn an excellent map here. I’m linking it here to help people find it (including myself, next time I need that map).

We also need overland connections to help bring that bandwidth to landlocked countries, to help them share and compete with each other, and to network rural areas. Fibre, copper and microwave are the traditional and expensive options, O3B wants to provide a satellite alternative, but the TIER group‘s WiLDNet project has the most disruptive potential in my view, potentially replacing microwave links with something that’s a hundred times cheaper and can be bought off the shelf.

TIER also wants to see their technology used to provide international bandwidth and compete with the undersea cables:

The vision is to connect Gilbraltar, which has low-cost world-class bandwidth and hosting, overland via long-distance Wifi through Morocco/Algerian, Mali, Burkina Faso, to Ghana.  This means crossing the Sahara, which is certainly not trivial.  (Timbuktu is roughly on this path.)  The article said 6 Mb/s, but I am thinking something much higher.  Although this is a crazy idea, I think it is much cheaper than many proposed projects, and if it worked you could grow the network over time and also increase BW for busy links, even moving to fiber once you have the traffic to pay for it.

(reference, video)