Boot Camp

AfNOG boot camp was absolutely massive this year. I think they had 75 people when they were only expecting 40. They took over half our classroom as well, which made setup tricky as we had to work around people and ask them to move repeatedly, and we couldn’t get all of our tables in to cable them up.

It was followed by the obligatory welcome dinner, at the White Sands’ outdoor restaurant, with the requisite number of speeches and applauses.

Today we had the first day of Scalable Services. Desktop installation hadn’t gone too well. My attempt to respin with fixes, wiping the unused space after the imaged partition, failed badly and resulted in a corrupted image, so we had to reimage those boxes.

Welcome

Luckily it seems that everyone brought laptops, so the PCs aren’t really needed. And the virtual machines seem to be working well so far. We haven’t yet had to compile any software on the virtual machines, and I hope it won’t be too slow when we do. We’re using 34 out of the 35 virtual machines that we created.

Tomorrow is my first session, a 1 hour practical on virtualisation, installing VirtualBox and FreeBSD, after Joel’s theory session.

Alan Barrett laying cable

I’m in Dar es Salaam, Tanzania for AfNOG 2011. I arrived on Wednesday morning at 7am (on the red-eye flight from London Heathrow) and I’m here until Tuesday 7th June.

Until now we’ve been setting up the venue. We’ve been super busy, working until midnight every night so far. We had to run our own cables, quite a lot of them (over 600 metres).

Running them through the windows was tricky, since we needed to be able to close them for security, and to allow the air conditioning to work. Someone (Alan?) came up with the genius idea of using tough palm leaves wrapped around them to protect them as they pass through the narrow gap between window panes. Bio-degradable trunking!

To cope with the power failures, Geert Jan built a monster Power-over-Ethernet injector to power the wireless access points in each room and keep the wireless network running.

The training workshops start tomorrow, Sunday 29th May, with the Unix Boot Camp, an introduction to Unix and the command line. We expect that many of the participants will have little experience with Unix, as has been the case in previous years. These free tools have immense benefits, both for us running the workshops and for the participants when they return home. But they are very different to the Windows environments that the participants are most familiar with. Without basic skills, they would struggle and hold back the group during the rest of the workshops.

Feeding the cable monster

I’m not involved in the boot camp, but after it finishes, we move straight into the main tracks, which last for five days. This year we have some new tracks: Network Monitoring & Management, Advanced Routing Techniques and Computer Emergency Response Team training.

We have also cancelled the basic Unix System Administration track (SA-E) this year, as it has finally been localised to most African countries, and therefore people have the opportunity to attend it locally at lower cost and build local communities. However, this leaves us with nowhere to cover more advanced systems administration techniques, which are some of my favourite topics, including:

Geert Jan and the Monster Injector

• virtualisation (desktops, servers and thin clients, VirtualBox, Xen, KVM, jails, lxc)
• system imaging (ghost, snapshots)
• backups (snapshots, Rsync, Rdiff-backup, Duplicity)
• file servers (NFS, Samba, sshfs, AFS, ZFS)
• virtualised storage (iSCSI, ATAoE, Fibre Channel, DRBD)
• cloud computing (Amazon and Linode virtual servers, scripting and APIs)
• cluster computing (Mosix, virtual machine host clusters)
• DHCP (for network management and booting)
• network security (firewalls, port locking, 802.1x)
• wireless networks (planning, monitoring, troubleshooting, WEP and WPA, 802.1x authentication)
• Windows domains and security (including Samba 4)

If participants show enough interest in these topics, they could be added in future. I think it’s unfortunate that the course is arranged into week-long tracks rather than half-day or one-day sessions from which people could pick and choose, Bar Camp style. That would make it much easier for people to run sessions on many new topics.

Some of our 80 desktop computers

In the past this would have been difficult, because we provided desktop computers for participants. It used to take us 3-4 days to set up 80-odd desktop PCs with customised FreeBSD installations. We’ve noticed that more and more people are coming to the workshops with their laptops, and this time we’ve made a big effort to shift from dedicated to virtual platforms, to reduce setup time and costs in future.

The hardest track to do this for, in my opinion, was Scalable Services English (SS-E), the one I’m working on. We were all pretty much agreed to stay with desktop PCs this year, making us the only track to do so. But when we arrived, we discovered that the mains power situation here is pretty awful. On Wednesday we had four power failures. We only have five UPS, not nearly enough to protect every desktop.

For participants with laptops, they effectively have their own built-in UPS. If we give them virtual machines to work with, then we only have to protect the hosts. We can keep those in the NOC (Network Operations Centre), where the UPS are, so they’ll be protected for around 15 minutes of any power outage, which we have to hope will be enough for the hotel to start their generator.

Cannibalising RAM

Some participants will probably forget their laptops, so we’ll provide them with desktops, but we have no way to UPS them. These desktops will be set up with FreeBSD, as in previous years.

We rented 80 machines from a local company. Some had Windows, in varying states of repair, some had no operating system installed. We decided to use some of these desktops as hosts for the participants’ virtual machines.They only had 2 GB of RAM each, but since we had plenty, we cannibalised eight others for their RAM to upgrade our machines to 4 GB each.

We decided to use VirtualBox for the virtual machines. It’s free, open source, can host on all major platforms (Windows, Mac, Linux and even FreeBSD), has a nice GUI and a command-line automation tool, supports bridged networking easily, and is relatively fast and efficient.

Imaging backend

We configured the master (that we’ll copy onto the other machines) starting with the setup from last year. We then had to install VirtualBox and build our first virtual machine inside it. Then we discovered that the virtual machine was unable to access the network in bridged mode. Packets sent by the virtual machine we simply never sent by the host. We needed to use bridged mode so that participants could run services on their machines simply by installing them. without requiring extra configuration on the host.

We had no Internet access for most of that day, because all three of our redundant providers were down for different reasons. Eventually we managed to use Geert Jan’s 3G dongle to get online and research the problem. We found that bridged networking doesn’t work in the binary package of VirtualBox 3.2.12 that comes with FreeBSD 8.2, so we had to wait until Internet access was fixed to download 120 MB of software (ports updates and VirtualBox 4.0.8) like this:

Imaging frontend

pkg_add -r portupgrade
portsnap fetch extract update
portupgrade virtualbox-ose virtualbox-ose-kmod

This took a long time, as VirtualBox is a large piece of software which also required us to download and build a new version of QT, but eventually it succeeded and the problem was solved.

We decided to put only five virtual machines on each host. Sometimes we would have the whole class compiling software from ports, which would slow down all of them significantly. We will use six or seven servers to host 30-35 virtual machines. On the master host, we created five copies of our master virtual machine by copying its hard disk like this:

cd .VirtualBox/HardDisks
for i in 1 2 3 4 5; do
	cp AfNOG\ SSE\ Master.vdi vm0$i.vdi
	VBoxManage internalcommands sethduuid vm0$i.vdi
done

Relocation

Then we created the virtual machines in the VirtualBox GUI and attached them to these new images. We needed to generate a new UUID for each disk image copy, using the undocumented sethduuid command above, otherwise VirtualBox would refuse to add the copies because it had a disk image already registered with the same UUID.

We could have created the virtual machines using the VBoxManage command as well, but it would have taken longer to work out how to use it than simply to create the five machines by hand. I later worked out the commands that we could have used:

cd ~/"VirtualBox VMs"
for i in {1..5}; do
	echo $i
	vmname=VM0$i
	diskimage="$vmname/FreeBSD.vdi"
	VBoxManage createvm --name "$vmname" --ostype FreeBSD
	VBoxManage modifyvm "$vmname" --memory 256 \
		--nic1 bridged --bridgeadapter1 bge0.219 \
		--nic2 bridged --bridgeadapter2 bge0.$[50+$i] \
		--vram 4 --pae off --audio none --usb on \
		--uart1 0x3f8 4 --uartmode1 server /home/chris/"$vmname"-console.pipe
	VBoxManage storagectl "$vmname" --name "IDE Controller" --add ide
	cp VM01/FreeBSD.vdi "$diskimage"
	VBoxManage internalcommands sethduuid "$diskimage"
	VBoxManage storageattach "$vmname" --storagectl "IDE Controller" \
		--port 0 --device 0 --type hdd --medium "$diskimage"
done

We named the images VM01 to VM05, which was important for running automated scripts on them. Then we configured VirtualBox to start them automatically at boot time, in headless mode, by adding the following lines to /etc/rc.conf:

vboxheadless_enable="YES"
vboxheadless_machines="VM01 VM02 VM03 VM04 VM05"
vboxheadless_user="inst"

We wrote a short script to help us apply the same command to all five virtual machines:

#!/bin/sh
# script to control all five virtual machines

command=$1
shift

for i in 1 2 3 4 5; do
	VBoxManage $command VM0$i "$@"
done

This allows us to log into a machine and do things like:

• ./manage acpipowerbutton to initiate a controlled shutdown of all five virtual machines
• ./manage modifyvm --macaddress1 auto to generate new, random MAC addresses after cloning the host
• ./manage startvm --type headless to get the virtual machines running again (headlessly, independent of the GUI)

The NOC

We knew that we wouldn’t have space to attach monitors and keyboards to all the hosts, and we’d have to fiddle about with cables in the hot NOC room (without working aircon) if we needed access to their consoles, so we added the ability to log into them remotely using VNC and GDM. To do this, we had to install the VNC server:

pkg_add -r vnc

Which unfortunately doesn’t come with the nice xorg loadable module that adds a built-in VNC server to the X server, making a fast and stateless remote control session possible. So we had to hack about with inetd, first by adding a service name with a port number to /etc/services:

vnc		5900/tcp

And then a service line in /etc/inetd.conf:

vnc	stream	tcp	nowait		root	/usr/local/bin/Xvnc Xvnc -inetd :1 -query localhost -geometry 1024x768 -depth 24 -once -fp /usr/local/lib/X11/fonts/misc/ -securitytypes=none

This requires us to enable the XDMCP protocol in GDM, in order for VNC to communicate with it to present a GDM login screen. So we replaced the contents of /usr/local/etc/gdm/custom.conf with the following:

[xdmcp]
Enable=true

[security]
DisallowTCP=false

And then restarted GDM:

sudo /usr/local/etc/rc.d/gdm restart

And checked that we could connect from another machine and got a login prompt:

vncviewer 196.200.217.128

Which did indeed give us a working login screen:

VNC graphical login on a FreeBSD virtual machine host

This method is very slow. I wanted to find a better way to access the guests, especially if their network configuration was broken. I tried to connect a host serial port to a pipe and then access that pipe from a shell command, eventually over ssh, in a similar way to the text-only console offered by Xen (xm console). The above VBoxManage commands set up a pipe in my home directory, and I wrote the following short script to access it:

#!/bin/sh
set -x
echo "Console for $USER"
nc -U /home/chris/$USER-console.pipe

I created user accounts for each virtual machine, with the same name, and set their shells to this script, so that when they log in, they would automatically be connected to the pipe. However I was unable to make it work well. In particular, because of incompatible terminal emulations, I was unable to run vi to edit configuration files in the guest. If you find a way around this, please let me know. I haven’t tried it yet, but conman looks like it might be a good bet.

I spent a long time searching for the hidden VNC support in VirtualBox 4. It’s undocumented (the manual only talks about RDP) and people on the IRC channel say that it doesn’t exist, but it does, at least when starting the guests in headless mode. I added the following lines to /etc/rc.conf:

vboxheadless_VM01_flags="-n -m 5901"
vboxheadless_VM02_flags="-n -m 5902"
vboxheadless_VM03_flags="-n -m 5903"
vboxheadless_VM04_flags="-n -m 5904"
vboxheadless_VM05_flags="-n -m 5905"

And then, after starting the guests in headless mode, I could connect to these ports and access the virtual displays, much more conveniently and much faster than by shutting down the guests using VBoxManage and starting them again using the VirtualBox GUI.

We used multicast to image the six virtual machine hosts from the master. This took about three hours, so we left it running overnight.

In the morning we checked that the hosts had been imaged successfully by booting them with their newly installed images, and gave them unique hostnames (host1.sse.ws.afnog.org etc.) and IP addresses.

We used the manage script to reset the MAC addresses of the network cards of each virtual machine on each host:

for i in 128 129 130 131 132 133 134; do ssh 196.200.217.$i ./manage acpipowerbutton; done
for i in 128 129 130 131 132 133 134; do ssh 196.200.217.$i ./manage modifyvm --macaddress1 auto; done
for i in 128 129 130 131 132 133 134; do ssh 196.200.217.$i ./manage startvm; done

Astral projection

Since they were all configured for DHCP, we could have got their IP addresses from the DHCP server, but we wanted to give them a nice naming scheme, so we logged in to each one (using the console and the VirtualBox GUI) and assigned it a unique hostname and a static IP address.

We checked that we could log into each virtual machine remotely using the SSH keys that we’d installed, and then we shut down the hosts and moved them to the NOC.

Boot camp starts tomorrow, next door, but we still have to arrange our room.

Classroom

We may have up to 37 people, our biggest class ever, in a room that’s about eight metres on a side, so layout of the room is a real challenge.

I wanted to do something to facilitate working in groups, such as each table having four places (two each side) and with its long axis front-to-back. This was vetoed because participants would have to turn their heads to see the projected screen, and it might be hard for them to take notes as a result.

So we’re going to have long, cramped benches instead. I think this is unfortunate, and I hope I can persuade people to try something more imaginative in future.

How to Install Computer Labs

If you ever need to set up a large number of computers in identical configurations, you have a few options:

• Install each one individually by hand
• Automate the standard install process, for example using:
  • AIK for Windows;
  • Kickstart for Red Hat Linux, CentOS and Ubuntu; or
  • sysinstall and mfsroot for FreeBSD.
• Configure one machine exactly how you like it, and then exactly duplicate the hard disk to the others (disk imaging)

The first option (manual installation) is extremely slow, tedious, error-prone, unlikely to result in identical machines, and does not speed up future installations or reinstallations.

The second option requires using rarely-used and less tested parts of the installer, slows down badly with multiple simultaneous installations (due to limited network bandwidth and bugs in the inetd TFTP server), and places limits on what you can customise. For example, it seems impossible to customise /etc/rc.conf using the installer on FreeBSD, and pre-installing SSH keys is tricky. I spent days writing a sysinstall script to automate the process. It would have taken just half an hour to set one machine up perfectly by hand, and then copy the system image onto all the other PCs in a few unattended hours.

Therefore I prefer the third option, system imaging.

What is System Imaging

Imaging is the process of making exact copies of one machine’s hard disk, including all partitions, onto another. This only works when the second hard disk is at least as large as the first. It works best when all the PCs are identical.

Imaging is independent of the operating system. You can image Windows, FreeBSD, any version of Linux, dual-boot and triple-boot installations, whatever you like.

We successfully used imaging to set up the PCs for these workshops:

• AfNOG 2009, Cairo, all tracks
• KENET BMO Training, Nairobi 2009
• AfNOG 2010, Kigali, for three tracks: SA-E, SS-E and SI-F

How to Image

Many systems administrators have heard of Norton Ghost and Acronis True Image, two of the most popular commercial applications.

However, open source alternatives such as G4L (Linux-based) and its ancestor G4U (FreeBSD-based) are pretty good, and completely free. G4L however lacks a website, and it’s not obvious how best to use it, hence this post.

G4L is quite similar to G4U, and I could have used G4U instead. But I find the Linux kernel’s hardware support a bit better than FreeBSD’s, and G4L supports multicasting, which enables it to install many machines at the same time with good performance.

Using Ghost for Linux (G4L)

I’ve successfully used Ghost 4 Linux (G4L) versions 0.27 and 0.33 for this process. 0.33 has multicast support, which allows setting up an entire room in one go, without wasting network bandwidth copying the same 4 GB disk image to each of 50 machines independently.

Set up an FTP server on your network with an account that supports downloads and uploads (e.g. on a local server on your network). Make sure it has plenty of disk space free, perhaps 40 GB. Create an “img” directory under the FTP user’s home directory for the images.

Download G4L and burn some CDs, maybe about five copies, or set up network booting (this conflicts with FreeBSD PXE installation and may require BIOS setup changes to enable PXE).

It’s a good idea to explore G4L and get used to the options, but please be very careful, as it has the potential to wipe your hard disk! So please use a machine with a fresh hard disk or which you don’t mind wiping.

To boot into G4L (you will need to do this several times below, but not yet, unless you just want to explore):

• Reboot or power up the machine
• Press the key to choose boot device
• If CD-ROM is not on the list, reboot, go into the BIOS and enable booting from CD-ROM
• Choose to boot from the CD
• Choose the default kernel at the GRUB screen (just press Enter)
  • If for some reason the default kernel doesn’t work, the machine hangs or crashes or doesn’t detect the network interface, then try one or two other kernels
• Wait for the kernel and initrd to be loaded (two long lines of dots)
• Then you can remove the CD, about one minute from cold boot, and start booting another PC
• Press space to skip each of the information/advertising screens (about 8 of them)
• Enter g4l at the prompt (if you go past this and get a shell, just type g4l at the shell prompt)
• You can access other consoles with Ctrl-Alt-F1 to F4, log in as g4l with no password, and run g4l, ifconfig, ping or whatever
• Choose Network Use (default)
• Choose Raw Mode (default)
• Check that you have an IP address (option B) or try again to acquire one by DHCP
• If you can’t get an IP address by DHCP, check your cabling and DHCP server

Create a Restore Image (optional)

Back up one of your PCs if necessary (if you plan to restore the PCs later) by:

• Follow the procedure above to get into Ghost for Linux
• Enter the FTP server’s IP address, username and password
• Choose an image name, e.g. backup_original_2010_07_22.img
• Choose the back up option
• Press Space to select the entire disk (mark it with an asterisk [*])
• Start backing up the image

This process can take 1-2 hours. In the mean time…

Set up the Master PC

If you don’t already have a master computer set up, it’s a good idea to WIPE THE DISK first. This makes the image much smaller, and transfer much faster. Please DO NOT do this if you have anything valuable on the master computer, for example an existing operating system installation that you want to keep.

Boot G4L on the PC that you will use as the master. Use DD to wipe the entire disk with zeroes:

dd if=/dev/zero of=/dev/sda bs=1M

Install FreeBSD or whatever operating system(s) on the master PC, and set it up exactly the way you want all of the PCs to be. Examples include:

• Install Gnome (gnome/gnome2)
• Install Xorg (x11/xorg)
• Install Firefox (www/firefox35)
• Install Xpdf (print/xpdf)
• Enable gnome and sshd in /etc/rc.conf, and add templates for the IP address configuration (this saves typing when setting all the machines to static IPs):

  hostname="pc01.sse.ws.afnog.org"
  ifconfig_bge0="dhcp"
  # ifconfig_bge0="196.200.219.101/24"
  defaultrouter="196.200.219.254"
  gnome_enable="YES"
  sshd_enable="YES"
  

• Create a user account (e.g. username afnog, password afnog)
• Log into Gnome, add firefox, terminal and the Downloads folder to your toolbar, and remove epiphany and evolution
• Edit /etc/fstab and add the proc filesystem:

  proc /proc procfs rw 0 0
  

  (this allows GDM to display the user list and shut down and restart the machine)

• Edit /etc/profile and set the default pager to less by adding:

  PAGER=less; export PAGER
  

• Set the timezone by softlinking /etc/localtime to something like /usr/share/zoneinfo/Africa/Kigali
• Create /etc/rc.local and have it run /usr/sbin/ntpd -qg to set the time once at boot

I recommend using DHCP on this machine. Otherwise all the imaged machines will boot up with the same IP address, causing IP address conflicts, and you will have to reconfigure them before you can access the Internet at all, or reconfigure them automatically.

Create some SSH keys for use in administering the machines. You may wish to set up the local server already and generate the keys there for security. I recommend adding the keys to /root/.ssh/authorized_keys. Please test that they work, and that sshd comes up automatically after boot!

Imaging the other PCs

On all the PCs (master and clones):

• Boot G4L as above
• Check that it has an IP address (option B)

Once a master is online, all the PCs will show “press any key to start”. Pressing any key on any computer will start all the machines imaging. If any PCs are not ready yet, you will have to cancel the imaging process on all of them and start again, or image those PCs later. So:

Start the master last! (when all the other PCs are ready)

Start the clones first, by following these steps on each one:

• Choose UDP Multicast Client (option U)
• Select the entire disk, /dev/sda with the space key
• Say yes, you’re sure
• When it says “Compressed UDP receiver”, it’s ready and waiting for a master to appear on the network

Then start the master:

• Get ALL the clones ready, as above, before doing this!
• On the master, choose UDP Multicast Server (option W)
• Select the entire disk, /dev/sda, with the space key
• Leave the options blank
• Say yes, you’re sure
• The master start accepting connections from clients, which will happen automatically. The screens on the clients will also change.
• Please check that every client says “Press any key to start”.
• If not, please check it for network problems, etc.
• DO NOT stop or kill the server now, unless you want to visit every client again!
• You can press Ctrl+C on the client and run g4l again to check the IP address, retry DHCP, and try the UDP Multicast Client option again.
• This is your last chance to join any remaining clients to the group for this imaging session!
• When all the clients are ready, press a key on the master to start transfer.

The master will show progress of the transfer, and an error line if any clients fail to respond. Clients that cause too many errors will be kicked out of the group and appear to “finish” early.

It’s difficult to tell if the imaging process finished successfully or failed on the clients. However it appears that FreeBSD is very good at detecting filesystem corruption, and will fail to boot if the image was not completely transferred. So you can test them by trying to boot FreeBSD and seeing if it boots completely or stops with a filesystem error. Ideally this would be improved in future versions of G4L.

How does one determine the number of users,  a wireless network can support. I need to buy a wireless router to support 2000 users within an organization. The problem is how do I determine this capability given the specs of the wireless router.

To put it in a better way “what determines the number of users a wireless router can support”[?]

Although I’m not an expert on wireless networks, I have worked with them a bit, and I sent a reply that might be useful to others (I hope).

I’m not sure there’s an easy answer to that question. Some factors that may influence the decision are:

• The total bandwidth available to a single wireless access point (AP), e.g. 54 MBps for an 802.11g router. This also depends on the level of 802.11 that the clients support. An 802.11b client will use much more airtime per packet than an 802.11g client, so if most of your clients are 802.11b then you won’t get more than 11MBps per AP, regardless of the theoretical maximum of the AP.
• The frequency space available. There are only three non-overlapping 802.11b bands (maybe fewer for 802.11g), so no matter how many APs you have, the most bandwidth you could get in a given spot cannot be more than three times the bandwidth of one AP. Also, if they form a contiguous roaming network (same SSID and key) you have little or no control over which one a client will associate with, so you can’t evenly divide the available bandwidth between the three that you can see.
• The guard time between different transmissions and for RTS/CTS round trips. This will cut your available bandwidth at least in half from the theoretical maximum, and more if you have hidden nodes (which is close to inevitable with thousands of clients, unless they are all in the same room).
• The maximum number of clients that can associate with a given router. Most APs don’t publish this number, but Cradlepoint routers can handle between 4 and 64 clients per router. Keenan Systems reckons that “Once you have more than 25 clients associated most access points start to break down”. I’d guess that Cisco kit has the highest limit, especially the professional versions (not Linksys branded) and el cheapo generic Chinese kit has the lowest.
• If the AP is serving DHCP and running NAT (acting as a router as well as an AP) then the translation and DHCP tables of the router will be a limit. Some router DHCP servers only allow class C subnets, with a maximum of 253 usable client IP addresses per AP. It’s probably more advisable to use a real machine (with a hard disk) as a DHCP server.
• Similarly, if you don’t do NAT on the AP, then whatever handles the NAT on your Internet gateway will see the IPs of the individual machines, and will therefore need to be able to handle however many simultaneous IPs your clients have, and connections that they make.
• Whatever your DHCP server, the number of IPs available in your network subnet will limit the number of clients who can have a valid unique IP address at one time.
• The bandwidth of your Internet connection. The minimum that I’ve seen working at all is 3kbps per client, or 6 MBps with 2000 clients. That should be real bandwidth, not contended upstream by the ISP, otherwise multiply by the contention ratio. Don’t forget to include your fixed clients as well.

The best advice I can give you, never having built a wireless network this large myself, is to:

• Grit your teeth and buy the best kit you can find on the market. Be prepared to pay through the nose, e.g. $1000 per AP or more.
• Talk to the manufacturers about the maximum number of associated clients, and get assurances in writing that their kit can handle the load. Preferably get them to propose a solution for 2000 clients, also in writing.
• Use small cells with directional antennae and lots of APs in areas where you expect more than 10 clients at peak times.
• Try to scale your network up smoothly rather than buying a complete solution in one go. Don’t try to support 2000 clients in the first year, let alone the first day.
• Monitor and graph the performance of the network, particularly bandwidth, wireless contention, number of errors and number of associated clients, and identify hotspots.
• Keep one or two APs spare, and deploy them in the areas that are seeing the most activity.

Sunday Folayan wrote:

Must this network be implemented with JUST ONE wireless router? With one router … If you run 802.11bg at 2.4ghz, you have just about 2Mbps of bandwidth to play with, from one AP. If you deploy 802.11a at 5.8Ghz, you should get better than 10Mbps. If any of the clients is 802.11bg, the AP will default to 802.11bg, even if it is capable of 802.11a. With 2000 users, that is an average of 1Kbps or 5kbps at the best per subscriber! Could this be what you want?

To put it in a different way … One single AP cannot do it.

And Hervey Allen wrote:

From what I’ve experienced wireless router specifications and claims often do not match what you will experience in real-world use. I know of several large-scale installations (10,000+ users and above) who ended up using Cisco Aironet series routers with Power over Ethernet capabilities (PoE).

I will double-check, but last time I was on-site the upper limit for one of these wireless routers was around 50 concurrent users with light to moderate use. That is, a single user running a torrent can make an access point almost unusable for the other 49 potential users…

It would be interesting to hear from others on the list who have large wireless installations what their experience has been, and what hardware they have used.

Issues of giving out addresses, roaming, recapturing addresses, etc… are quite important.

Patrick Okui wrote:

Joel Ja did a pretty good presentation on what he’s learned from setting up wifi installations for the various meetings/events at NANOG27. A few things have changed in the wifi world since 2003 but the concepts are still valid.

Hamish Downer wrote in a comment to this post:

This page has some good answers. It is about tech conferences, but the basic problem of getting lots of people on wifi in a single space is covered by the solutions.

I fully agree with Hamish, the page has excellent advice from people who have actually done this, unlike me.

Finally, Mark Tinka replied:

I generally wouldn’t recommend vendors on a public mailing list in such variable matters as wireless deployments, but given the scale you’re considering, Aruba came to see me once (uninvited, as usual), and they seemed to have some rather interesting things to say re: their wireless product portfolio, with particular regard to large scale installations.

You might want to add them to your shopping list, but my guess is the price point is way-up-there, what with their controllers and all.

But be careful about “buying” everything they tell you (same goes for other vendors). As others have mentioned, binding assurances from them as well as PoC’s (proof of concept) before you sign would be great!

I hope this helps someone. Please let us know how you get on.