View low bandwidth version

Archive for the ‘bandwidth’ Category

Mobiles for Scientific Research

Friday, July 9th, 2010

We know mobiles are very useful in areas where desktop computer and communications infrastructure is not easily available or affordable. And we’re very interested in mobile applications and scientific research in exactly these regions.

So I was very interested to see a new training workshop being run by the Science Dissemination Unit (SDU) of the Abdus Salam International Centre for Theoretical Physics (ICTP). The workshop is on Mobile Science: Sensing, Computing and Dissemination and the deadline for applications is tomorrow, July 10th.

Quoting from the announcement:

The Science Dissemination Unit (SDU) of the Abdus Salam International Centre for Theoretical Physics (ICTP), with the assistance of the University of Washington (USA) and of the UCLA Centerfor Embedded Networked Sensing (USA) will hold a Workshop on “Mobile Science: Sensing, Computing and Dissemination” in Trieste (Italy) from the 2 to the 5 of November 2010.

Mobile applications offer tremendous benefits to academic research and
education, and to society as a whole throughout the world. This is an
opportunity that deserves attention and promotion, especially in less
developed areas where mobile phones are the first telecommunications
technology in history to have more users than in the developed world.

The specific things that interested me were:

The Mobile Science workshop aims to engage the scientific community in developing countries in the design, development, and deployment of the newest mobile scientific applications;
i.e. advocating appropriate mobile applications in scientific
research/academia;
Participants will learn how to apply mobile technology tools to retrieve scientific data
I.e. designing mobile apps for science data collection;
how to apply appropriate web-based analysis to assimilate mobile data into scientific studies
I.e. web-based statistical analysis and presentation, like a free online version of SPSS? As far as I know this doesn’t exist yet. The closest that I can think of is the Google Docs spreadsheet, which is of course just a spreadsheet, requires an internet connection and doesn’t allow plugins for additional scientific analysis functionality. But there could be a very interesting app to develop here.
and how to share their scientific findings with a potentially large mobile audience.
I.e. low bandwidth design with an emphasis on web standards for cross-platform compatibility, so that it works on the largest number of mobile devices.

If you want to apply, better get on your bike (or modem?) because the deadline is tomorrow. If you want to do mobile scientific research applications, please get in touch, we’d like to help you.

Tags: , , , , ,
Posted in Mobiles, Offline, Open Source, PDAs, Web, agile, bandwidth, data collection | No Comments »

The (ongoing) need for speed

Monday, June 21st, 2010
Jakob Nielsen

Jakob Nielsen

13 years ago Jakob Nielsen wrote an important article stating that one of the most significant factors in web usability is speed.

In the work that we do, designing web applications that are used in developing countries, we have taken this advice very much to heart.

13 years later Jakob Nielsen has felt the need to write a new version of that article again. And I am glad he has! Despite the roll out of broadband web authors are still creating sites that are slow, although for different reasons, according to Jakob.

In the original article Jakob said that large images were the main culprit in causing slow web pages. Now he says, with the advent of broadband, large images are not the main problem.

Interestingly, with the sites we look at and the connection speeds we deal with, large images still are one of the main contributing factors to slow sites.

Jakob now lays the blame on too many fancy widgets.

I would agree with Jakob here. In my experience the size of javascript is now rivalling that of the large images for the sites we’re interested in.

The research into user interface response times is as true now as it was back in 1968 when it was done. From Nielsen’s article, remember these times:

  • 0.1 seconds gives the feeling of instantaneous response — that is, the outcome feels like it was caused by the user, not the computer.
  • 1 second keeps the user’s flow of thought seamless. Users can sense a delay, and thus know the computer is generating the outcome, but they still feel in control of the overall experience and that they’re moving freely rather than waiting on the computer.
  • 10 seconds keeps the user’s attention. From 1–10 seconds, users definitely feel at the mercy of the computer and wish it was faster, but they can handle it. After 10 seconds, they start thinking about other things, making it harder to get their brains back on track once the computer finally does respond.

A 10-second delay will often make users leave a site immediately.

Now consider the implications of these times in conjunction with your users’ connection speed, particularly if they happen to be in the developing world.

(see also our web design guidelines for low bandwidth connections.)

Tags: , , ,
Posted in Appropriate Technology, Design and Usability, bandwidth | No Comments »

Simulating low bandwidth: Publishers for Development

Tuesday, June 8th, 2010

We think that academic publishing is an area that’s both critically important to development, and simultaneously becoming more and more inaccessible to the people who need it most.

The average size of web pages has been growing much faster than the average speed of connections in developing countries, and journal websites are no exception, as you can see in Alan’s blog post:

Average page size has grown much faster than available bandwidth

Average Page Size vs Bandwidth

As Alan points out, the average journal’s home page in his sample would take over 90 seconds to load on average, for researchers at universities in developing countries. Usability research has shown that people expect a computer to respond within 30 seconds. Making them wait longer interrupts their concentration, causes dissatisfaction and annoyance, and they often abandon the process. The biggest factor in user satisfaction is speed of response.

While this research probably did not include users who are accustomed to slow and unreliable computers, I think it’s safe to say that most people would find it annoying and difficult to use the Internet on a dial-up modem. And even a modem would have been preferable to some of the Internet connections that I’ve experienced (and paid for) in some countries in the last few years.

Academics have little ability to persuade their universities to upgrade their internet connections, at a cost of several peoples’ salaries (several thousand dollars a month). The only people who can change this are the publishers of the journals, by optimising their journals’ websites for users with slower connections.

But how to persuade the publishers that this is important? We built a low bandwidth simulator ourselves, and took it to Oxford, to INASP and the ACU’s Publishers for Development conference.

What We Did

We set up spare machine as a bandwidth management box, and used it as a network filter for the participants. They could come and plug their laptops into the box, and browse the Internet and their own websites at a simulated slow speed.

Table with server, router and laptops with exercise cards stuck on top

Exercise Table

We configured the box for transparent bridging. This allowed us to insert and remove it from the network easily, just by switching over a network cable, to demonstrate the difference between fast and slow loading of pages.

We gave the participants at the meeting tasks to perform on various publishers’ websites, for example finding and downloading an academic paper by topic or researcher.

Participants watching and using the throttled laptops

Playing the Game

I think they found the activities enlightening, because we had some very good comments from some of the participants:

  • “We’re so pleased that Alan was able to work his magic at the recent PfD session – his delivery is innovative, dynamic and fact-packed so it really sparks enthusiasm from the audience… [which] is demonstrably channelled into action once people return to their places of work.“
    Publishers for Development Team
  • “It was really useful to try the low bandwidth! [Our site] is already considered fast but it made us think even more around this issue, what else can we do etc.“
    Anonymous Participant
  • “Alan Jackson’s information about bandwidth was kind of shocking even if I knew it before, but to really experience it was very valuable. We are going to redesign DOAJ’s home page and this must be the starting point.“
    Sonja Brage, DOAJ
  • “Site speed is a major consideration for us, and I really enjoyed Alan/Aptivate’s session, experiencing the exasperation of trying (and failing) to connect via low-bandwidth… I have a feeling that there is ‘excess baggage’ on a number of the pages…“
    James Kitchen, OECD

How We Did It

We used FreeBSD as the operating system for the software bridge, because its dummynet traffic shaper is relatively easy to use, and very good at simulating slow connections.

We wanted to use a laptop instead of a desktop machine, so that we could carry it to the conference easily, but we had hardware compatibility issues with FreeBSD on all the laptops we had available to us (mostly IBM Thinkpads). We ended up using a compact Fujitsu desktop box.

We installed FreeBSD 8 on it, and configured it to transparently bridge between two interfaces. Our internet access at the conference would be wireless, but we had issues with bridging wired and wireless interfaces together. So instead we used a Linksys WRT-54GL router with the Tomato firmware, which enables wireless client mode, to connect to the network:

WRT-54GL connected to FreeBSD throttler connected to network switch connected to client laptops

Throttler Network Diagram

And this is what it looked like in the room. Notice the essential coffee and cupcake, without which the system mysteriously failed to work:

FreeBSD server, wireless router and a laptop

Network Close Up

We configured the FreeBSD box to bring up the bridge automatically at boot time, and to load a set of ipfw firewall rules to enable dummynet, the traffic shaper. On this box, the ethernet interfaces are called em0 and rl0, so we added the following lines to /etc/rc.conf:

ifconfig_em0="up"
ifconfig_rl0="up"
cloned_interfaces="bridge0"
ifconfig_bridge0="addm em0 addm rl0 up dhcp"

firewall_enable="YES"
firewall_type="/etc/ipfw.rules"
dummynet_enable="YES"

Then we created /etc/ipfw.rules with the following contents:

# with bridge mode, two nics. em0 is wan
add pipe 1 all from any to any out recv em0
add pipe 2 all from any to any out xmit em0
add allow all from any to any
pipe 1 config delay 700ms bw 40Kbit/s mask dst-ip 0x000000ff
pipe 2 config delay 700ms bw 40Kbit/s mask src-ip 0x000000ff

This configuration creates two dummynet pipes. Pipe 1 is for traffic received on the external interface (downloads), and pipe 2 is for traffic being sent out of the external interface (uploads). We have to follow this by a rule which allows all other traffic, otherwise local traffic (on the box itself) is denied by default when the firewall is enabled, which breaks local DNS and inbound SSH and makes the box pretty unusable on the console.

Then we configure both pipes to allocate 40 Kbps (kilobits per second) for each individual IP address in the private subnet (allocated by the DHCP server on the Tomato router) and a 700 ms delay in each direction, which gives a 1400 ms round trip time. This is somewhat higher than the expected 600 ms round trip for a connection by geostationary satellite.

The end result is that each user connects a laptop to the switch behind the box, gets an IP address from the DHCP server on the router, is NATted by the router onto the public network, and is able to browse the Internet with a connection of 40 kbps upload and download. If you remove the FreeBSD box, by connecting the switch directly to the router, you can access the public network at full speed.

One issue was that the public network used a captive portal, which we had to log into. We didn’t want each client on our network to have to log in separately, so we enabled NAT on the router, and in wireless client mode, all the NATted clients get the MAC address of the router, so the public network thinks that they’re all the same PC and doesn’t ask them to log in again.

Why We Did it

We think that members of universities and research institutions need to be able to join and participate in the global research community as equals, in order to play their part in assisting development in their home countries.

Programmes such as PERii, HINARI and AGORA negotiate free or discounted online access to these journals for universities in developing countries. But the users still need to get online and access the content.

Online publishing for Western markets is usually designed for users with fast Internet connections, which Western universities have. But in other regions, universities often can’t afford fast connections, and this makes it very difficult for them to access these journals online.

Publishers for Development is bringing international publishers together who are interested in finding out how they might contribute to discourse and action around developing country access, encourage publication from developing country researchers and understand the diversity within research cultures/communities and the challenges these present.

Tags: , , , , , , ,
Posted in Design and Usability, Engineer's Log, Internet, Open Source, Web, bandwidth | 3 Comments »

The Censorship Arms Race

Wednesday, April 7th, 2010

Preface: This post discusses censorship. I want to be clear that I represent only my own personal views here, and I don’t personally support censorship in most cases. I think that freedom of access to information has a benefit and a cost, and the tradeoff depends on circumstances.

I think that censorship is useful when it serves a higher purpose, for example to save lives, or to save vital money for underfunded universities in countries where bandwidth is expensive and there are alternative ways for students to access the uncensored Internet for private browsing purposes. I’m opposed to censorship that requires leaving the country or changing your ISP to get around it.

Walubengo wrote on the BMO Training mailing list:

Am just from the student labs and came across this sneaky little [software]:

http://www.ninjacloak.com/

It basically allows my students to get behind the good old
dansguardian/squid proxy_firewall; essentially allowing them to visit
and download all and sundry (read porn, warez, torrents et al)

[H]ave been wondering why the clamour to “open-up” the internet “for
research” had gone down (now I know).

Any quick counters? (beyond just blocking ninjacloak.com, since they are likely to get an equivalent sooner rather than later)

I have never used ninjacloak and I don’t intend to, but I’m sure that if you post some logs of its use from your proxy server, we can figure out how to block it.

However, no security is perfect. There will always be ways around any security measure that we implement. However, no workaround is perfect either. Once we understand how it works, e.g. what the requests that it makes look like, we can block it.

This quickly turns into an arms race between the user and the administrator. The winner is usually the one with the most time, patience and determination. This may be a fight that you don’t want to take on.

In my view, if users really really want to access some blocked content, they will find a way. However, a good security system will make it possible to at least trace that they did so, if not exactly what they accessed. So my approach would be two-fold:

  1. Tackle the biggest problems first, and when they make sense. If someone uses ninjacloak to view a porn site once, it is hardly going to bring down your network, so you don’t need to care. If all your students are using TOR, AND it is bringing down your network, THEN it’s time to do something about it. If you don’t know what the biggest problem is, find out.
  2. Don’t forget that social measures are far more effective than technical ones. If students know that they are being watched, they are much less likely to try things like this. Make REALLY sure that everyone knows and understands your policy. When you find students bypassing your security, go and talk to them. If necessary, consider the use of formal sanctions, which are likely to have a stronger deterrent effect.

If users think they are being treated unfairly or harshly, it can increase their determination to fight the system. If you have a good reason for censoring, because you can show them how much damage their actions are causing to legitimate or intended uses (such as academic research), they are much more likely to understand and comply with your requests, hopefully avoiding the need for sanctions.

nb: but again, someone may ask, why not just open up the internet any way?

Because (and only when) it wastes your precious bandwidth that’s better used for your core purpose (e.g. academic research), which is why you pay for the connection in the first place.

Tags: , , , , , ,
Posted in Education, Infrastructure, Internet, Web, bandwidth | 2 Comments »

Network Management Basics

Wednesday, April 7th, 2010

I’ve been asked for some advice on how schools and universities can take advantage of the increased bandwidth available with the arrival of the TEAMS and EASSY submarine cables in East Africa.

Management of Internet connections is a big subject. Whole books have been written about it, including the freely downloadable How To Accelerate Your Internet (BMO Book). However, for anyone who doesn’t have time to read it, I will briefly summarise the most important points that I can think of:

  • have a clear, simple and strict Internet access policy, and enforce
    it.
  • have enough bandwidth, AT LEAST 3 kbps per computer, uncontended. So if you have 1000 computers, you should have 3 MBits dedicated bandwidth, or 60 MBps if it’s shared or contended with a 20:1 contention ratio (typical ISPs).
  • have competent network administrators. If you don’t have them, then hire or train them.
  • implement good network management practices, e.g. by following the advice of the BMO Book.
  • start by solving the problems that users complain most about, to give them the best possible service.
  • monitor your network to understand how Internet bandwidth is being used.
  • block misuses of Internet access that are causing problems for legitimate use of the Internet connection.
  • ensure that client PCs have good, fast antivirus, perform well, are
    regularly reformatted and reimaged, and have strong local security to prevent unauthorized software installation.

Far more information on all of these topics can be found in the BMO book. I suggest starting with the Introduction if you’re interested.

Tags: , , , , ,
Posted in Censorship, Education, Infrastructure, Internet, Software, Web, bandwidth, security | 2 Comments »

pmGraph – Bandwidth Monitoring for Networks

Saturday, February 20th, 2010
pmGraph video screencap

Video introducing pmGraph hosted by Vimeo

pmGraph is a free tool we produce to help administrators monitor bandwidth on networks.

Read more about it or watch the video above.

Many thanks to Mark for putting the video together.

Tags: ,
Posted in Appropriate Technology, Internet, Linux, Software, bandwidth | No Comments »

Lopad: high speed collaborative text editing over low bandwidths

Saturday, January 30th, 2010

Video of lopad team at #crisiscampLDN

Note: lopad is still a work in progress. The #crisiscampLDN team have made good progress, there will be a first cut up in the next few days. Getting it to the optimised, low bandwidth tool it needs to be is a project for the next few weeks.

I’m in “the ball room” (the other one’s called “the tree room” in case you’re wondering where the heck we are) at #crisiscampLDN and I’m thoroughly impressed by the efforts of @nickstreet and @mrchrisadams from Headshift.

What are they doing? Something really cool – making the first release of lopad. Check out the intro on blip.tv from @leashless above.

What is lopad?

Aptivate makes a service called loband – it takes high-bandwidth webpages and makes them work quickly on slow connections.

There’s a demand (e.g. from staff at UNFAO and users in Rwanda) for a related service, tuned for low bandwidth environments, that lets users easily collaborate over a text file.

This is the first addition to the lo* (pronounced lo-star) family of products: lopad.

Think of it as a super-lightweight Google Docs with no registration, instant visibilty of other user’s changes and fast performane on all internet connections.

There’s already a product called Etherpad which has been open-sourced since Google bought the company (to absorb the team into Google Wave). They plan to discontinue the public service in a few months time. In response to this, there are already some other public instances of Etherpad (e.g. PiratePad) as well as many private ones, but the immediate goal of the lopad project is to:

  • Create lopad.org – a free public instance of etherpad, promoted for use in the relief and development sectors but open to anyone
  • Optimise lopad.org to perform well on low bandwidth and/or high latency connections

Some of the things we’re going to be doing along the way will be getting a production server up and running, getting a copy of etherpad on there, rebranding it to “lopad”, analysing the opportunities for improving the bandwidth performance of the system and then implementing them.

In true meta-style, there’s more info and developer notes online at: http://etherpad.com/crisiscampLDN

Some highlights from the above:

I also think this will be a valuable tool for use by people in the Crisis Camp effort.

So: friends, coders, countrymen, come and join in!

Tags:
Posted in Crisis Camp, Internet, Web, bandwidth | No Comments »

Simulating low bandwidths: how to make sure your apps work in the field

Saturday, January 23rd, 2010

I’m going to write about four ways to simulate a slow internet connection and a bit of background about why you’d want to do it. Simulation is great but I’ll say this now: there’s no substitute for testing stuff in the field. However, before you release to the team on the ground or grab your bag and hop on a plane, read this.

Why simulate low bandwidths?

Aptivate builds online software for people in the international development sector. Our users are in places where internet connections are slow and unreliable – we need to make sure our stuff works for them.

More people are accessing the web using mobile handsets or mobile internet connections (3G dongles and tethered phones). The bandwidth, latency and stability characteristics of these links are very different to the “always on broadband” that most developers target.

Finally, we’re involved with CrisisCamp London over the next few weeks – part of an international effort by dedicated volunteers to provide remote technical support and build tools for individuals and organisations working in Haiti. What they build needs to work in that environment – hopefully this post is going to be useful.

So, if you think your technology is going to be used in the scenarios above, read on. If you’re just awesome and like to know about this stuff, read on too…

Four ways to simulate a slow connection

  1. Use a profiler, your brain and some common sense (easiest)
  2. Use Aptivate’s online low bandwidth simulator (easy)
  3. Use Sloppy a desktop Java app that simulates slow web links (pretty easy)
  4. Get a machine (with maybe two network interfaces) and do some IP traffic shaping (best results, not easy)

There are probably a few more but I’m on a plane from Rome to London to go the the first CrisisCamp and can’t think properly!

1. A profiler + simulate it in your brain.

This is particularly useful if you’re developing for the web.

YSlow Running on Reliefweb.int

There are a few online and in-browser tools that give you a breakdown of the resources your website is using.

Three important things to get from these tools is:

  1. What’s the bandwidth usage of each of my pages and typical interactions?
  2. How many individual requests are required for each page?
  3. How much content is cached?

The overall bandwidth is a feasibility test. We say aim for 25k per page, but use your own judgement – how fast is your user’s connection, how long will it take for them to get to something useful (hint – if it’s longer than 5-10 seconds: #FAIL)

The number of requests also gives you an indication about performance over high latency or intermittent connections – in short, use fewer objects and cache them when you can.

Finally, if you’ve got a network usage meter (I have a noddy one running that comes with iStatMenus on the Mac ) you can get a rough idea of how much bandwidth an app is consuming (should work fine even if you’re developing an app in a mobile simulator). I’ve seen stuff for Windows I can’t remember, on Linux you could use BWM or get fancy with logging modes in IPTables – Google for more.

That’s it.

2. Use Aptivate’s online Low Bandwidth Simulator

Aptivates Low Bandwidth Simulator

This technique is only useful if your site is accessible from a public URL. It only simulates bandwidth, not latency or packet loss.

We make Loband, and online service that strips the junk out of webpages and gives you a compressed, simplified version that works better on slow links.

As part of the Loband code, there’s a simulator which you can access here.

You plug in the URL of your site, select the bandwidth you want to simulate and hit go. I haven’t tested it recently with any serious AJAX/HTML5/Flex/Flash stuff so your mileage may vary if you make heavy use of these tools.

Do what a user would do with your app and see if it’s usable.

That’s it.

3. Use the Sloppy desktop Java app

Sloppy Java desktop bandwidth simulator

This technique is great if your site is running on a local dev box or even if it’s online. It only simulates bandwidth, not latency or packet loss.

Make sure Java lives on your machine. Download Sloppy. Run it, start it, point it at your app.

Do what you would have done with 2.

That’s it.

4. Get a machine, (maybe two network cards) do IP traffic shaping.

This technique is the best of the bunch: you can simulate bandwidth, latency and packet loss and do so for anything running on your machine or LAN. That’s anything: browser apps, mail clients, Skype, mobile simulators etc. It’s not hard but is a little fiddly. There are two broad ways you could do this: for yourself on a single machine or, for a bunch of people on a LAN.

Terminal Showing iperf measuring different bandwidths throttled by dummynet

iperf showing a dummynet throttled link

Quickly,  to do it for yourself, on your own machine to do app testing: if you’re running FreeBSD / MacOSX,, follow Bjørn Hansen’s tutorial.

It gets a bit trickier if you want to do it for several machines at once.

What we’re trying to do is turn a machine with two network interfaces (NICs) into a “router”. Traffic goes in/out of the first interface at normal speeds, but the traffic goes in/out of the second interface at user-selected levels of crapness (bandwidth, latency, packet loss)

Relatively speaking: this is easy on a Mac / BSD box, trickier on Linux and hard on Windows. While most laptops actually have 2 network interfaces (wifi + ethernet) – I normally do this with a desktop that’s got 2 NICs  or a laptop + a USB / CardBus/PCMCIA NIC.

On a Mac/BSD you’re going to be using ipfw to control the dummnynet traffic shaper. Man up to find out more. In short: ipfw’s a firewall that classifies packets (e.g. by which port or IP they’re going to) into “flows”. Dummynet takes a flow and sticks it in a “pipe”. A pipe emulates a link with given bandwidth, propagation delay, queue size and packet loss rate.

….how on earth do we get this working?

There are better tutorials than I can write quickly here and  here. But in brief:

  1. Get a BSD machine with dummynet (OSX 10.4+ is enabled by default, might need a kernel rebuild for FreeBSD) running with 2 NICs. Fire up a terminal, type in ifconfig and make sure you can see the two interfaces (en0 and en2 for me)
  2. Make sure you can route packets between interfaces.
  3. Make a pipe for the traffic between interfaces
  4. Configure your pipe, stick your traffic in there and smoke it.
  5. Tweak the pipe and simulate to you heart’s content.

In reality, this always takes me half an hour to get right – I’ve never had this go smoothly first time.

First things I check if it’s not working:

  • Is OSX / BSD doing some daft routing / automatic internet connection sharing that messing with your ipfw settings?
  • Are you routing using the right interfaces? I’ve actually got 7 network interfaces that show up in ifconfig to choose from (firewire, bt, vm, wifi, ethernet etc.)
  • bit/s and Byte/s are quite different…
  • Don’t despair, it will work, there’s pictures of me doing it here. :-)

Typical bandwidth / latency / loss scenarios

The key commands you’ll be running to set parameters will look like:

ipfw pipe 1 config bw 50Kbit
ipfw pipe 1 config delay 200ms
ipfw pipe 1 config plr 0.2

The three variables you have to play with are bw (bandwidth) plr (random packet loss rate) and delay (latency). Here’s a super-rough guestimate for some typical scenarios, please advise if I’m way out or there are other common scenarios:

Scenario Bw (Kbit) delay (ms) pr (ratio)
2.5G mobile (GPRS) 50 200 0.2
3G mobile 1000 200 0.2
VSAT 5000 500 0.2
Busy LAN on VSAT 300 500 0.4

What about windows and linux?!

I promise to update this bit with more info when I’ve got Linux, Windows boxes and Chris to hand.

In short though: with linux it’s the same idea, machine with 2 NICS, get them routing, use IPTables and the linux traffic shaper, tc. It’s not as good as dummynet (no packet loss IIRC) but gets the job done. For Windows, I’d honestly have to do some more research, last time I tried it, I just pulled out my Macbook.

Please add any tips and corrections of  below!

Tariq

Tags: , , , ,
Posted in Appropriate Technology, bandwidth | 12 Comments »

When it comes to websites… small IS beautiful

Thursday, July 9th, 2009

There are two reasons why you should make your websites as small as possible. By small I mean minimising the size of data your user must download to see your web pages.

The first reason is usability. Time and again it has been shown that users like speedy websites. Google and Amazon have recently found even a delay of half a second can mean a 20% drop in users. Obviously your site must provide what your audience is looking for, and it must make it easy to find, but the number one factor that contributes to a positive user experience is speed. Ideally you want your pages to load within 1 second. They must load within 10 seconds; research shows consistently that visitors will leave a site if it doesn’t load in 10 seconds or less, the fewer seconds it takes to load the more engaged a visitor will be. Even with the ever increasing connection speeds of broadband we are seeing in the UK, if you’re not careful, it’s still perfectly possible to make sites that are too slow.

The second reason – the reason that most interests me and Aptivate, the organisation I work for – is global accessibility. Like us, you may feel we have a moral duty to ensure important information is accessible in the developing world or you may see the developing world as an interesting emerging market. Either way, if you want your content to be accessible in the developing world you need to seriously consider the size of your web pages. Aptivate, has been focussing on this issue from the perspective of users in less developed countries. We’ve found that the majority of information is inaccessible; even information that is intended to be used by this audience. The fact is that the developing world is years behind the broadband revolution we are witnessing in the “global North”.

bandwidth vs page size

bandwidth vs page size

Not only that, but as more bandwidth becomes available in developing countries it is matched by increasing demand. We foresee that bandwidth will remain much lower in developing countries than in wealthy ones for some time to come. This must be considered when designing for a global audience.

Over the past 5 years the average web page has increased by 300%. Meanwhile, in developing country universities, we estimate the bandwidth available to an individual user will have increased by 20 – 60% – and this is from a very low starting point. Bandwidth is increasing slowly for developing country universities whilst bandwidth demands from their users and from websites, document downloads and on-line applications are increasing rapidly.

It CAN look good

When I talk to people about low-bandwidth friendly websites the first concern is that they would be somehow sub-standard. We must dispel the myth that low-bandwidth websites are boring and ugly. This is simply untrue.

Let’s make an analogy with building a house. If you wanted to build an energy efficient house would it have to be ugly? No. You may need to spend a bit more effort designing it in the beginning. The construction costs are nearly the same and there is no reason, other than the lack of imagination of your architect, that your house cannot be beautiful. And so it is with websites. The requirements to be small, fast, usable and globally accessible are just additional parameters for your designers. These additional requirements will be of negligible additional cost and yet will transform the user experience of all your users. Your designers are likely to produce a website that looks clean, clear and concise – all qualities that users have been found to prefer. If your main market is in the global North your users will benefit from a fast response which is the main contribution to their satisfaction in using your site. If your audience is in the developing world, designing for low-bandwidth will make the difference between them being able to see your website and not.

Small, fast, responsive web pages are good practice and are globally accessible. This is a win-win situation. The big players like Google and Amazon understand this. Others have not yet got the message.

Developing country universities

In 2008 Aptivate estimated that the bandwidth available to individual university students and researchers in low income and developing countries (for example, in most of Africa, parts of Latin America and South Asia) is 20 kb/s – which is about 1/100th the speed of a broadband connection to a typical UK home. While bandwidth will have increased since then it is still going to be about a factor of a 100 slower than the average domestic UK connection which is now over 3000 kb/s (3mb/s).

Recently I did a survey of 27 publishers’ websites. This was not an in-depth study just a quick temperature check but I think the results are still interesting. I chose the 27 publishers from the sponsors of a major conference. I “googled” each publisher then measured the size of the first page I got to from the Google search results, usually the publishers’ home page. The average page size was 250 kB which is not far off the current global average page size. However the largest was 800 kB while the smallest was 20 kB.

What does this mean for users in developing country universities? The average web page from this sample would take over a minute and a half to load. The table below shows the various page load times with times over 10 seconds high-lighted.

page load times in seconds

Connection Speed

Developing University

(20 kb/s)

Dial-Up

(56 kb/s)

UK Broadband

(3000 kb/s)

Page

Size

smallest (20 kB)

8

3

0.1

average (250 kB)

100

36

0.7

largest (800 kB)

320

114

2.1

These figures should be read as minimum download times. There are other factors besides bandwidth that effect download times like the complexity of the website. I find it’s pretty rare even in the UK to see pages loading in less than a second.

PDFs

If you’re a publisher it’s likely that you publish your articles as PDF files. In which case you may be asking yourself what’s the relevance of all this talk about web page optimisation. Firstly it should be noted that a lot of what I’ve said about web pages is true of PDF files as well. It is possible through bad formatting options to make PDFs unnecessarily large. PDF files can be optimised for printing which will make them higher quality but much larger. Alternatively they can be formatted for screen reading in which case they are a lot smaller. If you’re using a computer to read PDF optimised for screen reading you wouldn’t normally notice the difference… except in the amount of time you would have to wait for it. Giving the user a choice between these formats can help those with slow connections.

A year ago we did a small survey of PDF files from scientific journals. We found that most of the time these were well optimised. They were still large but this is because they contain a lot of information – graphs, charts, equations etc. When working with African university researchers we found that the large size of PDF files was not the biggest problem. The articles themselves represent high value content. Even if they take several hours to download (which, in some cases they did) this could still be tolerated by the user. They found ways of adapting to this for instance by doing other work while the article downloads or, in the rare cases where the power is left on, downloading the files overnight.

The real problem was the path the user had to follow to get access to the PDF article. While the PDF files represent valuable content for the user, the many web pages the user must navigate to gain access to the PDF usually represent little value. It’s important that this path is as direct as possible. We must be careful not to let too much branding or gadgetry thwart the user in their goal. While an African researcher may be prepared to start a PDF download that will take a long time they should not be expected to navigate through a dozen pages each of which may take several minutes to load. It is this kind of frustrating experience that will drive users from your site.

The causes

What makes web pages so big? Isn’t it the features that our users demand? Most of the time I don’t think it is. It’s just wastage and bloat.

When I get introduced to a new organisation I often have a look at their website and measure how big it is. If I have some spare time I like to see how hard it would be to halve the size of their home-page. This usually takes between 10 minutes and half and hour with little discernible difference to the user.

The most frequent culprits and the easiest to fix are the images. In many cases it would be better to change the design to rely less on large images. Even without changing the design large savings can be made simply by optimising the format of the images.

Next it’s worth trying to optimise the code. The HTML and CSS files that make up websites can be full of “comments”, white-space, unused sections and other unnecessary bits and pieces. It’s often straightforward to remove the wastage.

Another area of bloat is the JavaScript – chunks of code that are part of many websites and run on the users machine. Optimising the JavaScript can be easy or can be hard.

Sometimes the JavaScript just isn’t needed. For instance when it’s used for styling tricks which can now be done in more efficient ways.

Sometimes there’s lots of it that just isn’t used. There are JavaScript “library” files that contain many functions. A site may include a large library file but may only call one or two functions in it.

Sometimes the JavaScript comes as part of the Content Management System (CMS) the site uses. In this case it can be a bit trickier to sort out but still possible.

Things to do

If you’re interested in making your site faster and more globally accessible here are some ideas that might interest you.

The first step is find out how big your pages are. Tools like PingDom will measure the size of your pages. Tools like Google’s PageSpeed and Yahoo’s YSlow[5] will even make suggestion of what you can fix.

We have written on-line guidelines for designing website for global accessibility. We discuss the reasons why designing for low-bandwidth is a good idea and give concrete guidance on how to do that. We also list tools like YSlow and various automatic optimisers. You can see our Top Ten design guidelines here:

http://www.aptivate.org/webguidelines/TopTen.html

On the 11th of September (2009) we will be speaking at the ALPSP conference in Oxfordshire. We are also going to be running short “Halve Your Home Page” workshops – a hands-on session where we show you how shrink the size of your own site (email info@aptivate.org for details).

Tags:
Posted in Design and Usability, bandwidth | 5 Comments »

Low Bandwidth Web: Opera Turbo

Tuesday, June 9th, 2009

Aptivate (then Aidworld) was founded in 2003 by a group of techies and aidworkers wrestling with the question: how can you make the web usable for relief workers in the field?

Opera Turbo in Action

Opera Turbo in Action

The problem then was access to bandwidth and the cost of that access.

Typical satellite phone connection speeds were 9.6Kbps (think of cold treacle flowing uphill or the state of dial-up in the early 90s) and the cost would be anywhere from $2 to $20 per minute.

5 minutes to download something like cnn.com made it unusable and $100 for the privilege made it unaffordable.

We came up with loband – a free online service that simplifies web pages. It downloads them remotely, trims them down and  returns them to the user in a lightweight format. It can offer a 5-10x reduction in bandwidth used.

Fast forward 6 years and it’s interesting how similar the story is.

Some of us now have  fast desktop & mobile web connectivity, but websites have gotten heavier (the first page I get to on Facebook is 1.25MB…)  and we don’t always have access to our quick connections.

The fundamental issue is still there: the web can be slow and expensive if you’re not on a fast “unlimited data” connection.

Opera have been doing great things with their mobile browser for some time. They recently introduced the Opera Turbo feature into their desktop edition. The concept is similar to loband but its designed to integrate transparently into the browser.

Opera route all relevant traffic via their servers and return a compressed stream of data to the browser containing the content you want. From the picture above, you can see that they compress graphics to save bandwidth.

One thing I suspect they do (although I haven’t checked) is reduce the overall number of requests between the browser and the server. Going back to Facebook – it takes 92 HTTP requests to build my home page. That becomes painful if you’re on a low bandwidth, high latency connection. You effectively incur an overhead for each of those 92 requests.

If Opera can turn that into fewer, overall smaller requests – the Norwegians rock even more than I think they already do.

Tags: , ,
Posted in Appropriate Technology, Internet, Software, Web, bandwidth | 3 Comments »

« Older Entries