Video introducing pmGraph hosted by Vimeo
pmGraph is a free tool we produce to help administrators monitor bandwidth on networks.
Read more about it or watch the video above.
Many thanks to Mark for putting the video together.
Author Archive for tariq
pmGraph – Bandwidth Monitoring for Networks
Saturday, February 20th, 2010Lopad: high speed collaborative text editing over low bandwidths
Saturday, January 30th, 2010
Video of lopad team at #crisiscampLDN
Note: lopad is still a work in progress. The #crisiscampLDN team have made good progress, there will be a first cut up in the next few days. Getting it to the optimised, low bandwidth tool it needs to be is a project for the next few weeks.
I’m in “the ball room” (the other one’s called “the tree room” in case you’re wondering where the heck we are) at #crisiscampLDN and I’m thoroughly impressed by the efforts of @nickstreet and @mrchrisadams from Headshift.
What are they doing? Something really cool – making the first release of lopad. Check out the intro on blip.tv from @leashless above.
What is lopad?
Aptivate makes a service called loband – it takes high-bandwidth webpages and makes them work quickly on slow connections.
There’s a demand (e.g. from staff at UNFAO and users in Rwanda) for a related service, tuned for low bandwidth environments, that lets users easily collaborate over a text file.
This is the first addition to the lo* (pronounced lo-star) family of products: lopad.
Think of it as a super-lightweight Google Docs with no registration, instant visibilty of other user’s changes and fast performane on all internet connections.
There’s already a product called Etherpad which has been open-sourced since Google bought the company (to absorb the team into Google Wave). They plan to discontinue the public service in a few months time. In response to this, there are already some other public instances of Etherpad (e.g. PiratePad) as well as many private ones, but the immediate goal of the lopad project is to:
- Create lopad.org – a free public instance of etherpad, promoted for use in the relief and development sectors but open to anyone
- Optimise lopad.org to perform well on low bandwidth and/or high latency connections
Some of the things we’re going to be doing along the way will be getting a production server up and running, getting a copy of etherpad on there, rebranding it to “lopad”, analysing the opportunities for improving the bandwidth performance of the system and then implementing them.
In true meta-style, there’s more info and developer notes online at: http://etherpad.com/crisiscampLDN
Some highlights from the above:
- Nick forked etherpad to make lopad on github
- Pingdom tools is a good way to see how bandwidth is being used.
- How to (potentially) rebrand etherpad to lopad.
- How to set up an Etherpad server on CentOS minimal (where we’ll be hosting it)
- Simulating low bandwidths, how to make sure your apps work in the field.
I also think this will be a valuable tool for use by people in the Crisis Camp effort.
So: friends, coders, countrymen, come and join in!
Video: Introducing Crisis Camp London #1
Monday, January 25th, 2010
#crisiscampLDN team at work
On Saturday 23rd of January, Aptivate were out in force at Crisis Camp London which was kindly hosted by the London Knowledge Lab.
Björn had a chance to grab some footage on the day and put together a short video explaining what Crisis Camp is all about, what it’s like in London and what you might expect if you come along.
Crisis Camp London will be a regular event for the next few weeks, one which is going to grow in size and impact.
If you’d like to follow what’s going here are the links:
Ning: http://crisiscampldn.ning.com/
Twitter hashtag: #crisiscampldn
Twitter account: @crisiscampLDN
Crisis Camp London wiki: http://wiki.crisiscommons.org/wiki/Crisis_Camp_London
You can also click on the image above for a link you the video on YouTube.
Simulating low bandwidths: how to make sure your apps work in the field
Saturday, January 23rd, 2010I’m going to write about four ways to simulate a slow internet connection and a bit of background about why you’d want to do it. Simulation is great but I’ll say this now: there’s no substitute for testing stuff in the field. However, before you release to the team on the ground or grab your bag and hop on a plane, read this.
Why simulate low bandwidths?
Aptivate builds online software for people in the international development sector. Our users are in places where internet connections are slow and unreliable – we need to make sure our stuff works for them.
More people are accessing the web using mobile handsets or mobile internet connections (3G dongles and tethered phones). The bandwidth, latency and stability characteristics of these links are very different to the “always on broadband” that most developers target.
Finally, we’re involved with CrisisCamp London over the next few weeks – part of an international effort by dedicated volunteers to provide remote technical support and build tools for individuals and organisations working in Haiti. What they build needs to work in that environment – hopefully this post is going to be useful.
So, if you think your technology is going to be used in the scenarios above, read on. If you’re just awesome and like to know about this stuff, read on too…
Four ways to simulate a slow connection
- Use a profiler, your brain and some common sense (easiest)
- Use Aptivate’s online low bandwidth simulator (easy)
- Use Sloppy a desktop Java app that simulates slow web links (pretty easy)
- Get a machine (with maybe two network interfaces) and do some IP traffic shaping (best results, not easy)
There are probably a few more but I’m on a plane from Rome to London to go the the first CrisisCamp and can’t think properly!
1. A profiler + simulate it in your brain.
This is particularly useful if you’re developing for the web.
YSlow Running on Reliefweb.int
There are a few online and in-browser tools that give you a breakdown of the resources your website is using.
- Combination of Yahoo’s Yslow + Firebug for Firefox
- Safari or Google Chrome’s built-in (webkit based?) inspectors
- Pingdom tools online
Three important things to get from these tools is:
- What’s the bandwidth usage of each of my pages and typical interactions?
- How many individual requests are required for each page?
- How much content is cached?
The overall bandwidth is a feasibility test. We say aim for 25k per page, but use your own judgement – how fast is your user’s connection, how long will it take for them to get to something useful (hint – if it’s longer than 5-10 seconds: #FAIL)
The number of requests also gives you an indication about performance over high latency or intermittent connections – in short, use fewer objects and cache them when you can.
Finally, if you’ve got a network usage meter (I have a noddy one running that comes with iStatMenus on the Mac ) you can get a rough idea of how much bandwidth an app is consuming (should work fine even if you’re developing an app in a mobile simulator). I’ve seen stuff for Windows I can’t remember, on Linux you could use BWM or get fancy with logging modes in IPTables – Google for more.
That’s it.
2. Use Aptivate’s online Low Bandwidth Simulator
Aptivates Low Bandwidth Simulator
This technique is only useful if your site is accessible from a public URL. It only simulates bandwidth, not latency or packet loss.
We make Loband, and online service that strips the junk out of webpages and gives you a compressed, simplified version that works better on slow links.
As part of the Loband code, there’s a simulator which you can access here.
You plug in the URL of your site, select the bandwidth you want to simulate and hit go. I haven’t tested it recently with any serious AJAX/HTML5/Flex/Flash stuff so your mileage may vary if you make heavy use of these tools.
Do what a user would do with your app and see if it’s usable.
That’s it.
3. Use the Sloppy desktop Java app
Sloppy Java desktop bandwidth simulator
This technique is great if your site is running on a local dev box or even if it’s online. It only simulates bandwidth, not latency or packet loss.
Make sure Java lives on your machine. Download Sloppy. Run it, start it, point it at your app.
Do what you would have done with 2.
That’s it.
4. Get a machine, (maybe two network cards) do IP traffic shaping.
This technique is the best of the bunch: you can simulate bandwidth, latency and packet loss and do so for anything running on your machine or LAN. That’s anything: browser apps, mail clients, Skype, mobile simulators etc. It’s not hard but is a little fiddly. There are two broad ways you could do this: for yourself on a single machine or, for a bunch of people on a LAN.
iperf showing a dummynet throttled link
Quickly, to do it for yourself, on your own machine to do app testing: if you’re running FreeBSD / MacOSX,, follow Bjørn Hansen’s tutorial.
It gets a bit trickier if you want to do it for several machines at once.
What we’re trying to do is turn a machine with two network interfaces (NICs) into a “router”. Traffic goes in/out of the first interface at normal speeds, but the traffic goes in/out of the second interface at user-selected levels of crapness (bandwidth, latency, packet loss)
Relatively speaking: this is easy on a Mac / BSD box, trickier on Linux and hard on Windows. While most laptops actually have 2 network interfaces (wifi + ethernet) – I normally do this with a desktop that’s got 2 NICs or a laptop + a USB / CardBus/PCMCIA NIC.
On a Mac/BSD you’re going to be using ipfw to control the dummnynet traffic shaper. Man up to find out more. In short: ipfw’s a firewall that classifies packets (e.g. by which port or IP they’re going to) into “flows”. Dummynet takes a flow and sticks it in a “pipe”. A pipe emulates a link with given bandwidth, propagation delay, queue size and packet loss rate.
….how on earth do we get this working?
There are better tutorials than I can write quickly here and here. But in brief:
- Get a BSD machine with dummynet (OSX 10.4+ is enabled by default, might need a kernel rebuild for FreeBSD) running with 2 NICs. Fire up a terminal, type in ifconfig and make sure you can see the two interfaces (en0 and en2 for me)
- Make sure you can route packets between interfaces.
- Make a pipe for the traffic between interfaces
- Configure your pipe, stick your traffic in there and smoke it.
- Tweak the pipe and simulate to you heart’s content.
In reality, this always takes me half an hour to get right – I’ve never had this go smoothly first time.
First things I check if it’s not working:
- Is OSX / BSD doing some daft routing / automatic internet connection sharing that messing with your ipfw settings?
- Are you routing using the right interfaces? I’ve actually got 7 network interfaces that show up in ifconfig to choose from (firewire, bt, vm, wifi, ethernet etc.)
- bit/s and Byte/s are quite different…
- Don’t despair, it will work, there’s pictures of me doing it here.
Typical bandwidth / latency / loss scenarios
The key commands you’ll be running to set parameters will look like:
ipfw pipe 1 config bw 50Kbit
ipfw pipe 1 config delay 200ms
ipfw pipe 1 config plr 0.2
The three variables you have to play with are bw (bandwidth) plr (random packet loss rate) and delay (latency). Here’s a super-rough guestimate for some typical scenarios, please advise if I’m way out or there are other common scenarios:
| Scenario | Bw (Kbit) | delay (ms) | pr (ratio) |
| 2.5G mobile (GPRS) | 50 | 200 | 0.2 |
| 3G mobile | 1000 | 200 | 0.2 |
| VSAT | 5000 | 500 | 0.2 |
| Busy LAN on VSAT | 300 | 500 | 0.4 |
What about windows and linux?!
I promise to update this bit with more info when I’ve got Linux, Windows boxes and Chris to hand.
In short though: with linux it’s the same idea, machine with 2 NICS, get them routing, use IPTables and the linux traffic shaper, tc. It’s not as good as dummynet (no packet loss IIRC) but gets the job done. For Windows, I’d honestly have to do some more research, last time I tried it, I just pulled out my Macbook.
Please add any tips and corrections of below!
Tariq
Technology decisions in organisations great and small
Thursday, January 21st, 2010Ken Banks often writes about Social Mobile’s Long tail – it’s a really helpful concept; one that I find myself frequently using when explaining our work to others.
Social Mobile's Long Tail by Ken Banks
Whenever I see Ken’s picture, I’m reminded of the similar relationship between complexity and organisational size and I’m proud of how Aptivate works successfully across this spectrum. We try to bring the breadth of our knowledge, skills and experience to bear when working with everyone from communities in rural Zambia, to NGOs in the UK, international agencies in Europe and governments across Africa.
I think this is really important.
It’s great to be “the policy people” or “the community technology people” but you need people who span these worlds and can join the dots.
That’s us.
We’ve spent months in rural Zambia working with young women getting low-power computers, GPRS connections and mobile systems working to support local entrepreneurship. Now we have greater confidence offering advice on mobile monitoring and evaluation strategies for NGOs in the region, and in turn, to guide an international agency wanting to know what kind of policy monitoring is possible, and how data might integrate into their wider systems.
I had an enjoyable conversation yesterday with the folks at CAFOD who want to know if mobiles could strengthen their work at the local partner and international levels. I met them through BarCampAfricaUK last November and finally had a chance to catch up.
Personally, I’m really interested in working with medium-sized organisations trying to make better use of technology. I probably have similar conversations 2-3 times a month.
I think there are some common characteristics and challenges for these organisations:
- They already use some technology in the areas you’d expect: fundraising, communications, advocacy, admin and finance, and monitoring and evaluation.
- They don’t have much capacity to explore and understand how new technologies (e.g. mobiles, collaboration tools and media capture) or advances in current technologies (e.g. open standards, APIs, social media) can help their programmes.
- Local partners are already ahead of the game when it comes to the use of mobiles. This is typically out of necessity – even basic SMS is an astoundingly versatile medium.
- The “technology champions” in an organisation, the individuals who appreciate the possibilities, are not always the decision makers. They often don’t have the time to investigate these opportunities and present information around which decisions can be taken.
- Experimenting with the various tools out there can be challenging for the non-geek and it’s hard to find out about the realities of implementation.
- Consultants are expensive and companies who sell “off the shelf products” might not have the best interest of the organisation at heart.
- There are some great resources out there that catalogue technologies, there are also some good case studies that cover certain scenarios but there are few resources that specifically help people make decisions at the organisational level.
So here’s a promise: we’ll help you make decisions about technology. We’ll do a whole lot more, but at its simplest, we’ll do what it takes for you to decide what do to.
The first three things on my list of “how to support decisions” after my conversations yesterday are:
Write a blog post on technology decision making for medium-sized organisations, reassuring them that they’re in good company. (done)- Write a primer on “why use mobiles for data gathering and communication” with a goal to support decisions.
- Put together a “mobile gadget lab in a briefcase” to take to organisations so they can play with pre-configured versions of various tools on various devices supporting a couple of different workflows.
Do any of these thoughts resonate with you?
Comments most welcome!
Tariq
Agile Development and Retrospectives: Learning from failure?
Friday, December 11th, 2009I had an interesting chat with Alan last night about the role of “retrospectives” and it reminded me about the ICT4D Twitter Chat today around “Learning from Failure” being organised by the fine folks at Inveneo.
He was at the XPDay in London earlier this week – a 2-day hotbed of agile technology development geekery featuring a combination of traditional speaker sessions and open spaces.
Aptivate is a big advocate of using Agile methodologies. We see them as central to taking a participatory approach to international development. One thing we do after delivering a project is have a “debrief” or “retrospective” with the project stakeholders.
“Four key questions to focusing a community on learning and improvement” are described at www.retrospectives.com:
- What did we do well, that if we don’t discuss we might forget?
- What did we learn?
- What should we do differently next time?
- What still puzzles us?
Simple and sensible.
The key idea that Alan picked up on though was this: do a retrospective at the end of each iteration.
Every two weeks, every incremental release of a project, sit back, take an hour with your team and ask the above questions.
We’re certainly going to start doing this and from what was said at the XPDay – if you’re going to do one thing to improve your development process, do this.
Finally – ensure that all participants adhere to “Retrospective Prime Directive:”
Regardless of what we discover, we understand and truly believe that everyone did the best job they could, given what they knew at the time, their skills and abilities, the resources available, and the situation at hand.
Happy twitter chatting!
Translations, PDAs and Field Research
Wednesday, December 2nd, 2009Translation can be a real headache.
PDA used for interviews in Tanzania
Identifying text for translation, finding individual strings and phrases to avoid duplications, contextual exceptions, keeping track of them, revisions, collaborating remotely, reviewing, back-translating, integrating translations back into a finished product – you name it, the translation workflow has got it.
But first, a bit of background:
Aptivate started working with Camfed about 2 years ago when they were planning a major baseline study of their work supporting women’s education and empowerment in Africa. As part of their broader monitoring and evaluation work they wanted to understand the impact of their programme on areas such as attitudes towards girls education, awareness of HIV and sexual health issues and the effectiveness of community structures.
We trained young women from rural areas in Tanzania, Zambia and Zimbabwe to use PDAs for face-to-face interviews with teachers, students, parents and officials in the education system.
We used Palm Tungsten E2 PDAs and Solar Bags from Voltaic to run the exercise. We customised and bug-fixed a version of the excellent Episurveyor for use in the education context (it was designed as a health tool).
The surveys that Camfed created for the study (50-60 questions for 6 different stakeholders, many common questions) were designed in English and had to be available in the following languages:
- Swahili
- Shona
- Ndebele
- Bemba
- Lozi
The questionnaires were created in a spreadsheet – one sheet per stakeholder (e.g. parent, teacher) with a list of questions and optional responses on each sheet. We put together a tool in Excel to help with the translation process. Essentially it:
- Went around sheets indexing each cell with relevant text
- Built a single list of strings in a new sheet
- Presented only unique strings to a translator and locked the rest down
- Rebuilt the original surveys in the new language once the translation was completed
- Can repeat all the above to allow for back-translations too
This is a good example of the agile approach – do the simplest thing you can to get the job done well (and on a deadline!). The translations got done, we scripted the automatic translation of the EpiSurveyor survey files (which are XML objects) and that was, as they say, that.
Until I had a chat with Camfed yesterday and they asked – “you know that translation tool you made, can we use it for some other things we’re doing?”
Fantastic!
It’s great to have built a tool that starts to get useful beyond its original remit. The Excel tool we made isn’t suitable for general use yet and after using it for 2 years, there is plenty of scope for improvement around issues of collaboration and revision management.
Enter the internet.
I posted a question to MetaFilter yesterday on this subject and I got some really interesting responses I thought I’d share in case anybody is thinking of doing this kind of thing.
In particular, check out:
- The Open Translation Tools FLOSS Manual.
- The Globalsight open source translation management system
- The Pootle translation toolkit.
Happy translating!
Low Bandwidth Web: Opera Turbo
Tuesday, June 9th, 2009Aptivate (then Aidworld) was founded in 2003 by a group of techies and aidworkers wrestling with the question: how can you make the web usable for relief workers in the field?
Opera Turbo in Action
The problem then was access to bandwidth and the cost of that access.
Typical satellite phone connection speeds were 9.6Kbps (think of cold treacle flowing uphill or the state of dial-up in the early 90s) and the cost would be anywhere from $2 to $20 per minute.
5 minutes to download something like cnn.com made it unusable and $100 for the privilege made it unaffordable.
We came up with loband – a free online service that simplifies web pages. It downloads them remotely, trims them down and returns them to the user in a lightweight format. It can offer a 5-10x reduction in bandwidth used.
Fast forward 6 years and it’s interesting how similar the story is.
Some of us now have fast desktop & mobile web connectivity, but websites have gotten heavier (the first page I get to on Facebook is 1.25MB…) and we don’t always have access to our quick connections.
The fundamental issue is still there: the web can be slow and expensive if you’re not on a fast “unlimited data” connection.
Opera have been doing great things with their mobile browser for some time. They recently introduced the Opera Turbo feature into their desktop edition. The concept is similar to loband but its designed to integrate transparently into the browser.
Opera route all relevant traffic via their servers and return a compressed stream of data to the browser containing the content you want. From the picture above, you can see that they compress graphics to save bandwidth.
One thing I suspect they do (although I haven’t checked) is reduce the overall number of requests between the browser and the server. Going back to Facebook – it takes 92 HTTP requests to build my home page. That becomes painful if you’re on a low bandwidth, high latency connection. You effectively incur an overhead for each of those 92 requests.
If Opera can turn that into fewer, overall smaller requests – the Norwegians rock even more than I think they already do.
Web Optimisation: Google Page Speed
Friday, June 5th, 2009Bandwidth management and optimisation (BMO) is one of Aptivate’s strategic areas.
Optimising websites is a key ingredient.
Google Page Speed in Firefox
Google just introduced Page Speed which looks like it could be a useful tool. Much like Yahoo’s YSlow it sits on top of Firebug and tells you when you’ve done something naughty.
Aptivate publish web design guidelines targeted at authors developing content for audiences on low bandwidth connections. Google also publish a set of best practices for web performance.
This is interesting – we were motivated to write the guidelines because users in developing countries have a hard time accessing poorly optimised content. Google are motivated to write similar guidelines because they recognise that speed should be a commerical concern.
I think there’s a broadly shared goal that ultimately means good things for those on slower connections.
A quick aside on BMO:
Chris and Martin worked with KENET – The Kenya Education Network Trust, running bandwidth management workshops for network administrators in Kenya, in June 2009.
We’ve posted an overview of the workshop and resources we’re using on our OER Wiki.