View low bandwidth version

Author Archive for mark

Code Kitchen: Overriding and keyword parameters

Monday, September 27th, 2010

I hinted, in the first Code Kitchen that there was a bit more to tell of the story of or recent Django refactoring.

Recall that we created a family of filter classes with the superclass doing all the work of returning what we need for our object tree-view, leaving the subclasses to redefine the filter_x() methods to define selection criteria for X, Y and Z objects.

Now, all the objects in our hierarchy have a Boolean deleted attribute, and we never want to display deleted objects. We could specify this easily using Django’s filter() method by saying X.objects.filter(deleted=False), but we’ve chosen to do something slightly more subtle. Class Filter defines our default criteria like this:

class Filter(object):
    not_deleted = {'deleted':False}
    ...

    def filter_Xs(self, **criteria ):
        criteria.update(self.not_deleted)
        return X.objects.filter(**criteria)

Python functions take two kinds of parameters: positional parameters and keyword parameters. Passing a dictionary, preceded by two asterisks tells Python that the contents of that dictionary should be used as the keyword parameters for that function. Similarly specifying a formal parameter preceded by two asterisks tells Python that we want to be given all our keyword parameters in one dictionary instead of specifying them individually. This way our function always receives a dictionary of its keyword parameters; if no actual keyword parameters are specified in the function call, the dictionary will be empty.

Filter‘s other methods call filter_Xs() (also filter_Ys() and filter_Zs()) with no actual parameters. The method uses the dictionary’s update() method to add the ‘not deleted’ criterion to the empty dictionary which then becomes the actual keyword parameter list for the call to Django’s filter() method.

Filter‘s sub-classes override filter_Xs() (also filter_Ys() and filter_Zs()) to define more specific selection criteria. Though these may be arbitrarily complex, one simple solution is to add more selection criteria to be used by the filter method. Some of our tree filters look like this:

class CountryFilter(Filter):
   country = None

   def __init__(self,country):
        self.country = country

   def filter_Xs(self,**criteria):
        return  super(CountryFilter,self).filter_Xs(country=self.country)

   ...

A CountryFilter is instantiated with a Country object to match on. Its filter methods use super to call Filter‘s implementations of filter_Xs(), etc., specifying their additional criteria with conventional keyword parameters. The superclass implementations of these functions work the same as I described above except now they are adding ‘not deleted’ to to the ‘matches country’ criterion to from the final query.

Filter‘s methods look slightly odd at firs sight. But we write them once and, in return, the implementations of many specific filter sub-classes become simple and uncluttered.

Code Kitchen: Refactoring Django views

Saturday, September 25th, 2010

We, at Aptivate, love openness, participation and learning. We’re also into International Development and, often, we write software. Code Kitchen is a new idea to help us learn to become even better software developers using openness and participation. (more…)

The reality of Internet access in Africa

Thursday, July 8th, 2010

When we talk about making the Internet accessible over slow and unreliable connections, we speak from experience.  Our recent work in Zambia, for example,  taught us a lot about the realities of Internet in the field, in rural Africa.

We were helping to deliver hands-on computer training for 150 young Zambian women as part of our ongoing work with Camfed. The training was framed as an introduction to email; for many it was a first introduction to computers.

We did not expect great Internet connectivity. We set up  a local email system  for the training to avoid dependency on a good connection. While practical, this approach wasn’t ideal as we wanted to offer as much real experience of the Internet as possible.

The rural high school where we were working  has a small IT lab that is connected to the Internet through a directional WIFI link to a near by youth skills centre where there is a VSAT link to the rest of the world.

The environment was harsh for network equipment. The mains power supply was was poor, with regular cuts and brown outs. The school and skills centre both had back-up generators but these  were started manually and not always at the same time. The route to the Internet could not have been called reliable. Neither building normally had UPSs to condition the power to their network equipment, though fortunately we had brought some with us.

Adjusting the antenna to improve signal-to-noise ratio

When conditions conspired to supply power to all the right places, Internet access at the school worked, but it wasn’t great. A VSAT connection has an unavoidably high latency and costs prohibit high bandwidth connections. The  Internet connection at the school was a nominal 1Mbit/s with a 1:10 contention ratio which means we were sharing our satellite bandwidth with nine other user sites. This works well in countries with abundant Internet connectivity as network use is effectively intermittent and sharing doesn’t cause much degradation of service. In Zambia, and much of Africa, a connection like his is often shared between a number  schools, hospitals and telecentres, etc., meaning network use is almost constant. Our experience of bandwidth was often that of the worst-case: 12Kbyte/s.  Though we soon found there were more serious problems with the connection.

Corruption

The web is surprisingly forgiving over a lossy connection experiencing data corruption, and at first it was easy to dismiss slow page load times as a limitation of the connection. Though it soon became obvious that something else was wrong. The image above shows corruption to image data downloaded from a popular mapping site.

A web page with corrupted content can be simply reloaded, but when downloading Ubuntu packages to update a machine we found the md5sums weren’t matching.

The following tests helped us uncover network faults:

  • Pinging hosts on the Internet route using a larger packet (-s packet size).
  • Downloading a large file with a known hash and comparing the computed value. If the hash value didn’t match, then repeating the download and checking if the computed value has changed.
  • Using MTR to see at what router packet loss is occurring.
  • Using Network Diagnostic Tool from a JAVA enabled browser to measure network performance.

Having identified the problem we wanted to establish that the corruption wasn’t happening on our own network: we repeated  the above tests while directly connected to the school’s router, and using more than one computer. The corruption was happening somewhere upstream of our connection, so we called our local ISP (by mobile telephone):

“Can we get an engineer to look into this?”

“Our engineers only visit Samfya district as part of a regular inspection”

“How often do your engineers come out?”…

“Once a month”.

No such visit was likely before we left Zambia. We needed  to find a workaround.

When running a shell over SSH on the trouble connection,  every so often the session would close with a “Corrupted MAC on input” error. This indicates that the last SSH packet failed verification, presumably because the underlying transport layers failed to guarantee the integrity of the data. It seemed like this characteristic could be useful in building a system to help download large files.

Being able to drop connections when ever packets are corrupted isn’t in itself going to help download complete files. But it does mean that a file transfer application running over SSH will never unknowingly write incorrect data to the file (up to the point that SSH MAC checks can detect the problem). This lead to the idea of a brute force solution by persistently running a file transfer application over SSH until the whole file is successfully downloaded. The file transfer application must be able to resume part-way through a file if the link is dropped. A suitable application is rsync.

If files are available on a remote host which can be accessed using public key authentication so that it can be left running unattended, the described work around becomes in shell script form;

return 1
until [ $? -eq 0 ] ; do
 rsync -axrv --partial --inplace <remote-host>:directory .
done

With this method we were able to retrieve the Ubuntu packages we needed up update our servers.

Is there a better way?

Having to manually transfer data through a third machine is inconvenient, a better solution would be transparent to the existing network. Although we can’t easily solve the underlying problem, which is the failure of the relatively small size of the TCP checksum to guarantee the data integrity (in this instance), it is possible to create the conditions where it’s no longer a problem. To do this we might look at reliable transport protocols implemented on top of UDP, which could then be used to tunnel the troubled TCP connections.

Network Diagnostic Tool

Ubuntu in Zambia

Friday, May 7th, 2010

Ubuntu in Zambia was the title of my talk at OggCamp10: I described  our recent work using Ubuntu based low-power computers for training in rural Zambia.Telling this story to the geeks at OggCamp reminded me of the role the Open Source desktop operating system played in this successful project.
(more…)

Mark Skipper joins the team

Tuesday, May 4th, 2010

Today we’re celebrating Mark Skipper’s first day with Aptivate. In fact Dr Skipper has been working with us for over a year now, as a volunteer and as an independent consultant in IT and training; this is his first official day as staff.

Mark has plenty of practical experience of both IT and education. As senior lecturer in computer science, researcher and IT trainer he has worked at Xerox PARC, De Montfort University, Imperial College London, and INRIA Rennes. As independent IT consultant he has worked with large organisations like IBM and the BBC as well as with many very small ones. Somehow, into this full career, he managed to squeeze two years in Kenya, with VSO, as trainer and capacity builder for a rural IT college.

His technical wizardry is balanced with a deep passion for the human aspect of learning, change and development. In addition to a PhD in programming language theory (but really about supporting different points-of-view in software development) he has a Certificate in Education For Sustainability from Schumacher College and lots of training in Nonviolent Communication.

“I’m thrilled to be working with Aptivate”, says Mark, “they’re a great team bringing fun and participation to development projects through IT”.

When not working for us, you’ll find him doing facilitator training for Cambridge Carbon Footprint, communication consultancy for the Cambridge Climate Change Charter, or growing vegetables, tending his chickens or installing Ubuntu. He tweets as axiologic and blogs, occasionally,  as Bitterjug.