As a hobby I’m a photographer, so I do lots of graphic work as editing for invites, posters, mag covers etc. does any one know of a software that will be as great as photoshop for linux (Ubuntu), not Gimp, Gimp is very limited. I tried using WINE but it doesnt work well, very slow.

It was true for a long time that working with digital photos on Linux was difficult. The main problem was that the most popular software that users are familiar with, Photoshop and Lightroom, is simply not available for Linux (due to unfortunate business decisions by Adobe).

There have been open source alternatives to Photoshop for a while, notably Gimp, but Gimp is difficult to use and the interface is quite annoying:

I’d say the major limiting factor is the desire to slit your wrists while arguing with the unwieldy interface layout.

Luckily today I came across (thanks to Alastair Otter of GLUG) this excellent article:

Photography with Open Source / Linux

This includes detailed descriptions of the many applications available, most of which I hadn’t heard of before, and look forward to trying. I’ve been particularly impressed by Adobe Lightroom’s ease of use and powerful histogram correction features, and I was considering buying a copy even though it doesn’t run on Linux, but I will definitely try Darktable now.

For users who really, really have to have a Photoshop user interface, GimpShop may help.

And before you say “this is not work related”, I’ve often needed to edit photographs to prepare brochures, promotional materials and for website work

It seems that digital photography on Linux has made excellent progress in the last year, and I’m very happy to see it.

How to Install Computer Labs

If you ever need to set up a large number of computers in identical configurations, you have a few options:

• Install each one individually by hand
• Automate the standard install process, for example using:
  • AIK for Windows;
  • Kickstart for Red Hat Linux, CentOS and Ubuntu; or
  • sysinstall and mfsroot for FreeBSD.
• Configure one machine exactly how you like it, and then exactly duplicate the hard disk to the others (disk imaging)

The first option (manual installation) is extremely slow, tedious, error-prone, unlikely to result in identical machines, and does not speed up future installations or reinstallations.

The second option requires using rarely-used and less tested parts of the installer, scales poorly in performance for simultaneous installations, and places limits on what you can customise. For example, it could be impossible to customise /etc/rc.conf using the installer on FreeBSD, and pre-installing SSH keys is tricky. I also spent days writing a sysinstall script to automate a process that I could have done in half an hour by configuring a single machine manually.

Therefore I prefer the third option, system imaging.

What is System Imaging

Imaging is the process of making exact copies of one machine’s hard disk, including all partitions, onto another. This only works when the second hard disk is at least as large as the first. It works best when all the PCs are identical.

Imaging is independent of the operating system. You can image Windows, FreeBSD, any version of Linux, dual-boot and triple-boot installations, whatever you like.

We successfully used imaging to set up the PCs for these workshops:

• AfNOG 2009, Cairo, all tracks
• KENET BMO Training, Nairobi 2009
• AfNOG 2010, Kigali, for three tracks: SA-E, SS-E and SI-F

How to Image

Many systems administrators have heard of Norton Ghost and Acronis True Image, two of the most popular commercial applications.

However, open source alternatives such as G4L (Linux-based) and its ancestor G4U (FreeBSD-based) are pretty good, and completely free. G4L however lacks a website, and it’s not obvious how best to use it, hence this post.

G4L is quite similar to G4U, and I could have used G4U instead. But I find the Linux kernel’s hardware support a bit better than FreeBSD’s, and G4L supports multicasting, which enables it to install many machines at the same time with good performance.

Using Ghost for Linux (G4L)

I’ve successfully used Ghost 4 Linux (G4L) versions 0.27 and 0.33 for this process. 0.33 has multicast support, which allows setting up an entire room in one go, without wasting network bandwidth copying the same 4 GB disk image to each of 50 machines independently.

Set up an FTP server on your network with an account that supports downloads and uploads (e.g. on a local server on your network). Make sure it has plenty of disk space free, perhaps 40 GB. Create an “img” directory under the FTP user’s home directory for the images.

Download G4L and burn some CDs, maybe about five copies, or set up network booting (this conflicts with FreeBSD PXE installation and may require BIOS setup changes to enable PXE).

To boot into G4L:

• Reboot or power up the machine
• Press the key to choose boot device
• If CD-ROM is not on the list, reboot, go into the BIOS and enable booting from CD-ROM
• Choose to boot from the CD
• Choose the default kernel at the GRUB screen (just press Enter)
  • If for some reason the default kernel doesn’t work, the machine hangs or crashes or doesn’t detect the network interface, then try one or two other kernels
• Wait for the kernel and initrd to be loaded (two long lines of dots)
• Then you can remove the CD, about one minute from cold boot, and start booting another PC
• Press space to skip each of the information/advertising screens (about 8 of them)
• Enter g4l at the prompt (if you go past this and get a shell, just type g4l at the shell prompt)
• You can access other consoles with Ctrl-Alt-F1 to F4, log in as g4l with no password, and run g4l, ifconfig, ping or whatever
• Choose Network Use (default)
• Choose Raw Mode (default)
• Check that you have an IP address (option B) or try again to acquire one by DHCP
• If you can’t get an IP address by DHCP, check your cabling and DHCP server

Create a Restore Image (optional)

Back up one of your PCs if necessary (if you plan to restore the PCs later) by:

• Follow the procedure above to get into Ghost for Linux
• Enter the FTP server’s IP address, username and password
• Choose an image name, e.g. backup_original_2010_07_22.img
• Choose the back up option
• Press Space to select the entire disk (mark it with an asterisk [*])
• Start backing up the image

This process can take 1-2 hours. In the mean time…

Set up the Master PC

Boot G4L on the PC that you will use as the master. Use DD to wipe the entire disk with zeroes:

dd if=/dev/zero of=/dev/sda bs=1M

This makes the image much smaller, and transfer much faster.

Install FreeBSD or whatever operating system(s) on the master PC, and set it up exactly the way you want all of the PCs to be. Examples include:

• Install Gnome (gnome/gnome2)
• Install Xorg (x11/xorg)
• Install Firefox (www/firefox35)
• Install Xpdf (print/xpdf)
• Enable gnome and sshd in /etc/rc.conf, and add templates for the IP address configuration (this saves typing when setting all the machines to static IPs):

  hostname="pc01.sse.ws.afnog.org"
  ifconfig_bge0="dhcp"
  # ifconfig_bge0="196.200.219.101/24"
  defaultrouter="196.200.219.254"
  gnome_enable="YES"
  sshd_enable="YES"
  

• Create a user account (e.g. username afnog, password afnog)
• Log into Gnome, add firefox, terminal and the Downloads folder to your toolbar, and remove epiphany and evolution
• Edit /etc/fstab and add the proc filesystem:

  proc /proc procfs rw 0 0
  

  (this allows GDM to display the user list and shut down and restart the machine)

• Edit /etc/profile and set the default pager to less by adding:

  PAGER=less; export PAGER
  

• Set the timezone by softlinking /etc/localtime to something like /usr/share/zoneinfo/Africa/Kigali
• Create /etc/rc.local and have it run /usr/sbin/ntpd -qg to set the time once at boot

I recommend using DHCP on this machine. Otherwise all the imaged machines will boot up with the same IP address, causing IP address conflicts, and you will have to reconfigure them before you can access the Internet at all, or reconfigure them automatically.

Create some SSH keys for use in administering the machines. You may wish to set up the local server already and generate the keys there for security. I recommend adding the keys to /root/.ssh/authorized_keys. Please test that they work, and that sshd comes up automatically after boot!

Imaging the other PCs

On all the PCs (master and clones):

• Boot G4L as above
• Check that it has an IP address (option B)

Once a master is online, all the PCs will show “press any key to start”. Pressing any key on any computer will start all the machines imaging. If any PCs are not ready yet, you will have to cancel the imaging process on all of them and start again, or image those PCs later. So:

Start the master last! (when all the other PCs are ready)

Start the clones first, by following these steps on each one:

• Choose UDP Multicast Client (option U)
• Select the entire disk, /dev/sda with the space key
• Say yes, you’re sure
• When it says “Compressed UDP receiver”, it’s ready and waiting for a master to appear on the network

Then start the master:

• Get ALL the clones ready, as above, before doing this!
• On the master, choose UDP Multicast Server (option W)
• Select the entire disk, /dev/sda, with the space key
• Leave the options blank
• Say yes, you’re sure
• The master start accepting connections from clients, which will happen automatically. The screens on the clients will also change.
• Please check that every client says “Press any key to start”.
• If not, please check it for network problems, etc.
• DO NOT stop or kill the server now, unless you want to visit every client again!
• You can press Ctrl+C on the client and run g4l again to check the IP address, retry DHCP, and try the UDP Multicast Client option again.
• This is your last chance to join any remaining clients to the group for this imaging session!
• When all the clients are ready, press a key on the master to start transfer.

The master will show progress of the transfer, and an error line if any clients fail to respond. Clients that cause too many errors will be kicked out of the group and appear to “finish” early.

It’s difficult to tell if the imaging process finished successfully or failed on the clients. However it appears that FreeBSD is very good at detecting filesystem corruption, and will fail to boot if the image was not completely transferred. So you can test them by trying to boot FreeBSD and seeing if it boots completely or stops with a filesystem error. Ideally this would be improved in future versions of G4L.

So I was very interested to see a new training workshop being run by the Science Dissemination Unit (SDU) of the Abdus Salam International Centre for Theoretical Physics (ICTP). The workshop is on Mobile Science: Sensing, Computing and Dissemination and the deadline for applications is tomorrow, July 10th.

Quoting from the announcement:

The Science Dissemination Unit (SDU) of the Abdus Salam International Centre for Theoretical Physics (ICTP), with the assistance of the University of Washington (USA) and of the UCLA Centerfor Embedded Networked Sensing (USA) will hold a Workshop on “Mobile Science: Sensing, Computing and Dissemination” in Trieste (Italy) from the 2 to the 5 of November 2010.

Mobile applications offer tremendous benefits to academic research and
education, and to society as a whole throughout the world. This is an
opportunity that deserves attention and promotion, especially in less
developed areas where mobile phones are the first telecommunications
technology in history to have more users than in the developed world.

The specific things that interested me were:

The Mobile Science workshop aims to engage the scientific community in developing countries in the design, development, and deployment of the newest mobile scientific applications;

i.e. advocating appropriate mobile applications in scientific
research/academia;

Participants will learn how to apply mobile technology tools to retrieve scientific data

I.e. designing mobile apps for science data collection;

how to apply appropriate web-based analysis to assimilate mobile data into scientific studies

I.e. web-based statistical analysis and presentation, like a free online version of SPSS? As far as I know this doesn’t exist yet. The closest that I can think of is the Google Docs spreadsheet, which is of course just a spreadsheet, requires an internet connection and doesn’t allow plugins for additional scientific analysis functionality. But there could be a very interesting app to develop here.

and how to share their scientific findings with a potentially large mobile audience.

I.e. low bandwidth design with an emphasis on web standards for cross-platform compatibility, so that it works on the largest number of mobile devices.

If you want to apply, better get on your bike (or modem?) because the deadline is tomorrow. If you want to do mobile scientific research applications, please get in touch, we’d like to help you.

The average size of web pages has been growing much faster than the average speed of connections in developing countries, and journal websites are no exception, as you can see in Alan’s blog post:

Average Page Size vs Bandwidth

As Alan points out, the average journal’s home page in his sample would take over 90 seconds to load on average, for researchers at universities in developing countries. Usability research has shown that people expect a computer to respond within 30 seconds. Making them wait longer interrupts their concentration, causes dissatisfaction and annoyance, and they often abandon the process. The biggest factor in user satisfaction is speed of response.

While this research probably did not include users who are accustomed to slow and unreliable computers, I think it’s safe to say that most people would find it annoying and difficult to use the Internet on a dial-up modem. And even a modem would have been preferable to some of the Internet connections that I’ve experienced (and paid for) in some countries in the last few years.

Academics have little ability to persuade their universities to upgrade their internet connections, at a cost of several peoples’ salaries (several thousand dollars a month). The only people who can change this are the publishers of the journals, by optimising their journals’ websites for users with slower connections.

But how to persuade the publishers that this is important? We built a low bandwidth simulator ourselves, and took it to Oxford, to INASP and the ACU’s Publishers for Development conference.

What We Did

We set up spare machine as a bandwidth management box, and used it as a network filter for the participants. They could come and plug their laptops into the box, and browse the Internet and their own websites at a simulated slow speed.

Exercise Table

We configured the box for transparent bridging. This allowed us to insert and remove it from the network easily, just by switching over a network cable, to demonstrate the difference between fast and slow loading of pages.

We gave the participants at the meeting tasks to perform on various publishers’ websites, for example finding and downloading an academic paper by topic or researcher.

Playing the Game

I think they found the activities enlightening, because we had some very good comments from some of the participants:

• “We’re so pleased that Alan was able to work his magic at the recent PfD session – his delivery is innovative, dynamic and fact-packed so it really sparks enthusiasm from the audience… [which] is demonstrably channelled into action once people return to their places of work.“
  Publishers for Development Team
• “It was really useful to try the low bandwidth! [Our site] is already considered fast but it made us think even more around this issue, what else can we do etc.“
  Anonymous Participant
• “Alan Jackson’s information about bandwidth was kind of shocking even if I knew it before, but to really experience it was very valuable. We are going to redesign DOAJ’s home page and this must be the starting point.“
  Sonja Brage, DOAJ
• “Site speed is a major consideration for us, and I really enjoyed Alan/Aptivate’s session, experiencing the exasperation of trying (and failing) to connect via low-bandwidth… I have a feeling that there is ‘excess baggage’ on a number of the pages…“
  James Kitchen, OECD

How We Did It

We used FreeBSD as the operating system for the software bridge, because its dummynet traffic shaper is relatively easy to use, and very good at simulating slow connections.

We wanted to use a laptop instead of a desktop machine, so that we could carry it to the conference easily, but we had hardware compatibility issues with FreeBSD on all the laptops we had available to us (mostly IBM Thinkpads). We ended up using a compact Fujitsu desktop box.

We installed FreeBSD 8 on it, and configured it to transparently bridge between two interfaces. Our internet access at the conference would be wireless, but we had issues with bridging wired and wireless interfaces together. So instead we used a Linksys WRT-54GL router with the Tomato firmware, which enables wireless client mode, to connect to the network:

Throttler Network Diagram

And this is what it looked like in the room. Notice the essential coffee and cupcake, without which the system mysteriously failed to work:

Network Close Up

We configured the FreeBSD box to bring up the bridge automatically at boot time, and to load a set of ipfw firewall rules to enable dummynet, the traffic shaper. On this box, the ethernet interfaces are called em0 and rl0, so we added the following lines to /etc/rc.conf:

ifconfig_em0="up"
ifconfig_rl0="up"
cloned_interfaces="bridge0"
ifconfig_bridge0="addm em0 addm rl0 up dhcp"

firewall_enable="YES"
firewall_type="/etc/ipfw.rules"
dummynet_enable="YES"

Then we created /etc/ipfw.rules with the following contents:

# with bridge mode, two nics. em0 is wan
add pipe 1 all from any to any out recv em0
add pipe 2 all from any to any out xmit em0
add allow all from any to any
pipe 1 config delay 700ms bw 40Kbit/s mask dst-ip 0x000000ff
pipe 2 config delay 700ms bw 40Kbit/s mask src-ip 0x000000ff

This configuration creates two dummynet pipes. Pipe 1 is for traffic received on the external interface (downloads), and pipe 2 is for traffic being sent out of the external interface (uploads). We have to follow this by a rule which allows all other traffic, otherwise local traffic (on the box itself) is denied by default when the firewall is enabled, which breaks local DNS and inbound SSH and makes the box pretty unusable on the console.

Then we configure both pipes to allocate 40 Kbps (kilobits per second) for each individual IP address in the private subnet (allocated by the DHCP server on the Tomato router) and a 700 ms delay in each direction, which gives a 1400 ms round trip time. This is somewhat higher than the expected 600 ms round trip for a connection by geostationary satellite.

The end result is that each user connects a laptop to the switch behind the box, gets an IP address from the DHCP server on the router, is NATted by the router onto the public network, and is able to browse the Internet with a connection of 40 kbps upload and download. If you remove the FreeBSD box, by connecting the switch directly to the router, you can access the public network at full speed.

One issue was that the public network used a captive portal, which we had to log into. We didn’t want each client on our network to have to log in separately, so we enabled NAT on the router, and in wireless client mode, all the NATted clients get the MAC address of the router, so the public network thinks that they’re all the same PC and doesn’t ask them to log in again.

Why We Did it

We think that members of universities and research institutions need to be able to join and participate in the global research community as equals, in order to play their part in assisting development in their home countries.

Programmes such as PERii, HINARI and AGORA negotiate free or discounted online access to these journals for universities in developing countries. But the users still need to get online and access the content.

Online publishing for Western markets is usually designed for users with fast Internet connections, which Western universities have. But in other regions, universities often can’t afford fast connections, and this makes it very difficult for them to access these journals online.

Publishers for Development is bringing international publishers together who are interested in finding out how they might contribute to discourse and action around developing country access, encourage publication from developing country researchers and understand the diversity within research cultures/communities and the challenges these present.

We use Migrate4J to implement database migrations in this Java application. This requires us to write Java code to migrate up to, and down from, each specific database version, by making the required database changes: adding tables and fields, changing field names and types, and modifying data.

However, in our team the database designer is not the person writing these migrations. The designer is working on his copy of the database design, keeping in mind backwards compatibility with the LCTT Access database, and giving me Postgres schema dumps. I have to compare these dumps to identify what has changed, and write the migration code.

What Changed?

First of all, how does one compare dumps? I found Subversion and Diff to be very helpful. We keep the currently-implemented schema checked into Subversion here as a Postgres dump. When I receive a new one, I replace this file, but don’t immediately check it in. I can use the svn diff command, or the Subclipse plugin’s Compare With feature, to see all the changes since the last revision.

Unfortunately Postgres dumps contain some lines that change every time and which aren’t helpful to me, so after I update the dump, I run a command to remove them:

sed -i.orig -e '/^-- TOC entry/d' -e '/^-- Dependencies:/d' master-schema-from-aaron.sql

And then show the differences:

svn diff --diff-cmd=diff -x "-u -F TABLE" master-schema-from-aaron.sql > master-schema-from-aaron.diff

which produces a file that I can load into a syntax highlighting editor (I often pipe it into less instead), and which looks like this:

@@ -554,7 +596,7 @@ -- Name: bundle_type_group; Type: TABLE;
 CREATE TABLE bundle_type_group (
     id integer NOT NULL,
     description character varying(255) NOT NULL,
-    is_qty_allowed smallint,
+    is_qty_allowed smallint NOT NULL,
     record_version bigint NOT NULL,
     is_deleted smallint NOT NULL
 );

This is an extract from a unified diff. The first line, starting with @@, is a header that begins a new section: a block of changed lines, also called a changed hunk or chunk. It includes line numbers from the old and new dump files. It shows three lines of unchanged context above and below the lines that changed.

In this case the line CREATE TABLE bundle_type_group identifies the table being modified, but sometimes the context may not be enough. The last line containing the word TABLE is shown in the header, and normally this helps to identify the table as well.

So this section represents a change to the bundle_type_group table. What changed? A line has been deleted from the dump, and a line has been added. The deleted line is prefixed with - (minus) in the difference file, and the added line is prefixed with + (plus). These lines represent columns in the table.

In this case, the column removed and the column added are both called is_qty_allowed. Because the name is the same, but the types are different, this almost certainly represents a type change to an existing column. If the names were different but the types were the same, it probably represents a renamed column, and if the names and types both differ, it’s probably a deletion of one column and creation of another, discarding the old contents of the column.

It’s worth discussing any unclear changes with the database administrator to be sure exactly what needs to be done. Sometimes there will be data-only migration changes that don’t appear in the schema at all. For example you might decide one day that all people currently called John in the database should now be called Jean, or you might need to add a row to a system table. These can also be done with Migrate4J, but they are not structural (schema) changes.

Creating a New Migration

Assuming that you already have migrations configured in your application, you will have a migration package, where all the classes are named Migration_number. In our case, the migration package is org.wfp.rita.db.migrations. Identify the next migration number in this package, which is usually one higher than the highest number present. Create a class in the package with this name, using this template:

package org.wfp.rita.db.migrations;

/* cleaner sources: */
import static com.eroi.migrate.Execute.*;
import static com.eroi.migrate.Define.*;

public class Migration_2 implements Migration
{
    public void up()
    {
    }

    public void down()
    {
    }
}

Now you can write code to implement the database changes (both schema and data) that you discovered earlier. Each new change is part of an upward migration, and the code that implements it should go into the up method.

It’s important to be able to reverse changes as well. If a schema update fails, you may want to back down to a previous schema, fix the problem that caused it to fail, and try to update again. The code to reverse the change, which is called a downward migration, goes into the down() method.

Note that most migrations lose data in either the forward or the reverse direction (up or down respectively), so you would be well advised to make an automated backup of the database before applying any migrations, in addition to your standard database backup procedures.

Creating Tables

The Execute.createTable() method takes the table name, and an array of Columns. You can create a new Column with one of these constructors:

• new Column(String columnName, int columnType)
• new Column(String columnName,
  int columnType,
  int length,
  boolean primaryKey,
  boolean nullable,
  Object defaultValue,
  boolean autoincrement)

columnType

The type of the column, from java.sql.Types, e.g. Types.INTEGER, Types.FLOAT, Types.VARCHAR.

length

The length of CHAR and VARCHAR columns. The length of all other column types, particularly DECIMAL, must be specified in another way, see below.

primaryKey

True if this column should be part of the primary key, or false otherwise (the default). You can have any number of columns in the primary key, and RITA uses composite primary keys extensively.

nullable

True if this column should be allowed to contain NULL values, and false otherwise.

defaultValue

The default value for new rows. If you set this to null, and the column is not nullable, then a value must be supplied for each record inserted.

autoincrement

True if the column should contain automatically-assigned numbers, using the AUTO_INCREMENT attribute in MySQL, or IDENTITY columns or sequences on databases that support them.

To create a new table called persons, with three columns:

ID

an automatically-assigned integer primary key

fish

a float

rope

a string, 40 characters long, not nullable, defaulting to nylon

we could use the following code in the up migration:

Execute.createTable(new Table("persons", new Column[]{
    new Column("id", Types.INTEGER, -1, true, false, null, true),
    new Column("fish", Types.FLOAT),
    new Column("rope", Types.VARCHAR, 40, false, false, "nylon", false)
}));

Unfortunately this syntax doesn’t allow specifying unique keys, indexes, foreign keys, and precision and scale of decimal columns when the table is created. There is another, shorter syntax which allows specifying the precision and scale:

createTable(table("persons",
    column("id", INTEGER, notnull(), primarykey()),
    column("fish", NUMERIC, precision(8), scale(5)),
    column("rope", VARCHAR, length(40), notnull(), defaultValue("nylon")),
    ));

If that still seems like too much work, and you have a database dump of your new schema, have a look at generating from Postgres dumps below.

The reverse, which you would normally put into the down() method, is simply to drop the table.

Dropping Tables

Dropping a table is as simple as:

Execute.dropTable("persons");

Note that all data in the table will be lost. To recreate the empty table structure in the reverse migration, just create it again.

Adding Columns

To add an INTEGER column called hairs to the persons table, you would add the following code to the up() method:

Execute.addColumn(new Column("hairs", Types.INTEGER), "persons");

The addColumn method takes a Column object, which you can create using either of the methods new Column(...) or column(...) described under creating tables above. The column(...) method is shorter, and the only way to specify the scale and precision of decimal (NUMERIC) columns.

If the change is adding a column, the reverse is to remove the column again, which belongs in the down() method:

Execute.dropColumn("hairs", "persons");

Note that your newly added column will contain default values for all records. If you know what the values should be, or can recreate them using a query, you could execute SQL queries to populate it. Also note that if you migrate down past this version, the column will be dropped and all data contained in it will be lost.

Removing Columns

This is the exact opposite of Adding Columns above. Put the dropColumn() in the up migration, and the addColumn() in the down migration.

Note that migrating down past this migration will not restore the data that was in your column before. If you know what it was, or can recreate it using a query, you could reinsert it using SQL queries.

Renaming Columns

Changing the name of a column does not lose any data. For example, we can rename the column called fish to hats in the persons table, and hope that people don’t try to wear their pet haddock:

Execute.renameColumn("fish", "hats", "persons");

The down() migration trivially renames the column from the new name back to the old name.

Indexes

You can add indexes to columns, both to improve search performance, and to enforce the uniqueness of values in certain columns. The addIndex() method takes an Index object, which you can either create by calling its constructor, or more concisely by calling index() or uniqueIndex(). Both take the same parameters:

index(String indexName, String tableName, String... columnNames)

indexName is the name of the index, which can be null to generate a name automatically. However, such indexes cannot reliably be removed, so I recommend always naming your indexes explicitly. tableName is the name of the table that the index will be applied to, and columnNames is a list of names of columns that will be included in the index.

For example, to uniquely index the fish and rope columns in the persons table:

Execute.addIndex(uniqueIndex("uk_fish_rope", "persons", "fish", "rope"));

You can drop an index, for example for downward migration, using the index name and the table name:

Execute.dropIndex("uk_fish_rope", "persons");

Foreign Keys

Foreign keys link one table to another, to enforce referential integrity between tables. You can create them with Execute.addForeignKey(), which takes a ForeignKey object. There are four ways to construct a ForeignKey:

• ForeignKey(String name, String parentTable, String parentColumn, String childTable, String childColumn)
• ForeignKey(String name, String parentTable, String parentColumn, String childTable, String childColumn, CascadeRule deleteRule, CascadeRule updateRule)
• ForeignKey(String name, String parentTable, String[] parentColumns, String childTable, String[] childColumns)
• ForeignKey(String name, String parentTable, String[] parentColumns, String childTable, String[] childColumns, CascadeRule cascadeDeleteRule, CascadeRule cascadeUpdateRule)

As you can see, these are just the four combinations of whether parentColumns and childColumns are single column names or arrays of column names, and whether the cascade rules are specified or not (they default to “none” if not supplied).

For example, to force a person’s fish_id column to point to the ID of a record in the fish table, you could use this:

Execute.addForeignKey(new ForeignKey("fk_persons_fish", "persons", "fish_id", "fish", "id"));

You can drop a foreign key, for example for downward migration, using the key name and the child (referenced) table name:

Execute.dropIndex("fk_persons_fish", "fish");

Executing Queries

You can execute any arbitrary SQL statement, for example to insert rows into a newly created table or populate a newly created column:

Execute.executeStatement(Configure.getConnection(),
    "INSERT INTO users SET name = 'fred', password = 'flintstone'");
Execute.executeStatement(Configure.getConnection(),
    "UPDATE users SET age = 42 WHERE name = 'barney'");

Although data modification language is much more standard across databases than data definition language, it’s important to be careful only to use ANSI SQL in such statements if cross-database compatibility is important for your application (or might become important in future).

Generating Automatically

If you already have a table structure in a database somewhere, for example if you are retrofitting migrations to an existing project, or if you prefer using GUI tools to design databases, and to reduce the risk of errors, you may want to generate the migration code automatically.

I wrote a script to create Migrate4J migrations automatically from Postgres database dumps. It’s not perfect, it probably only handles the SQL that we actually use, and it’s not well tested, but it may help you. Just run it with the name of the exported schema dump file as its parameter, and it will generate Java code on the standard output, that you can copy and paste into a Java source file.

If the schema will continue to change, and you want help with creating new table definitions in future, you can save the generated output to a file under version control. When you need to generate migration code for a new schema, just overwrite that file, and use svn diff as before to show the differences. They will now be expressed in Java code, which is easier to copy and paste into a new migration.

Applying Manually

In Eclipse, with a migrate4j.properties file on your classpath, you should be able to open the Migrate4J JAR file in Eclipse, expand the com.eroi.migrate package, right-click on Engine and choose “Run As/Java Application”.

Applying Programmatically

As we are using Hibernate, we get a database connection using its Work class, and use it to invoke the migration engine:

// set up Migration schema and run all migrations
m_Session.doWork(new Work()
{
    public void execute(Connection connection) throws SQLException
    {
        Configure.configure(connection, "org.wfp.rita.db.migrations");
        Engine.migrate();
    }
});

Version Control

If I don’t check in the master schema changes immediately, when does it happen? I try to wait until I have all the schema changes implemented in Hibernate annotations and migrations, and run as many tests as I feel the need to run, before checking everything in.

This ensures that the documentation checked in is consistent with the code at that point in time, that I can see the changes to the SQL dump, the Hibernate mappings and the migrations for a single schema update and compare them side-by-side, and reduces the risk of checking in broken code.

(c) Fatuma Iseje

Why robots? As a child I grew up programming the Sinclair ZX81. A small computer about the size of a book, with a touch-(in)sensitive membrane keyboard and 1kB of memory. With computers we spend long hours staring at the screen interacting with a virtual world made of intangible pixels. I remember being constantly frustrated that my computer couldn’t do anything in the REAL world. It couldn’t turn lights on and off, it couldn’t move and back then it couldn’t even makes noises. In fact I killed the poor thing when I dropped a soldering iron through the CPU while trying to interface it to my physical world.

It’s all different now. With the advent of devices like the Arduino it’s now easy and cheap to get the computer to reach out into our world. And we can reach into its world. Not only does an Arduino let us physically interact with the computer it can also be automonous. We can make things that interact with us without needing the computer attached to the other end of its USB umbilical cord.

For those that haven’t come across it yet, an Arduino is a simple micro-controller board. OK… what’s so amazing about that, there’s been micro-controller boards around for decades? Well, a few things. Firstly it has a USB port so you can easily connect it to your computer. It has its own simple programming environment using a variant of the C programming language. So it’s not dumbed down at all – no visual programming environment or anything like that. And critically there is a large on-line community eager to share and support people who do not come from a technical background. Its design is open source, it’s cheap (£20) and you can find even cheaper clones.

This exploding field called “Physical Computing” is making computing more fun, more compelling and also sometimes more useful.

Ok robots might be fun, but why send them to Zambia?

The computer used to be like a calculator on steroids. It could “compute” things. If you knew a bit about SIN or COS you could make it draw a circle. Or calculate a volume. Now, although computers are enormously more powerful than the Sinclair ZX81, most of us use them like some mash-up of a TV and a phone. They give us access to information and lets us communicate. These are important and useful functions. But I worry most of us are in danger of treating our computers like new cars – like devices with “no user serviceable parts inside“.  Instead of seeing them as tools that we can fix, modify, improve, re-invent and tailor to our needs.

It’s more than that. Computers aren’t just useful for the stuff they do. I believe that programming computers, tinkering with them, making electronic circuits and building robots builds highly transferable skills. Critical thinking, logical and abstract reasoning and problem solving. In addition, programming an Arduino makes you a dab hand at searching the Internet.

I got into robots at BuildBrighton /  BrightonRobot – the Brighton hacker-spaces based out of The Skiff – and in particular through Steve Carpenter’s amazing kits.

Eva with the Drawbot - (c) Fatuma Iseje

As an introduction to robotics, we’ve used the “DrawBot” packaged up by Steve with excellent instructions. How complicated and expensive do you think an autonomous drawing robot would be? How about 6 components that require no soldering costing about £3?

Steve has also designed a great robot kit based around the Arduino – the BoxBot. The kit is designed using the Interlocking T-Bolt Construction method, a cost-effective technique for low volume manufacture. With servos and everything you need minus the Arduino, the kit’s about £60. It’s a very flexible kit, you can bolt sensors or motors all over it – but this is not Lego. You need to drill holes in it.

Reading the educational posts about robotics I can see that from a teacher’s point of view an Arduino and the BoxBot may not be ideal. They don’t neatly fit into lesson plans. They don’t just leap into life in a way that keeps a lesson of 30+ students moving along. They take a bit more application and research to use. But unlike Lego, Meccano and other modular systems, they blur the distinction between the kit and the rest of the world. Once you’re empowered to drill holes and wire up your BoxBot you find you’re up for drilling holes and wiring up anything.

Using robots in this context is an experiment for us. We don’t know how it will go. But it’s an experiment the young women of the Samfya Resource Centre are excited about.

We’ll keep you posted on the progress of our latest bit of R&D.

“Hi” and “Mulishani” to Bridget, Everlyn, Fatuma, Mary, Mildred and Penelope at the SRC from the Aptivate gang!

Crisis Camp London

We’re getting ready for the second Crisis Camp London day, this Saturday, at the London Knowledge Lab.

Considering how quickly last Saturday was put together we had a great turn out. We’re hoping for, and expecting, an even larger turn out this time.

We’re getting out all the laptops, network gear, and spare projectors we can find. We’ve got enough flip-charts to paper the Albert Hall. I even heard that Domino’s were considering sponsoring the pizza again – and with added vegan options too. Many thanks to them!

We’re on conference calls to the US Crisis Camps and projects. We’re part of a strong team looking after the organisation, infrastructure and facilitation of the London event.

It really is quite something the speed with which this is coming together and the level of interest and motivation. This looks set to become an important movement.

If you are technical (eg. an IT project manager, a coder, a user experience person) or have relief experience and want to help Crisis Camp London support Haiti, then please come along.

• find out more
• register

You can also help by printing out and displaying the Crisis Camp London Poster

#CrisisCampLDN

“What’s the difference between Open Spaces and BarCamps?”

Here’s how I described it to a good facilitator friend of mine (Julian from DecisionLab)…

“I suppose a BarCamp isn’t exactly like a pure Open Space. It’s got it’s own nuance.

For instance it’s generally not well regarded to plan BarCamps too far in advance. For no real reason in particular, it’s just not the BarCamp way.

People do prepare some stuff to take with them to a BarCamp knowing they will have an opportunity to talk about it – but they make no commitment until the day so they can change it or not give it at all.

Having people declare before hand makes it feel much more like a traditional conference and I think will discourage others from joining in on the spur of the moment.

One thing to remember about BarCamps, the big difference between BarCamps and Open Spaces is the ludic nature of BarCamps – they’re supposed to be fun and don’t have a particular outcome – they’re more like Popular Education rather than group decision making.

I think of it like this. Imagine you’re 11 years old and you and a gang of your friends are mad keen on technical lego. You’ve arranged one weekend to have a sleep-over with a lego theme. You’re parents have sorted out pizza and ice cream for everyone. Everybody brings lego with them and you take the half finished robot you’re working on. During the lego-fest the gang comes up with an idea – wouldn’t it be cool if we put all our lego together and see if we can make a bridge over the stair well.

THAT is a BarCamp. You might prepare something because it’s fun. But you’re just going along to learn, be inspired, get enthusiastic, hang out with some cool people and have a bit of a laugh.

The idea of getting people to coordinate their talks so they don’t have duplicates is I think wrong. You should have duplicates. If that’s what people want to talk about – duplicates give more opportunities for people to hear about it.”