• PDF files
• Microsoft Office DOC, XLS and PPT files
• and the new XML equivalents, DOCX, XLSX and PPTX.

I found various tools online to help extract this text, largely thanks to Stack Overflow here and here. This ended up with a hodgepodge of tools:

• PDF Miner for PDF files
• python-docx for DOCX files
• DocToText for PPTX, XLSX and PPT files
• antiword for DOC files

There were a number of problems with this hodgepodge:

• I was unable to find any Python or command-line solution for old Excel (XLS) files;
• These solutions did not extract metadata, only document text;
• The choice of which tool to use depends on the MIME type returned by the file(1) command, which varies depending on the OS (Debian/Ubuntu or CentOS) and which version of the library is installed

Another Stack Overflow post recommended Apache Tika for metadata extraction. It appears to support all the document formats that we need, and to have auto-detection of the document format, which solves all the MIME type problems as well. However, it introduces a new problem: it’s written in Java, which is hard to access from Python.

Luckily I found some instructions for building a Python wrapper around Tika, using some tools that I’d never heard of, and this seemed like a good approach. Unfortunately the installation process is very non-standard, which would not fit in with our fabric-based automated deployment process, and would make it harder for users to install the Intranet themselves.

The instructions are somewhat outdated at the time of writing, as they refer to Tika version 0.7, while 1.0 has been released. I was unable to register for an account to update that page, so I wrote to the author with the details that I discovered, and will also document here that the following command works for me:

python ../jcc/jcc/__main__.py \
        --include /usr/share/java/org.eclipse.osgi.jar \
        --jar tika-parsers-1.0.jar \
        --jar tika-core-1.0.jar \
        java.io.File java.io.FileInputStream \
        java.io.StringBufferInputStream \
        --package org.xml.sax \
        --include tika-app-1.0.jar \
        --python tika --version 1.0 --reserved asm

I was able to go further than this, and package Tika in a way that makes it easy to install with Pip, and thus integrate with our deployment process.

The wrapper is written using JCC, which works by generating and compiling C++ code that links to the Java classes, and then a Python wrapper around that C++. This means that it needs to be recompiled for each platform, so I couldn’t just distribute a binary blob with the Intranet (I had the same problem with DocToText above).

The version of setuptools on our servers doesn’t support JCC’s shared library mode. JCC dies with an error if it’s not explicitly disabled or the patch applies. I couldn’t do either of these as part of our standard deployment process. So I patched JCC to disabled shared mode, since we don’t need it anyway. I also added some patches to allow various setup.py commands used by pip to be forwarded through JCC to the setup function call.

This seems to be enough to allow you to install JCC like this:

pip install git+git://github.com/aptivate/jcc.git

I also wrote a setup.py file that handles pip’s command line invokations and passes the necessary options to JCC, and JCC’s invocation of the setup function. This seems to be enough to install the package using pip:

pip install git+git://github.com/aptivate/python-tika.git

and you can use the last parameter as a package specification in pip_packages.txt, or whatever you pass to pip -r.

You can find the pip-installable Tika package, complete with Tika 1.0 JAR files, in our python-tika repository on Github. This will save you the work of downloading and compiling Tika and all of its dependencies. I have started a discussion with the JCC developers about merging these changes into the upstream project.

In the past I have used Adobe fireworks which seemed unrivalled in the open source world but pngquant is just great.

The command line tool allows you to squish the 32bit png files output from Inkscape into 8 bit files with any number of colours from 2 to 255. To make the test I used an 11.9 kB inkscape file which when processed using pngquant and 24 colours was perfectly acceptable and now had only a 3.1 kB file size. Yippee!!. I’m a happy bunny

(MyPaint bunny 39 kB -> Gimp bunny 53 kB -> happy pngquant bunny 13 kB)

Aptivate work almost exclusively with open source software so it seems a great idea to give Microsoft the push, and frankly I’d had enough of waiting 9 minutes for my laptop to reboot. That’s enough time for making 6 people a cup of tea, water the plants and rearrange the desk; all good things ONCE a day not 5 times when the PC crashes. 3 years of filling up with junk makes a Windows PC very very sluggish and an unhappy designer with a tidy workspace.

Changing over – a gradual process

Leaving Microsoft was never going to a straight switch. Leaving a web platform if you are dependent on it for your income is a scary thing.

Since i cut my design teeth on Apple Macs and Adobe software in 1995, I have moved gradually to Windows in a bid to be one step closer to the end user who on the whole use this platform. I was still, however heavily dependent on Adobe products, primarily Fireworks and Dreamweaver for interface design and development, but also Illustrator for vector graphics and Photoshop for photo editing. I know they are good, but really? couldn’t I achieve a good web result without them?

Open source alternatives within Windows.

Going back a bit to before the anger towards the laptop really kicked in. Finding open source alternatives was an exciting challenge. These were readily available for Windows so I didn’t have to switch, just try them out. I found a really useful website www.osalt.com which helps you to find open source alternatives to commercial software.

Web development – Aptana 2

Essentially I wanted an html/css editor with code completion, ftp client and project manager.. I tried Amaya, Bluefish KompoZer and Mozilla SeaMonkey which all had great features, but none of them did ALL the things I wanted together and I really was a bit spoilt by having them all rolled into one with Dreamweaver. Finally I found Aptana 2 which whilst a bit fiddly to get started with seemed to have everything I needed, hurray!

• Best thing about Aptana – has to be the available plugin support for different types of project such as php, python/django, javascript, svn and git. It’s very comprehensive.

• What I miss the most from Dreamweaver – nothing… had I still been dependent on the WYSIWYG editor in Dreamweaver then I could argue that this would be a bit deal, but since most of my work is now with dynamic database driven sites, I tend to use Firebug to make visual tweaks before committing them to code.

Web design – Inkscape

A key part of my work involves developing brand assets, icons and other vector graphics for both web and print design. For this I had depended on Illustrator. Quite quickly however I discovered Inkscape . What an amazing product! it’s a bit clunky but it lets me do 90% of the key design tasks I did in Illustrator and 60% of tasks I used to do in Fireworks. Essentially it is the easiest transition I have made on this journey and means that I now only use Inkscape for designing for the web.

• Best thing about Inkscape – The ability to export 32 bit png files from any selection, the page or object. The other best bit is all the native SVG features I have yet to discover!
• What I miss the most from Illustrator – nothing worth mentioning.
• What I miss the most from Fireworks – Image optimisation – no image editing software that I’m aware of does a better job of compressing and optimising all nature of image files than Fireworks. It creates an 8 bit alpha png with a tiny size and smooth, edging where transparent bits kick in. This for me is the single most important missing feature of any open source alternative. Inkscape needs a good image optimising tool.

Photo editing – Gimp

Scaling, cropping, optimising photos; that’s a big part of creating photographic content for a website. Mainly I leave that to the end user but sometimes I create photo-based graphics too.
Gimp seems the most logical choice, as it suggests so itself, but I’m not finding it intuitive and it seems to crash more often than not. Fireworks has a limited range of bitmap editing tools compared to Photoshop but integrates really well within the context of creating an interface design as bitmap and vector graphic objects live happily on the same layer or multiple layers. Now there are several open source alternatives for this work such as but I’ll admit that I’m not finding a great solution and again it’s the optimisation issue that makes me frustrated.

Designing for Low Bandwidth

If we need to create websites with small file sizes for countries with low bandwidth then we need powerful optimisers. I found that Fireworks was great because of the high compression rates achievable which result in files more than half the size achievable using other programmes, open source or commercial. So here is a proposal for a future project for Aptivate – create an 8 bit Alpha transparency image optimiser that challenges Fireworks.

Switching to Ubuntu

A couple of months later I reformatted and partitioned the hard drive of my Dell Precision 9300 workstation. I still wanted access to Windows, so did a dual boot with an extra partition for shared files. Apart from my video card packing in unexpectedly (nothing to do with the installation I’m assured) , the installation went without a hitch. I was amazed by how intuitive the Ubuntu interface was. Using the software install centre, I was able to quickly and directly install all the applications I needed. Start up and shut down was a mere seconds and as I was in production mode within a couple of hours. I was more than happy to find Dropbox, Skype and Acrobat Reader were also available. All in all it was pain free, and… Ubuntu is actually graphically beautiful (not something I had anticipated at all).

The dreaded Terminal

Well it’s inevitable even for a designer. The WYSIWYG addicts worst fear, THE TERMINAL!!! agghh!! . A Baptism of code fire at Aptivate, no namby pamby intro here. I had pulled down the designer’s defence gate and in trickled (and sometimes poured) an endless stream of code, configuration files, settings, database fixtures, tables, smart tricks and speedy scripts, screen sharing and networking wizardry. Ah! then I had a cup of tea. I’ll be a terminal ninja one day.

Do I miss Windows?

I don’t miss windows. I booted into it shortly after installing Ubuntu, and found it an empty experience, like going back to a house I you’ve just moved out of but still have keys to. Can’t find the kettle to make a cup of tea so not staying. I recently installed virtual box and now run several different versions of Windows XP to help me cross-browser debug CSS.

Moving forward

Apart from using my new open source toolbox to help Aptivate refresh it’s current website, there are things I want to do. It would be great to contribute to the open source community and generally help to improve already great products such as Inkscape. I’ve already started submitting Inkscape icons to the Open Clip Art Library and it would be great if we developed the image optimiser I mentioned before. There is also a lot of potential in interactive svg’s which would be great to explore especially since Internet Explorer 9 supports them.. oh Microsoft you are never forgotten…

If you are designer reading this, and fancy getting involved with Aptivate’s open source efforts somehow, get in touch. If you are a developer and have a secret optimiser up your sleeve, please let us feed you and keep you amused because we want it!!!

Jeff continues to try and solve the low bandwidth/high latency problems that aid workers face in the field every day and that we encountered in Indonesia. We all know the joy of VSAT networks that slow to a crawl because either some folks on the team are downloading stuff they shouldn’t be downloading or all the computers are infected with bandwidth sucking viruses. It appears Jeff has moved one step closer to sorting out some of the problems surrounding bandwidth optimization by utilizing the Go programming language.

Rather than try and explain to you what Jeff has done I’ll let you read ‘A rate-limiting HTTP proxy in Go‘ and ‘How to control your HTTP transactions in Go‘ and sort out what he is talking about. Hopefully, this post will bait Jeff into leaving a lengthy comment that explains exactly what the hell he is up to.

My understanding is that Jeff is developing two useful tools:

• A web proxy that simulates low bandwidth connections, similar to the Loband Simulator. Jeff’s version is probably more accurate than ours because it doesn’t need to modify the web page, but ours might be easier for non-developers to try out, because you don’t have to install any software.
• A web proxy that can be fed prepackaged content repositories to serve up, so that you can take pre-prepared content (offline websites) with you into the field, and browse them through the proxy as though you were online.

People have been trying to make offlineable websites for a long time. Some of the best examples so far are using entirely client-side (in-browser) technology, such as the Logistics Operational Guide, developed by the World Food Programme for the Logistics Cluster, which can run entirely offline using Google Gears.

Gears had a lot of potential for developers to create offlineable websites, but Google has abandoned its future development in favour of the open standard HTML5, which is not ready yet. So there’s no obvious and future-proof way to develop offlineable websites at the moment. Jeff’s proxy, combined with a spidering system, could be one way to download an entire site, even if it wasn’t designed to be downloaded by the developers.

Another important potential comes from content management systems (CMS) such as WordPress, Drupal and Joomla. More and more websites are developed using such systems, rather than coded from scratch. The systems know all of the pages on the site, and the links between them, and could easily build an offlineable version of the site for download into Gears, HTML5 or Jeff’s proxy. And one plugin could potentially enable thousands of sites to be offlineable, especially if it was included in the CMS distribution and enabled by default.

A few wikis such as MediaWiki, MoinMoin, DocuWiki and JSPWiki have a programming interface (XML-RPC or WebDAV) that allows a smart client to download pages in their original text format, which could make them more efficient to store offline and also potentially editable offline. Jeff’s proxy could be extended to support sites built in such wikis automatically. There are still some limitations to this approach:

• The pages would not look the same as the online versions, since the styling wouldn’t be downloaded and the effects of CMS plugins would not be visible;
• It would probably still be quite slow to download an entire site this way, by spidering, without server-side support for downloading multiple pages at once;
• Few websites are built out of Wikis, so the potential maximum reach is limited compared to better support for WordPress, Drupal or Joomla.

Anyway, I wish I knew Go, and had time to hack on Jeff’s proxy tools.

Hibernate often executes the INSERT operations in batches, which means that the actual failing values are not visible, because the PreparedStatement API gives you no way to get them out, and Hibernate doesn’t let you intercept them being set. The insert can also happen long after you created the object. These facts makes it very hard to find and fix the invalid data.

I decided to write a wrapper for PreparedStatement to capture the values being set by Hibernate, and a new Batcher to wrap the PreparedStatements returned by the driver in wrapper objects of my class. I was about to start laboriously writing yet another delegator class that does the boring work of implementing 100 methods and delegating each one individually to the wrapped class. I love Java so much.

Luckily I stopped and figured that someone might have done this before, and indeed I found an implementation by Holy. I adapted it slightly and integrated it into RITA.

To replace the default batcher with my own, to enable the wrapping of statements, I just had to add the following line to my Hibernate configuration properties:

hibernate.jdbc.factory_class = org.wfp.rita.db.CapturingBatcher$Factory

Thanks, Jakob Holy!

I’d like to report a bug – pmgraph is unable to get data from postgresql database to which nfacctd is writing…

javax.servlet.ServletException: org.postgresql.util.PSQLException:
ERROR: column "src_port" does not exist

It turns out that pmacct, up to and including version 0.12.4, uses a different column name for the source port if the database is MySQL or SQLite:

if (!strcmp(config.type, "mysql") || !strcmp(config.type, "sqlite3")) {
  strncat(insert_clause, "src_port", SPACELEFT(insert_clause));
  strncat(where[primitive].string, "src_port=%u", SPACELEFT(where[primitive].string));
}
else {
  strncat(insert_clause, "port_src", SPACELEFT(insert_clause));
  strncat(where[primitive].string, "port_src=%u", SPACELEFT(where[primitive].string));
}

I really wish applications wouldn’t change their behaviour arbitrarily like this. To work around it, we would have to hard-code the database types in pmGraph as well, or add an option to switch the column names. Since pmGraph uses JDBC to access the database, it’s not even obvious which driver names are accessing an (underlying) MySQL or SQLite database. So we need to switch the column names, but if we can get pmacct fixed then we can ease the pain for new users in future.

I reported a bug to Paolo Lucente, the lead developer of pmacct, through their mailing list. Paolo agreed to change this behaviour, even though it will break backwards compatibility. We spent some time discussing how to do this in a way that would minimise any impact on existing users.

To do this, we took advantage of pmacct’s existing system of database table versioning, which means that you can still use the oldest table structures, even with the most recent version of pmacct. Paolo agreed to create a new schema version that uses the same column names for all databases, so that pmacct will remain backwards-compatible for all users unless they deliberately choose to change their database schema version.

As we chose to standardise on the PostgreSQL column names, the column names will change for MySQL users between schema versions 7 and 8, so we’ll need to add a configuration option to pmGraph to allow users to choose whether they want the old or the new column names. This is the very same switch that I wanted to avoid in pmacct, but pmGraph has fewer users so it has less impact.

As a hobby I’m a photographer, so I do lots of graphic work as editing for invites, posters, mag covers etc. does any one know of a software that will be as great as photoshop for linux (Ubuntu), not Gimp, Gimp is very limited. I tried using WINE but it doesnt work well, very slow.

It was true for a long time that working with digital photos on Linux was difficult. The main problem was that the most popular software that users are familiar with, Photoshop and Lightroom, is simply not available for Linux (due to unfortunate business decisions by Adobe).

There have been open source alternatives to Photoshop for a while, notably Gimp, but Gimp is difficult to use and the interface is quite annoying:

I’d say the major limiting factor is the desire to slit your wrists while arguing with the unwieldy interface layout.

Luckily today I came across (thanks to Alastair Otter of GLUG) this excellent article:

Photography with Open Source / Linux

This includes detailed descriptions of the many applications available, most of which I hadn’t heard of before, and look forward to trying. I’ve been particularly impressed by Adobe Lightroom’s ease of use and powerful histogram correction features, and I was considering buying a copy even though it doesn’t run on Linux, but I will definitely try Darktable now.

For users who really, really have to have a Photoshop user interface, GimpShop may help.

And before you say “this is not work related”, I’ve often needed to edit photographs to prepare brochures, promotional materials and for website work

It seems that digital photography on Linux has made excellent progress in the last year, and I’m very happy to see it.

This is something that ought to be really simple. It’s probably the most common use case of VPNs, Windows has a built-in VPN client, and Cisco routers can be used as VPN servers. EWB want it to be simple, because they have non-technical remote workers. It turned out to be much harder and take much longer than I expected.

Information Overload

One of the biggest problems was the lack of useful information, and the profusion of useless. The information fell mainly into four categories:

• Cisco marketing materials touting the benefits of VPNs and their expensive Concentrator and WebVPN products;
• Cisco knowledge base articles describing the setup of complex VPN scenarios;
• Cisco command references with little or no details on what each command actually does, or how to use them together;
• Cisco exam study sites with inaccurate, out-of-date or cookie-cutter command sequences, with even less explanation of what the commands actually do.

Because I couldn’t find what I was looking for, and had to work it out the hard way, I’ve written it up in the hope that it will help others.

I would recommend any organisations that simply want to share files to seriously consider a file-sharing service like DropBox or raw Amazon S3 instead of a local file server and VPN. In many cases the low upload bandwidth of ADSL connections, combined with internal office use of the connection. will make a VPN impractically slow, especially compared to Amazon’s unlimited upload and download bandwidth. But EWB already had the file server and they just wanted to access it remotely, not to change how they work.

Our scenario is simple: an internal office network with private IP addresses, a Cisco 1800 router providing ADSL connectivity for the office, and remote field workers running Windows desktops.

Getting the Client

For simplicity, we and EWB had hoped to use the built-in VPN client on Windows, which would remove the need to download and install software on the remote workers’ machines. But unfortunately the Cisco 1800 does not support this. Windows uses L2TP over IPSEC for modern, secure VPNs, as a replacement for the old insecure PPTP protocol. But Cisco has crippled the L2TP support in this router, and it only supports raw IPSEC. Only their more expensive routers support serving L2TP over IPSEC, allowing simple direct connections from Windows.

Raw IPSEC is the only remaining option on this router, but it’s difficult to configure due to its complexity, and the number of choices that need to be made. The standard requires both sides to have the same settings configured, but provides no way to do this automatically. Manual configuration would make life very hard for the remote workers. To solve this problem, Cisco has a non-standard protocol for auto-configuration of the clients:

Establishing a VPN connection between two routers can be complicated, and it typically requires tedious coordination between network administrators to configure the two routers’ VPN parameters.

The Cisco Easy VPN Client feature eliminates much of this tedious work by implementing Cisco’s Unity Client protocol, which allows most VPN parameters to be defined at [the] IPSec server.

Cisco Easy VPN Client for the Cisco 1700 Series Routers

So we needed to find a replacement client that was easy to use and could talk to the Cisco. Preferably a free one.

Then we discovered that although Cisco’s own VPN client is technically free, you can’t actually download it without a support contract, which neither we nor EWB have.

In the end we found that if you go to Cisco’s VPN client software page, find the filename of the latest version of the client, and Google it, you’ll find that several people have had enough of this nonsense and posted the client online, so it can be downloaded.

Of course it’s important to be aware of the potential for viruses in copies that you download from random sites on the Internet, as well as fake download sites that lead you around in circles of free registrations, credit card details and pop-up porn adverts. This site worked fine for me, but it may have been taken down by Cisco’s attack dogs by the time you read this.

Security with Obscurity

We decided to choose a configuration that trades some security for ease of use. So instead of authenticating with certificates, we used pre-shared keys. The VPN server has its own login system anyway, which provides an additional layer of security once the remote user is connected to the VPN.

Names and Addresses

Connecting clients need to be allocated an IP address to use over the VPN. We could have used public IPs, or private IPs in the same subnet (with proxy ARP), but we chose to use private IPs in a different subnet. This makes the routing easier, as clients and local network servers will know that they have to route the traffic via the router anyway, and it allows EWB to implement stricter network access policies for VPN clients, if they wish.

We needed to create a local pool (not a DHCP pool) to draw these addresses from:

ip local pool vpnpool 192.168.2.100 192.168.2.200

Keys to the Kingdom

We created an ISAKMP (IKE) policy to specify the authentication method and the level of encryption to be used for negotiation of IPSEC Security Associations (SAs). We chose to make this the first, highest priority policy, and to use AES-256 encryption (strong and fast), Group 2 (1024-bit) Diffie-Hellman key exchange, and pre-shared keys for client authentication as noted above:

crypto isakmp policy 1
 encr aes 256
 authentication pre-share
 group 2

Then we specified the pre-shared key itself. This is the only thing that stops random clients on the Internet from connecting to your local network, so it’s even more important than a strong wireless network key. Of course this is not the real key:

crypto isakmp key ThisKeyMustBeKeptSecret address 0.0.0.0 0.0.0.0

We specify that any IP address can use it by using the wildcard address, 0.0.0.0 0.0.0.0.

At the End of the Tunnel

It seems to be common in corporate environments that, when a user is connected to a VPN, all of their Internet traffic is routed through the VPN. It certainly makes it easier for the network administrators, as they don’t have to define specific routes for the tunnel, but it wastes their bandwidth and makes Internet access much slower for the remote workers, so we decided not to do this.

Just routing a single subnet through a tunnel is called a split tunnel. I couldn’t find simple documentation on setting it up, so I used the Cisco Easy VPN Remote example, extracting just the bits we needed to route only the 192.168.1.0/24 subnet through the tunnel.

First we have to create an access control list (ACL) that defines, on the local (source address) side, what traffic clients should route into the tunnel:

ip access-list extended ewb_office_split_tunnel
 remark Defines which local (office) networks a remote VPN client will route to
 permit ip 192.168.1.0 0.0.0.255 192.168.2.0 0.0.0.255

I’m not sure if the second half of the ACL is actually necessary. It doesn’t appear to make any difference if I specify any instead of 192.168.2.0 0.0.0.255.

Client Configuration

We use Cisco’s EzVPN (Unity) protocol, as described earlier, to configure connecting clients automatically. To do this, we have to tell the server what configuration should be sent to clients when they connect:

crypto isakmp client configuration group EWB
 key ThisKeyMustBeKeptSecret
 dns 192.168.1.1
 wins 192.168.1.2
 pool vpnpool
 acl ewb_office_split_tunnel
 netmask 255.255.255.0

A little explanation about what these options do:

crypto isakmp client configuration group [name]

The name must match the group name that the client uses when it connects. This is how the server decides which configuration to send to the client.

key

For some reason the client needs to be told what key to use, even though it’s already been entered by the user, and the client knows it because it wouldn’t be able to get this far in the negotiation without it!

dns

Tells the client which DNS server to use, for resolving local (private) hostnames, or resolving inside the split horizon. You can specify a second DNS server after the primary one. You probably only need this if you’re running a Windows domain, in which case it should point to the domain controller, or if you have split horizon DNS.

wins

Tells the client which WINS server to use, for resolving local SMB server names. Again, you probably only need this if you’re running a Windows domain, in which case it should also point to the domain controller.

pool

Tells the server which local pool (not DHCP pool) to assign the client’s address from. You can specify any name here, even a pool that doesn’t exist, but clients won’t be able to connect unless the pool name is a valid local pool.

acl

This ACL, which we defined earlier, is used to tell the clients which subnets are reachable through the connection (split tunnel mode). If no acl statement is used, the tunnel is not split, and a default route is set through the VPN tunnel instead.

netmask

Defines the network mask that the client will apply to its client interface, in combination with the IP address assigned from the pool.

Profiling

Next, we create an ISAKMP profile on the server which tells the server to assign IP addresses automatically, and which virtual template to use when creating the virtual-access interfaces for the server side of the tunnel. We haven’t defined the virtual template yet, but we will in a second.

crypto isakmp profile ewb_isakmp_profile
   match identity group EWB
   isakmp authorization list sdm_vpn_group_ml_4
   client configuration address respond
   virtual-template 1

When a client connects using the group name EWB, it will check for network authorization using the AAA list name sdm_vpn_group_ml_4 (or default if that list doesn’t exist), respond to IP address requests from the client (using the pool defined in the client configuration above), and create a local virtual-access interface based on virtual template number 1.

You should use the same group name that you used for the client configuration above, instead of EWB, unless you’re EWB of course.

Strong Encryption

Now we define the level of encryption used for data communications with hosts on the internal network, as opposed to securing the negotiation process. We start by defining a transform set which uses 256-bit AES encryption, the SHA hash algorithm and LZS compression for data packets:

crypto ipsec transform-set ewb_encryption esp-aes 256 esp-sha-hmac comp-lzs

Then we create an IPsec profile that links these settings to the ISAKMP profile that we defined above:

crypto ipsec profile ewb_ipsec_profile
 set transform-set ewb_encryption
 set isakmp-profile ewb_isakmp_profile

Virtual Template

Now we define the template for the virtual interfaces, that we referenced above in the ISAKMP policy:

interface Virtual-Template1 type tunnel
 ip unnumbered Vlan1
 zone-member security in-zone
 tunnel mode ipsec ipv4
 tunnel protection ipsec profile ewb_ipsec_profile

We use ip unnumbered Vlan1 to set the IP address of the virtual-access interfaces to the address of the router on the local LAN (in this case it’s a VLAN bridge), which allows you to ping the router using its internal IP address (192.168.1.1 in our case) when you’re connected to the VPN, which is a useful connectivity test.

We place the virtual interfaces into the in-zone (internal zone) which means that they have full access to the local network, which is not very secure, but simplifies things. We also specify that this interface accepts only traffic encrypted with IPsec and bound to the profile that we created earlier. I’m not sure why it needs to be bound in both directions, as the IPsec profile is connected to the ISAKMP profile which is connected to this virtual interface already.

Client Setup

That should be it for the server-side setup. To configure a client, install the VPN software you downloaded earlier, start it, create a new IPsec configuration, and enter the following details:

Server

The public IP address of the VPN server

Group Name

The same group name that you used on the server earlier

Pre-Shared Key

The same key that you entered on the server earlier

Now click on the Connect button, and after a few seconds the window should minimize to the system tray, and you should be connected to the VPN. You can check this by pinging the internal IP address of the router (e.g. 192.168.1.1) and if that works, the IP addresses of whatever internal servers you want to connect to.

If it doesn’t work, use the Log menu to enable logging, try to connect again, and check the results on the Logging tab. You can also try enabling IPsec debugging on the router, in run mode (not configuration mode):

debug crypto engine packet
debug crypto ipsec error
debug crypto isakmp error
debug crypto verbose
terminal monitor

When the configuration works, write it to the router’s non-volatile memory to ensure that you don’t lose it when you next reboot the router:

write

And that’s it!

References

Here are some random unsorted links to pages that I found useful while figuring out how to do this:

• Configuring a Cisco Router to Accept VPN Connections (even simpler example, without split tunnels)
• Configuring Internet Key Exchange for IPsec VPNs (good general overview of how Cisco’s IPsec works)
• Cisco Easy VPN Remote configuration guide
• Cisco VPN client downloads
• Cisco ARP Commands
• Cisco ISAKMP command reference
• Configuring Zone Policy Firewalls

Management of Internet connections is a big subject. Whole books have been written about it, including the freely downloadable How To Accelerate Your Internet (BMO Book). However, for anyone who doesn’t have time to read it, I will briefly summarise the most important points that I can think of:

• have a clear, simple and strict Internet access policy, and enforce
  it.
• have enough bandwidth, AT LEAST 3 kbps per computer, uncontended. So if you have 1000 computers, you should have 3 MBits dedicated bandwidth, or 60 MBps if it’s shared or contended with a 20:1 contention ratio (typical ISPs).
• have competent network administrators. If you don’t have them, then hire or train them.
• implement good network management practices, e.g. by following the advice of the BMO Book.
• start by solving the problems that users complain most about, to give them the best possible service.
• monitor your network to understand how Internet bandwidth is being used.
• block misuses of Internet access that are causing problems for legitimate use of the Internet connection.
• ensure that client PCs have good, fast antivirus, perform well, are
  regularly reformatted and reimaged, and have strong local security to prevent unauthorized software installation.

Far more information on all of these topics can be found in the BMO book. I suggest starting with the Introduction if you’re interested.

We use Migrate4J to implement database migrations in this Java application. This requires us to write Java code to migrate up to, and down from, each specific database version, by making the required database changes: adding tables and fields, changing field names and types, and modifying data.

However, in our team the database designer is not the person writing these migrations. The designer is working on his copy of the database design, keeping in mind backwards compatibility with the LCTT Access database, and giving me Postgres schema dumps. I have to compare these dumps to identify what has changed, and write the migration code.

What Changed?

First of all, how does one compare dumps? I found Subversion and Diff to be very helpful. We keep the currently-implemented schema checked into Subversion here as a Postgres dump. When I receive a new one, I replace this file, but don’t immediately check it in. I can use the svn diff command, or the Subclipse plugin’s Compare With feature, to see all the changes since the last revision.

Unfortunately Postgres dumps contain some lines that change every time and which aren’t helpful to me, so after I update the dump, I run a command to remove them:

sed -i.orig -e '/^-- TOC entry/d' -e '/^-- Dependencies:/d' master-schema-from-aaron.sql

And then show the differences:

svn diff --diff-cmd=diff -x "-u -F TABLE" master-schema-from-aaron.sql > master-schema-from-aaron.diff

which produces a file that I can load into a syntax highlighting editor (I often pipe it into less instead), and which looks like this:

@@ -554,7 +596,7 @@ -- Name: bundle_type_group; Type: TABLE;
 CREATE TABLE bundle_type_group (
     id integer NOT NULL,
     description character varying(255) NOT NULL,
-    is_qty_allowed smallint,
+    is_qty_allowed smallint NOT NULL,
     record_version bigint NOT NULL,
     is_deleted smallint NOT NULL
 );

This is an extract from a unified diff. The first line, starting with @@, is a header that begins a new section: a block of changed lines, also called a changed hunk or chunk. It includes line numbers from the old and new dump files. It shows three lines of unchanged context above and below the lines that changed.

In this case the line CREATE TABLE bundle_type_group identifies the table being modified, but sometimes the context may not be enough. The last line containing the word TABLE is shown in the header, and normally this helps to identify the table as well.

So this section represents a change to the bundle_type_group table. What changed? A line has been deleted from the dump, and a line has been added. The deleted line is prefixed with - (minus) in the difference file, and the added line is prefixed with + (plus). These lines represent columns in the table.

In this case, the column removed and the column added are both called is_qty_allowed. Because the name is the same, but the types are different, this almost certainly represents a type change to an existing column. If the names were different but the types were the same, it probably represents a renamed column, and if the names and types both differ, it’s probably a deletion of one column and creation of another, discarding the old contents of the column.

It’s worth discussing any unclear changes with the database administrator to be sure exactly what needs to be done. Sometimes there will be data-only migration changes that don’t appear in the schema at all. For example you might decide one day that all people currently called John in the database should now be called Jean, or you might need to add a row to a system table. These can also be done with Migrate4J, but they are not structural (schema) changes.

Creating a New Migration

Assuming that you already have migrations configured in your application, you will have a migration package, where all the classes are named Migration_number. In our case, the migration package is org.wfp.rita.db.migrations. Identify the next migration number in this package, which is usually one higher than the highest number present. Create a class in the package with this name, using this template:

package org.wfp.rita.db.migrations;

/* cleaner sources: */
import static com.eroi.migrate.Execute.*;
import static com.eroi.migrate.Define.*;

public class Migration_2 implements Migration
{
    public void up()
    {
    }

    public void down()
    {
    }
}

Now you can write code to implement the database changes (both schema and data) that you discovered earlier. Each new change is part of an upward migration, and the code that implements it should go into the up method.

It’s important to be able to reverse changes as well. If a schema update fails, you may want to back down to a previous schema, fix the problem that caused it to fail, and try to update again. The code to reverse the change, which is called a downward migration, goes into the down() method.

Note that most migrations lose data in either the forward or the reverse direction (up or down respectively), so you would be well advised to make an automated backup of the database before applying any migrations, in addition to your standard database backup procedures.

Creating Tables

The Execute.createTable() method takes the table name, and an array of Columns. You can create a new Column with one of these constructors:

• new Column(String columnName, int columnType)
• new Column(String columnName,
  int columnType,
  int length,
  boolean primaryKey,
  boolean nullable,
  Object defaultValue,
  boolean autoincrement)

columnType

The type of the column, from java.sql.Types, e.g. Types.INTEGER, Types.FLOAT, Types.VARCHAR.

length

The length of CHAR and VARCHAR columns. The length of all other column types, particularly DECIMAL, must be specified in another way, see below.

primaryKey

True if this column should be part of the primary key, or false otherwise (the default). You can have any number of columns in the primary key, and RITA uses composite primary keys extensively.

nullable

True if this column should be allowed to contain NULL values, and false otherwise.

defaultValue

The default value for new rows. If you set this to null, and the column is not nullable, then a value must be supplied for each record inserted.

autoincrement

True if the column should contain automatically-assigned numbers, using the AUTO_INCREMENT attribute in MySQL, or IDENTITY columns or sequences on databases that support them.

To create a new table called persons, with three columns:

ID

an automatically-assigned integer primary key

fish

a float

rope

a string, 40 characters long, not nullable, defaulting to nylon

we could use the following code in the up migration:

Execute.createTable(new Table("persons", new Column[]{
    new Column("id", Types.INTEGER, -1, true, false, null, true),
    new Column("fish", Types.FLOAT),
    new Column("rope", Types.VARCHAR, 40, false, false, "nylon", false)
}));

Unfortunately this syntax doesn’t allow specifying unique keys, indexes, foreign keys, and precision and scale of decimal columns when the table is created. There is another, shorter syntax which allows specifying the precision and scale:

createTable(table("persons",
    column("id", INTEGER, notnull(), primarykey()),
    column("fish", NUMERIC, precision(8), scale(5)),
    column("rope", VARCHAR, length(40), notnull(), defaultValue("nylon")),
    ));

If that still seems like too much work, and you have a database dump of your new schema, have a look at generating from Postgres dumps below.

The reverse, which you would normally put into the down() method, is simply to drop the table.

Dropping Tables

Dropping a table is as simple as:

Execute.dropTable("persons");

Note that all data in the table will be lost. To recreate the empty table structure in the reverse migration, just create it again.

Adding Columns

To add an INTEGER column called hairs to the persons table, you would add the following code to the up() method:

Execute.addColumn(new Column("hairs", Types.INTEGER), "persons");

The addColumn method takes a Column object, which you can create using either of the methods new Column(...) or column(...) described under creating tables above. The column(...) method is shorter, and the only way to specify the scale and precision of decimal (NUMERIC) columns.

If the change is adding a column, the reverse is to remove the column again, which belongs in the down() method:

Execute.dropColumn("hairs", "persons");

Note that your newly added column will contain default values for all records. If you know what the values should be, or can recreate them using a query, you could execute SQL queries to populate it. Also note that if you migrate down past this version, the column will be dropped and all data contained in it will be lost.

Removing Columns

This is the exact opposite of Adding Columns above. Put the dropColumn() in the up migration, and the addColumn() in the down migration.

Note that migrating down past this migration will not restore the data that was in your column before. If you know what it was, or can recreate it using a query, you could reinsert it using SQL queries.

Renaming Columns

Changing the name of a column does not lose any data. For example, we can rename the column called fish to hats in the persons table, and hope that people don’t try to wear their pet haddock:

Execute.renameColumn("fish", "hats", "persons");

The down() migration trivially renames the column from the new name back to the old name.

Indexes

You can add indexes to columns, both to improve search performance, and to enforce the uniqueness of values in certain columns. The addIndex() method takes an Index object, which you can either create by calling its constructor, or more concisely by calling index() or uniqueIndex(). Both take the same parameters:

index(String indexName, String tableName, String... columnNames)

indexName is the name of the index, which can be null to generate a name automatically. However, such indexes cannot reliably be removed, so I recommend always naming your indexes explicitly. tableName is the name of the table that the index will be applied to, and columnNames is a list of names of columns that will be included in the index.

For example, to uniquely index the fish and rope columns in the persons table:

Execute.addIndex(uniqueIndex("uk_fish_rope", "persons", "fish", "rope"));

You can drop an index, for example for downward migration, using the index name and the table name:

Execute.dropIndex("uk_fish_rope", "persons");

Foreign Keys

Foreign keys link one table to another, to enforce referential integrity between tables. You can create them with Execute.addForeignKey(), which takes a ForeignKey object. There are four ways to construct a ForeignKey:

• ForeignKey(String name, String parentTable, String parentColumn, String childTable, String childColumn)
• ForeignKey(String name, String parentTable, String parentColumn, String childTable, String childColumn, CascadeRule deleteRule, CascadeRule updateRule)
• ForeignKey(String name, String parentTable, String[] parentColumns, String childTable, String[] childColumns)
• ForeignKey(String name, String parentTable, String[] parentColumns, String childTable, String[] childColumns, CascadeRule cascadeDeleteRule, CascadeRule cascadeUpdateRule)

As you can see, these are just the four combinations of whether parentColumns and childColumns are single column names or arrays of column names, and whether the cascade rules are specified or not (they default to “none” if not supplied).

For example, to force a person’s fish_id column to point to the ID of a record in the fish table, you could use this:

Execute.addForeignKey(new ForeignKey("fk_persons_fish", "persons", "fish_id", "fish", "id"));

You can drop a foreign key, for example for downward migration, using the key name and the child (referenced) table name:

Execute.dropIndex("fk_persons_fish", "fish");

Executing Queries

You can execute any arbitrary SQL statement, for example to insert rows into a newly created table or populate a newly created column:

Execute.executeStatement(Configure.getConnection(),
    "INSERT INTO users SET name = 'fred', password = 'flintstone'");
Execute.executeStatement(Configure.getConnection(),
    "UPDATE users SET age = 42 WHERE name = 'barney'");

Although data modification language is much more standard across databases than data definition language, it’s important to be careful only to use ANSI SQL in such statements if cross-database compatibility is important for your application (or might become important in future).

Generating Automatically

If you already have a table structure in a database somewhere, for example if you are retrofitting migrations to an existing project, or if you prefer using GUI tools to design databases, and to reduce the risk of errors, you may want to generate the migration code automatically.

I wrote a script to create Migrate4J migrations automatically from Postgres database dumps. It’s not perfect, it probably only handles the SQL that we actually use, and it’s not well tested, but it may help you. Just run it with the name of the exported schema dump file as its parameter, and it will generate Java code on the standard output, that you can copy and paste into a Java source file.

If the schema will continue to change, and you want help with creating new table definitions in future, you can save the generated output to a file under version control. When you need to generate migration code for a new schema, just overwrite that file, and use svn diff as before to show the differences. They will now be expressed in Java code, which is easier to copy and paste into a new migration.

Applying Manually

In Eclipse, with a migrate4j.properties file on your classpath, you should be able to open the Migrate4J JAR file in Eclipse, expand the com.eroi.migrate package, right-click on Engine and choose “Run As/Java Application”.

Applying Programmatically

As we are using Hibernate, we get a database connection using its Work class, and use it to invoke the migration engine:

// set up Migration schema and run all migrations
m_Session.doWork(new Work()
{
    public void execute(Connection connection) throws SQLException
    {
        Configure.configure(connection, "org.wfp.rita.db.migrations");
        Engine.migrate();
    }
});

Version Control

If I don’t check in the master schema changes immediately, when does it happen? I try to wait until I have all the schema changes implemented in Hibernate annotations and migrations, and run as many tests as I feel the need to run, before checking everything in.

This ensures that the documentation checked in is consistent with the code at that point in time, that I can see the changes to the SQL dump, the Hibernate mappings and the migrations for a single schema update and compare them side-by-side, and reduces the risk of checking in broken code.