Friday 5 December 2014

TripVis MVP

I have a minimum viable product!

  • Minimum: there really isn't very much I could strip out, without loosing something. It is running on the smallest setup, its a bit slow, and there's already a backlog of ideas. But they all add on to this. Yes, there's an about page and a terms page, but I think they signal something important about intentions and reasons for something I'm putting out there. 
  • Viable: well, it does something. Whether anyone else wants it remains to be seen, but that's the point. 
  • Product: yes, it's a thing (a cloud hosted web service if you want). You can use it. One day I might add paying for it, and make it a product in the business sense of the word. But for now it is a pet project and a vehicle for exploring ideas, an ideas product.
So, here it is:

Not too much fanfare for now - try a few things, see what happens. If you give it a go I'd be delighted to hear from you:

Emergent Technology and Economic Transformations

BBC Radio 4's Start the week with William Gibson and Judy Wajcman (and others, but I mention them because it is their comments I respond to below) on science fiction on 24th November touched on issues that I've read about in Economic Transformations by Lipsey, Carlaw and Bekar.

The Gibson quote that hit me was "emergent technology is the big change driver in society and long has been", which then led into a comment that technology emerges rather than being legislated into existence and that we're not unique in living in a time of technological change. This idea that technology has produced unpredictable but sustained economic change over a long period is core to the economics book. They also discuss that these changes are not directly tied to individual products but to more fundamental technologies. Not every new product can have this large-scale, sustained transforming approach, most are part of a larger technology. Many significant social changes are prompted by combinations of technology, rather than individual new technologies. These might build on each other, e.g. electricity then telegraph; or in combination, e.g. DNA and computers. Another issue they discuss at length is that the emergence of the technology, its widespread application, and the social / economic benefit are often separated by a significant lag. This isn't just buying new gadgets beyond early adopters, but the application of technology in ways that create efficiencies at a social level which then enables large scale changes in patterns of living and production - such as the connection between steam power, factory production and movement from cottage industry to city living. It is the social effects which make the technology a significant driver of the economy, and the technology needs experimentation and a foundation of science to reach the stage of having widespread, substantial and sustained effects as the basis for products but transcending an individual product.

On the radio they touched on many other issues. I'm not going to comment on everything, but busyness was another: the need to be constantly doing something (or at least be seen to be busy), to be up to the minute. Despite a pair of jobs, a hobby project, a blog and a couple of Twitter accounts (and a non-tech life) I'm going to claim bucking the trend here! I've been meaning to write about that book for a while, but haven't found the seed for something to write until now. Because I don't engage on a real time basis, but via a podcast, it has taken me until yesterday to listen to the programme. Because I try to percolate what I blog, let the ideas filter and brew, there's another day to posting here. Good ideas don't have to be had now. Despite first mover advantage, or the need of others to build on your idea, the quality of the idea matters too. Reflecting the previous point, big ideas that really impact what you do take time. In my technical work I've found that some of the most fundamental changes to how I work are based in principles and connections between ideas that have matured over time. The maturation happens both in the wider world, but also in my mind - satisfying a need via an approach rather than a specific thing.

Of course inspiration finds you when you're working, but you don't need to make a show of working for inspiration to find you - because the show is rarely the work. The gaps between work are really important if you work in ideas, they give the brain breathing space ... listen to a podcast, read a book!

Friday 7 November 2014

Comment On Slik and Spinach

I felt moved to comment on a post on Kevin Rutherford's blog, about code design, and forgot to link from here...

I couldn't resist a note of humour about the analogy, but the real point was about design. For the most part I was agreeing with him: Exceptions are no place for routine events. But, "loosing a conditional" was, in my opinion, over egging the pudding. The situation would still need handled, just somewhere else. In the case of his example, by another actor (think micro-service, probably).

Anyway, it's a good post and my comment is below silk and spinach - on paperboys, newsagents and exceptions.

Thursday 6 November 2014

9 Tests with FluentLenium

As well as work projects I like to have a personal project which lets me try new things. As past blog entries may have suggested the current one is a web site, running on NinjaFramework. Having got the hang of the core underlying technology I had produced a chunk of MVP site. Now, rather belatedly, it was time to introduce tests on the web interface as well as the code. Ninja has FluentLenium packaged in. It functions as a fluent wrapper to Selenium. FluentLenium was new to me and so I set about breaking it. Here are the edited highlights of that process and a few lessons learnt (breaking things is a really good way to learn).

My starting point was the simplest possible test: load the home page, check that it had some text that was about right (key words in key places rather than full text). Use the example code as a basis.
I started with the HtmlUnitDriver as the engine. It worked.
Next I tried navigating to the about page. Still nothing clever behind the scenes. Then pressing the back button. Ah. Need to switch javascript on. Now it moaned bitterly in warnings about a lot of things, mostly because I'm using Bootstrap.
Another two "load a simple page" tests followed. The warnings were a bit tiresome. The code was getting scrappy. Although the API was FluentLenium they were reading more like a script. Time for a bit of structure, with common elements of the scripts factored into methods and a bit more of the fluent style.
Next test was moving into testing user registration, login, and logout. This involved a form, inputting data, pressing a button that (via javascript) would POST the data and AJAX the response. HtmlUnitDriver was no longer the tool for the job - its JavaScript just isn't up to it. I already had firefox on my local test VM. I was able to add it to my CI slave, the only problem being getting a compatible version of firefox (see here). That also made all the warning noise go away, which was nice.
But, this threw me into the world of asynchronous interactions and waiting for things to happen at the same time as AJAX updates to pages and cookie setting that would vary depending on the success / failure of the operation. The script approach grew messier and the tests less reliable. Pretty soon I had a test of register - login - logout - log back in that just wouldn't work. I could see the web page on the screen updating, but the tests were either telling me that it hadn't changed; or that the cached copy of the page was stale. Later (with lots of await statements and some pruning of the tests) I had passing tests on my laptop which broke on CI.

The details of the route out aren't important, although there is a snapshot encapsulated on StackOverflow. The eventual approach is the interesting thing:
First, embracing FluentLenium and its approach and not poking about with the underlying Selenium. There was a big refactoring into using the Page design pattern, and @Page annotations. This, obviously, started with commenting out the breaking code and reworking the things that I already had working. The separation between user-level script and web-page-internal tests really helped clarity. There was enough similarity between the two test classes that they started to share some code in a super class (mostly starting the driver); and also between the pages (all the static final Strings that refer to the web page and tests for logged-in state that come out of common framework HTML rather than being page specific). Indeed, in a site that uses templates this is to be hoped for.
Next, using the @AjaxElement. Well, I'd tried that along the way, but it didn't help. Because of the next thing. But, as part of the overall picture it works and makes the rest work!
Then, simplifying the tests. The scripted approach encouraged adding bits to the stories represented by the test. I needed to remember that the existing test was testing something. The additions were new tests. In particular this meant I was coming back to @Page objects and expecting change, but finding breakage. Each page field could only be used at one place in the test; once navigation moved on the field was no longer helpful. Getting an @AjaxElement works once; not for a whole string of prod / test cycles. Having more than one field for the various visits to a given page was considered, but could always be avoided so far by having the tests test just one thing.
Finally and most trickily, making the assert call to test that the @AjaxElement is what it ought to be from the test object rather than within a method of the page object. Not an immediately obvious step. So this (for me) is broken:
class Page {
  @AjaxElement e;
  public void checkE() {
class ETest {
  EPage p;

  public void test_eStuff() {

and this is not:
class EPage
  @AjaxElement e;
  public void getE() {
    return e.getText();
class ETest {
  EPage p;

  public void test_eStuff() {

There was also some fixing of HTML id tags to aid testing that went on, but no real change to the pages (although I did contemplate this at one point).
Since that breakthrough I've added another two tests of user account activity, each with as much ease as I added that second test to load the about page. The code reads well and the collection of classes isn't expanding so fast. Tests still pass on CI. I'm happy. New features can be developed against tests like this, in outside-in style.

Saturday 25 October 2014

Cut and Paste

I replied on Twitter to Grady Booch and Valentin T Mocanu earlier (Booch in turn was commenting on a Quora answer). Being Twitter it was a bit brief and slightly flippant. The thrust was "cut and past programming is fine" ... "fine for the first 2-3 years".
My contribution: using example code teaches what you ask of it.

I've learnt a lot with examples; I've taught a lot with examples. But, what I learnt by using examples was not how to touch type from a magazine, nor how to copy and paste from examples. Like most programmers that work with others, I look at what others write. Sometimes it gets me through a block. Sometimes I see something new. As I use it I mentally fit it to my existing models. Then I prod what I've got, see what breaks it, how it changes, what else the documentation says about this new thing.

When I use a new library or service I often reach for example code. OOP is very good at giving us generality: lots of classes, lots of methods. But the first use of something is usually pretty close to the common case. API documentation rarely describes that (many thanks to those that do!) and simplifying the common case sometimes gets lost in the quest for elegance. Examples show that path. Then, as before, documentation can be explored to vary from this. Exploration gives a structure to build a model from.

Testing, of course, is a great tool for doing this exploration, prodding an idea and seeing what happens. Asking "my code does X, but what I really want is Y, how do I get there?".

Blind cut and paste, of course, teaches next to nothing and doesn't solve all problems. You can get by on it, but it would be hard to tread a new path that way. I see this frequently, and in many ways it puzzles me. One of the great attractions of programming, to me as a kid, was that I had an infinite set of problems to solve to make the computer do something fun. And, unlike electronics which I also enjoyed, the chances of my programming breaking the computer in a way I couldn't fix were small. (Of course if you ask the 13 year old me you wouldn't get that as an answer!) But, some people get the fear when you suggest that they try something new with some example code. I don't think it is fear of breaking it that is the barrier really, but a self-perception of incompetence - much as is involved in shyness. A lack of belief that they have the mental tools to make that exploration, or that their tools aren't as sharp as others' tools. Leading people into sharpening those tools, and believing that they can do this is quite a lot of what teaching programming comes out as. Give people the confidence to have a go, some pointers and feedback, and a few principles for the next round of exploration and pretty soon you have Kolb's learning cycle.

The other part of my answer was "Also good for using libs that are short on principles themselves." Maybe more than slightly flippant. But, occasionally, I use some library or tool, and find myself wondering how it works. An example gets me started. But straying from the one true path leads to failure. Sometimes there's a principle which has been broken or hidden or isn't the principle I was hoping for. Sometimes it's all a bit ad-hoc, a "tool" extracted from a single project that hasn't yet evolved principles and general applicability away from the use-case of the original project. Sometimes there's a principle, but it is unevenly applied.

I'll finish with an example: Java's ImageIO Input and Output streams and the standard Input and Output streams. Getting an Image written to an output stream and passed onto an input stream elsewhere, without specifying writing to a file, really could to be easier. The ImageInputStream and ImageOutputStream don't extend InputStream and OutputStreams, indeed an ImageOutputStream is also an ImageInputStream while the non-Image versions don't relate like this. *NIX is deep enough in my blood that I want pipes to do both input and output and just work, but in this problem the streams are in the same place and Java's PipedStreams need mucking about with threads if there's much data (and plugging a piped pair of streams into an ImageOutputStream without extra threads certainly broke for me). So, here's my example (arrived at by seeing several other examples, relating to my mental model of images and I/O, exploring, adjusting and combining):
  ByteArrayOutputStream os = new ByteArrayOutputStream();
  doRender(data, os); 
    /* method arranges the rendering object with the right parameters leading to
       ImageIO.write(theRaster, imageType, outStream);  */
  byte[] imgBytes = os.toByteArray(); // you know where you stand with an array
  ByteArrayInputStream is = new ByteArrayInputStream(imgBytes); // clunk
  docStore.putData(data.getName(), RENDER_MIME_TYPE, is, imgBytes.length);

This works - the rendered file ends up on my docStore (S3 in this case). But there's a part of me that would really rather not see that array. My images are (already) 60-80kB. I'm aware that at some point memory use may prompt me to revisit this (but lets not over-engineer it yet).

So, there you are: an example for you, which I wrote (today as it happens) by learning from a mix of examples and API documents (some of that learning happened years ago, of course). Whether the example is necessary because I misunderstood the principles of the libraries, or because the libraries haven't best applied available principles I'll leave to you to decide!

Friday 10 October 2014

CouchDB Install

I've installed CouchDB a couple of times recently, for development VMs and then adding it to an existing Jenkins CI image now that my project has a test that needs it. This isn't as simple as "yum install..." yet, at least for the systems I've tried - although an Ubuntu package is available. Continuing the tradition of this blog documenting installations, below is a log of the Jenkins slave install, which follows what I did on my CentOS VMs but without the mistakes and dead-ends. This runs on an AWS EC2 t2.small instance, with a configuration including Amazon's 64bit Linux, Ninja Framework and PostgreSQL mostly described in previous posts. So, I assume you have a basic machine up and running.

First some package installs, mostly as advised by the documentation on the other packages to install, but starting with the inevitable update:

sudo yum update
sudo yum install autoconf curl-devel ncurses-devel openssl-devel
sudo yum install gcc gcc-c++ libicu-devel

Erlang is built from scratch: to get a current version and to avoid a crypto fail in some Linux distros (including CentOS which I have on my development VM, which allows erlang and couchdb to compile and install, but fail on starting couchdb). Couchdb is compiled against this later.

curl -O 
tar xzf otp_src_17.3.tar.gz
cd otp_src_17.3
export ERL_TOP=`pwd`
// make a cup of tea
sudo make install
cd ..

Next, we need Python, in particular the 2.7.x version, and SpiderMonkey, again not the newest one, but version 1.8.5:

curl -O
tar xzf Python-2.7.8.tgz
cd Python-2.7.8
sudo make install
cd ..
curl -O
tar xzf js185-1.0.0.tar.gz
cd js-1.8.5/js/src
// start to drink tea
sudo make install
cd /etc/
sudo vi local-x86_64.conf
and add the entry: /usr/local/lib, so that the js libraries are picked up.
sudo ldconfig

And now to CouchDB. Note that they have mirror sites, the one I used might not be the most appropriate for you.

sudo adduser --system --home /usr/local/var/lib/couchdb --no-create-home --shell /bin/bash --comment "CouchDB Administrator" couchdb
curl -O
tar xzf apache-couchdb-1.6.1.tar.gz
cd apache-couchdb-1.6.1
// tea is about right now
sudo make install
cd /usr/local/var/run
sudo chown -R couchdb:couchdb couchdb/
sudo chmod 0770 couchdb/
cd ../log/
sudo chmod 0770 couchdb/
sudo chown -R couchdb:couchdb couchdb/
cd ../lib
sudo chown -R couchdb:couchdb couchdb/
sudo chmod 0770 couchdb/
cd ../../etc
sudo chown -R couchdb:couchdb couchdb/
sudo chmod 0770 couchdb/
cd rc.d
sudo cp couchdb /etc/rc.d/init.d
sudo service couchdb start
sudo chkconfig --levels 235 couchdb on

To check it is working, curl should respond with something like:

{"couchdb":"Welcome","uuid":"d420a7e9b585cb74039a54f1874b6a0f","version":"1.6.1","vendor":{"name":"The Apache Software Foundation","version":"1.6.1"}}

Finally, tidy up the download and build directories; make a new AMI; and point the Jenkins EC2 plugin's config for the slave at that AMI; and test. For the purposes of a short-lived CI slave whose firewall only talks ssh password protecting access with new admin users is probably overkill. However, for my application there is a database creation script which makes some assumptions about admin users; and some test cases which assume a test database. So, in the Jenkins config for the job pre-steps we have:

curl -sS -X PUT http://localhost:5984/_users/org.couchdb.user:dan -H "Accept: application/json" -H "Content-Type: application/json" -d '{"name": "dan", "password": "mumblemumble", "roles": [], "type": "user"}'
curl -sS -X PUT http://dan:mumblemumble@localhost:5984/test

And observe the test passing!
For completeness, that first test is below. It doesn't do anything terribly clever: posts a new document on CouchDB and gets it back. However, it re-uses the NinjaProperties fake idea from my previous post about fakes and injection, so completes two loops!
package models;

import java.util.HashMap;
import java.util.Map;
import java.util.UUID;
import models.PersistCouchDB.IDRev;
import org.junit.AfterClass;
import org.junit.Test;
import static org.junit.Assert.*;
import org.junit.BeforeClass;

 * @author Dan Chalmers
public class PersistCouchDBTest {
  public static final String DB_NAME = "test";  // note, this needs to be created on CouchDB in the test environment!

    static PersistCouchDBStub db;
    public void test_postNewDoc() throws BrokenSystemException {
      HashMap map;
      String randomString;
      IDRev postResult;
      Map getResult;

      map = new HashMap();
      map.put("testing", "1two3");
      randomString = UUID.randomUUID().toString();
      map.put("random", randomString);
      postResult = db.postNewDoc(DB_NAME, map);

      getResult = db.getData(DB_NAME,;
      assertEquals("1two3", getResult.get("testing"));
      assertEquals(randomString, getResult.get("random"));

   static class PersistCouchDBStub extends PersistCouchDB {
     PersistCouchDB is abstract as it just 
       - reads the relevant Ninja configuration 
       - and contains helper methods to arrange the basic access patterns.
     This works, but doesn't do anything very helpful with real data.

  public static void createDBObject() throws BrokenSystemException {
    FakeNinjaProperties props;

     props = new FakeNinjaProperties();
     props.put("couchdb.readserver", "localhost:5984");
     props.put("couchdb.writeserver", "localhost:5984");
     props.put("couchdb.connection.username", "tripvis");
     db = new PersistCouchDBStub();

Saturday 6 September 2014

Ninja, Injection and Fakes

The Ninja Framework has a convenient system for loading properties from a configuration file, application.conf. This includes a system of modes, where a property can be prefixed with one of three modes: dev, test or prod. So, when Ninja is running as a web server the configuration can easily be adapted to the three common situations, while keeping a single configuration file in the version control and CI process. This looks something like the following:


public abstract class PersistDB {

    NinjaProperties ninjaProperties;

    public static final String DB_READ_SERVER_URL_PROP = "db.connection.readurl";
    public static final String DB_READ_SERVER_URL_DEFAULT = "jdbc:postgresql://localhost:5432/tripvis";
    public static String dbReadServerUrl = DB_READ_SERVER_URL_DEFAULT;
    // etc

    protected void loadProps() throws BrokenSystemException {
        if (ninjaProperties != null) {
            dbReadServerUrl = ninjaProperties.getWithDefault(DB_READ_SERVER_URL_PROP, DB_READ_SERVER_URL_DEFAULT);
            // etc
        } else {
            logger.log(Level.SEVERE, "No NinjaProperties created, database config is stuffed");

The injection saves creating the object which deals with NinjaProperties and loading the file, which is very handy. However, there is a downside. When running JUnit tests the Ninja framework isn't loaded in the normal way, so the relevant classes aren't constructed by the framework but by the test object. (Yes, yes, I know, my example accesses a database, so this is stretching "unit" a bit.) This means that the injection no longer happens for free. Indeed, you're stuck with the hard-coded defaults. This is an issue in several ways:
  1. The test doesn't exercise the loading of the properties at all.
  2. If the properties required by the test environment change then the code has to change.
  3. If the developers' unit testing environment and the CI environment aren't the same then this approach stops working.
Fakes come, more or less, to the rescue. NinjaProperties is an interface, so we can create a fake properties object instead of the real one. This looks something like the following:

public abstract class PersistDB {
    public void fakeLoadProps(NinjaProperties props) throws BrokenSystemException {
        logger.log(Level.INFO, "loading fake properties");
        ninjaProperties = props;

public class UsersDBTest {
    static UsersDB db;

    public static void createDBObject() throws BrokenSystemException {
        UsersDBTestFakeProps props;

        props = new UsersDBTestFakeProps();
        props.put("db.readserver", "localhost");
        props.put("db.writeserver", "localhost");
        props.put("db.connection.url", "jdbc:postgresql://localhost:5432/tripvis");
        props.put("db.connection.username", "dan");
        props.put("db.connection.password", "YoHoHo");
        db = new UsersDB();

public class UsersDBTestFakeProps implements NinjaProperties {

    private HashMap props;
    public UsersDBTestFakeProps() {
        props = new HashMap<>();
    public void put(String key, String value) {
        props.put(key, value);
    public String get(String key) {
        return props.get(key);

    public String getWithDefault(String key, String defaultValue) {
        if (props.containsKey(key)) {
            return get(key);
        } else {
            return defaultValue;
// etc for any other methods needed for the tests

More work than injection, but mostly confined to the tests and not that tricksy.

Note, this is just a fake - a simple class that fulfils the interface, but takes a big short-cut. It is marginally more clever than the canned response from a stub, but doesn't go as far as checking that the right calls are made like a mock would. The mock approach would help test that the right queries are being made, but at the moment they're all bundled together in the loadProps() method so we're a good refactoring step away from that right now. For those that aren't familiar with the mock / fake / stub distinction I'd recommend reading Martin Fowler and Emily Bache.

Recalling the issues from before, I'm left with some caveats:
  1. The test doesn't touch the properties file at all. You still need to do some system testing to do that. But this approach flags an issue in code (rather than config) faster than that will.
  2. If the properties required by the test environment change then the test code has to change. Could be worse. If this is really an issue then the fake could load a file.
  3. If the developers' unit testing environment and the CI environment aren't the same then the code above needs tweaked to spot the environment and load different values into the fake. A bit fiddly and loading a file might start looking clever as soon as this gets at all involved, e.g. with more than one developer.
Final thought: Injection is kind of cool at runtime, but not without its own issues.

Tuesday 19 August 2014

Comment on "An Appeal to CS Teachers"

I recently read An Appeal to CS Teachers on 8th Light which talks about recent graduate programmers using far too many comments. This resonated with some experiences I had, and links back to an earlier post here, Programming 102, on teaching Java and Unit Testing. In this article Dariusz Pasciak comments on the problem of students commenting every line of code, and proposes two reasons that this might be happening:
It helps teachers read the code. 
It forces students to think more about what they're doing.

Here's my response...

I've taught university CS programming (most recently Java to 1st years, but programming in the context of other topics and a few other languages over the years) and usually give (just a few) marks for "appropriate" comments.

Particularly for the early years of a degree I do find I have to guide many of my students towards "useful" comments, rather than per-line or somewhat random commenting. A few are doing this because they are working out what is happening and worry about coming back to something that makes no sense. A few have been told somewhere in the past to comment everything, as suggest in Dariusz's post. Some are just erring on the side of caution, reasoning that "if Dan likes comments more can't hurt, right"?

The two rationales are interesting. 
It helps teachers read the code.
First, I should say that most students write coursework solutions that really aren't hard to understand, although in more openly specified projects why they have chosen to write the code like that maybe less so. However, the ones that are hard to understand are often those that have missed the point somewhere along the way and are most in need of some understanding and helpful feedback. For helping me understand what the students have written I've moved to unit testing.  Even the baroque enthusiasts and somewhat confused can write fairly understandable tests.
It forces students to think more about what they're doing.
As Dariusz says "Students just beginning to pick up programming already have a lot to think about.". Quite. There isn't any need to make life harder, it only ends up obscuring the point. The unit testing also works for thinking about what they're doing, but separating the thinking step out. More thinking when you're writing code that's at the edge of your ability isn't going to end well for anyone. Have a think then do the doing. Write a test; make it pass; refactor.

My standard advice to students is something like:

  •  Write me a test case, that describes the intention of the method pretty well (unless we're talking about trivial getters, setters etc).
  •  A good class / method / variable name removes the need for a sentence of comment. That may not remove the need for comments, but eliminates stuff that just clutters the code up.
  •  JavaDoc style per-class and per-method comments are useful for an overall picture and also for describing how to use the code, which the code itself might not make so obvious.
  • By all means comment some non-obvious step, clever trick or thing to come back to. Probably at a block within a method level. Then think about whether that bit would be better refactored into something else. At that point the comment may well vanish.
So, I'm with him: no need to hide the code in a flood of comments!

Friday 1 August 2014

Softly Softly

It has been nearly a month, so what progress?

Firstly, I've had a lot of admin to deal with. Most necessary, little passes the "interesting things with interesting people" test, none merits writing further about right now!

I've been working on a website in my spare time, using Ninja (discussed elsewhere). There's been quite a lot of learning along the way (a bit more about actually using Ninja, finding my way round Bootstrap, learning how Injectors work, revisiting graphics programming I last did many years ago, weighing up design approaches...), and at some point I hope to distill a post of things that are useful, repeatable and not covered thoroughly elsewhere out of this. Right now that isn't ready. The next step involves me broadening my NoSQL horizons a bit. I can see the steps from here to a first public version, just need to find the play-time to take them.

Another project has involved a lot of looking at licences and other legal stuff, and considering how that fits products that are still being developed and a lot of "what ifs" about future investors. It isn't the technical side of building a thing, nor the working for people. It matters, a lot, to the business though so it merits the investment in time. We've also had some really excellent partners giving us advice, so thank-you to them. However, there's very little I ought to openly say about the detail. We've been "nearly there" many times over quite a long time, but today some of the deals were signed. Which is a relief.

So, the blog isn't dead, I haven't had an extended holiday and done nothing, nor taken up basket weaving. But it has been a "bunch of stuff" sort of month, rather than a "here's a thing" sort.

Wednesday 2 July 2014


The Open Rights Group have produced the Blocked! web service that allows you to test whether sites are being blocked by the major U.K. Internet Service Providers. The idea is classic government "think of the children" without thinking of the clumsiness of the technology, the harm that could do and the weak level of protection it does provide. Really, the web has to be controlled with something more sophisticated than a light switch. And, controversially, a switch that doesn't really work creates an illusion of protection which if anything will reduce the more effective measures, such as parenting.

As if to prove the point, TalkTalk are blocking this blog! That's a significant number of people that are unable to access my subversive and corrupting influence. I've put in a request to unblock.

Friday 27 June 2014

Not A Polymorphic Method

Since the first version of this post I hit on a much tidier solution, and the post has been updated to reflect this.
I was writing a renderer yesterday. I haven't done this in a long time, but ended up at a problem I remember not finding a nice solution to before (probably c. 2000!). I think that the solution I came up with yesterday was nicer than 14 years ago, which is good. However, Java 1.7 didn't let me do quite what I wanted...
I've got some features to draw. Using inheritance they derive from a base class. The features are created by parsing some input and may be numerous and long lived. The details are unimportant here, so we'll reduce them to:
abstract class Feature { ... }
class FeatureA extends Feature { ... }
class FeatureB extends Feature { ... }

I have a separate rendering class. This walks over the features and draws them, as you'd expect. I want to be able to decouple this from the building of the feature set. When I say decouple I mean separate machines, possibly repeated renderings, possibly using different rendering engines on the same data over time. One intuitive (to me) way to write this is:
class RenderEngine { 
  List features;

  void render() {
    for (Feature f : features) {

  private void renderFeature(Feature f) {
    // Feature is abstract, this just catches unimplemented renderings

  private void renderFeature(FeatureA f) { ... }
  private void renderFeature(FeatureB f) { ... }

Of course this doesn't work! Only renderFeature(Feature f) gets used. Java doesn't give me choose a method implementation by the dynamic type of the arguments within an object.

So, what to do?
I really don't want to put the rendering code into Feature - it is busy enough describing the feature and helping the parser abstract from the raw data.
The Decorator pattern isn't really applicable as Feature and rendering have different purposes and different base classes.

I could use a sequence of test, cast, calls:
if (f instanceof FeatureA) { renderFeatureA((FeatureA)f); }
It worked. But, to me, it also smells. Long lists of tests for classes feels like I've used inheritance then broken it, and it would be easy to muck up adding features later. Also, all these tests would be going on in a loop.

The next solution I ended up with is influenced by the fact that all features of a given class will be rendered the same. No exceptions. The Command pattern is the starting point. I can refactor my rendering methods into rendering classes. The methods only use a very little state from the RenderEngine class so the objects will be easy to set up. There will be a lot of feature objects of few types so I don't want to tell each object about the renderer individually - especially as I can't be sure of the renderer at creation time. The trick was to use a static variable to hold the reference to the rendering object from the feature class. However, note that the following is also "broken" as a solution:
abstract class Feature {
  static RenderEngine render;

The sub-types use the one static variable in Feature so both return whichever one is assigned last. This is also the downside of the approach - if the rendering of a feature depends somehow on the oveerall rendering of that task and there may be multiple concurrent tasks then the statics are broken.
Java 8 would give abstract static, but I'm not using that here (yet).

At the point the rendering object is assigned the class of the feature is definitely known. I can just use a method in the feature sub-types to assign to a field of those classes. Using inheritance to avoid the duplication would be better, but the code is trivial. The getter is then defined as abstract in Feature and written for each Feature sub-type. So I end up with code roughly as follows:
abstract class Feature { 
  abstract FeatureRender getFeatureRender(); 

class FeatureA extends Feature { 
  static FeatureRender renderA;
  static void setRenderObject(FeatureRender ro) {
        renderA = ro;
  public FeatureRender getRenderObject() {
    return renderA;
class FeatureB extends Feature { ... }

abstract class FeatureRender {
  abstract void render() ;

class RenderA extends FeatureRender { ... }

class RenderEngine { 
  List features;

  void render() {
    RenderA ra = new RenderA( ... );
    for (Feature f : features) {

My Feature classes are now aware of rendering, but don't hold any of the code to make it happen. I have a sequence of calls to set the rendering before the render loop, and at each iteration I get the rendering object. I think that is more elegant than the sequence of test, cast, call that I had in the rendering loop before. Adding a new feature still involves adding a line of code to set the rendering object, although arguably cleaner and more efficient code than the instanceof test. One bonus of splitting the rendering methods out into their own classes is that the little collection of constants for each goes with it. None is a big step, but together they make RenderEngine much tidier than it was before.

The statics are a problem however, as this is now part of a web service, rather than a stand-alone application. So, I might be rendering with two different rendering classes at the same time - for different outputs. This required a change of direction. One which, having arrived at it, I quite like. In essence, the RenderEngine takes over responsibility for mapping feature objects to rendering objects - but without the clumsy instanceof tests. The solution looks roughly as follows:
class RenderEngine { 
    protected Map renderingMap;
    public void setupRendering() {
        RenderFeature rf = new RenderFeatureA(sizing);
 renderingMap.put(FeatureA.class, rf);
        // and similar for other features, also set any parameters due to overall document
    private void renderFeatures(Graphics2D g2d) {
        for (Feature f : features) {
     renderingMap.get(f.getClass()).render(g2d, f);

Also, this approach has greatly simplified the various feature classes: they no longer need to refer to rendering at all. This means that the rendering is in a more coherent group of classes and the coupling between the feature data and feature drawing is much reduced; in particular the data doesn't need to be aware of the drawing any more. This decoupling would also have been present in my initial, impossible, approach - it was only adopting the command pattern that created it.
And the obligatory testing note: I was writing tests as I went. But unit testing graphics is hard so it didn't pick up the failure to draw the right thing. Now I have it working I could test by example, but first time round I couldn't. Even so, I did have a test of running the rendering so it made a tight cycle of refactoring through the various solutions with the aid of a logger and debugger. I could quickly see when it worked and when it didn't, and when my changes broke something else.

Friday 20 June 2014

Ninja Web Framework: Hello World

In a little bit of spare time this week I've been trying out a new thing. We've been using the Play framework in one project, with some success. However that bit was led by someone else and while I had seen the basics I couldn't really claim to have used it myself. It was time for me to explore it a bit more deeply for something new, rather than straight servlets on Tomcat. However, it seemed to be hard work:
Java isn't first choice for Play. That's OK, but I wanted to use Java.
I have a mild preference for NetBeans, partly because I find the UI more intuitive, partly because I find getting NetBeans and Jenkins to use the same build file easier than in eclipse. NetBeans has some Scala support but doesn't fully integrate with Play.
Finally, working in an environment where I run a VM on my laptop for development testing is a bit clunky in Play. (I have grown bored of untangling the software installs for a mix of projects at the MacOS level. New project, new local linux VM.) Play just seemed to be more involved in the writing step than felt natural to me - I like to be able to pull my code back from the infrastructure.
The "hello world" step is often the hardest with a new tool, but this was clunky enough for me to look around.

I soon came across the Ninja Web Framework. Their selling points are: simple to program in Java web framework; easy to use an IDE; works with testing and CI; talks JSON and html; built-in authentication and email. Particularly attractive was the model of development which doesn't need a lot of server setup, with Maven doing all the package management beyond Java and Maven itself. The "works with testing" point merits expanding a little. In the configuration there is a built in notion that you might want different configurations for development, test and production environments. I can see that making life easier.

There was quite a lot of package management for maven to do - it spent a while being busy when creating the project and again when first building / running. Checking back with mvn -o dependency:list reveals 116 packages from a variety of sources. This complex set of dependencies worries me slightly. Projects which die or become radically different will have a knock on effect. The rate of regression testing that might arise as changes happen will make good configuration management a must. And because if any of the imports gets used again in our / some other code on the same server the opportunity for evolving to run multiple versions of one 3rd party package isn't pretty. But I'll set this aside for now and see how it goes.

As I said above, "hello world" is often the hardest step with something designed to be very general. The overhead over the simple approach is wince making. However, the documentation stepped me through the basics and I soon had a web page up. Then I changed the text. Then I added a page. Then I spoofed a session. And so on, little tweaks to learn my way around. Commenting out some HTML broke it for a while, but mostly it did what I expected! And in the end the structure is pretty logical and not over-complex, where over-complex is defined as "requires more than one page in my notebook":

Routes map URLs (optionally with regular expressions and variable parts) to controller methods.
Controllers can call other code and build a response. There are also lots of annotations, for filters, brining in config file parameters, parsing JSON, setting session cookie data etc.
The view is achieved using Freemarker templates. These can import other template files, making consistent headers and footers easy. They work with i18n properties files. The rendering of the result in the controller can pass in parameters which are picked up in the template.

Returning to the issues I had in the first paragraph:

Java just works.
NetBeans (and other IDEs) just work.
The IDE issue is at least in part fixed because of the looser coupling between code and a running Ninja environment. 

So, I haven't done anything very clever with it yet. But I think I know where I'll try using it on a proper project soon. I'll report back when I've got a fuller picture!

Friday 13 June 2014

Programming 102

Posts here have been conspicuous by their absence for a while, but things have been moving on. My energy has been focussed on delivering a revamped "programming 102", so development and business have been very much part time. I've just finished the marking and thought that would make a reasonable basis for a few observations.

The course teaches Java, building on a previous Java course. So the real basic basics have been covered. Whether everyone's got them or not is another thing. Any programming course has a tail of participants that, for various reasons, just don't quite click with the ideas. Like the difference between classes and instances, and how you go from one to the other. At the other end, some have pretty well developed skills already and need to be kept interested.

I mentioned a revamp, and apart from updating against Java versions and my own style of presenting the main change was to push test driven development. This isn't radical, but it was new for (I think) all of these students. The previous course had used BlueJ, with its very visual define / construct / inspect flow. Which I think it great for getting started. Testing makes more sense with a complex program that changes - which was the end goal. However, it provides a useful analogy to the visual approach with our more basic problems. At some point constructing objects and checking what they do by hand gets long winded. The few that hung onto BlueJ hit this about half way through. Testing allows that "I want one of these, one of those, to set that variable, and see what this method returns" flow to be scripted. Making the leap that its the same thing, done in a different way, needs a pushing - if you're finding coding hard then testing can feel like yet more code with this new set of commands to learn. If you've programmed before then for simple problems the value of getting out of the hacking rut can be a hard sell - and tests are written second for the marks. But, I think a significant proportion of my class found problems where "what do I want this to do?" or "this test lets me test a sequence of commands that can then be plugged in to the real code" was a useful step and even more found problems where making their code work was complex enough that the JUnit approach was more efficient than an ad-hoc testing approach.

So, for a first go, I think that worked. Plans for next time include videoed coding demos. Watch this space for a sample! All that teaching makes me want to stretch myself in a different way, and with a bit of luck learn something new. With a bit of luck something bloggable.