Wednesday 20 January 2016

make-ing docker

I mentioned some time ago that I've been exploring Docker. I then changed my main job and the blog went rather quiet. I have a contract that gives me some time and IP to myself, and this is finally turning into a new project. Between personal projects and work I've spent a whole lot more time with Docker. So, a blog post, about building Docker images, that might be useful to someone...

A bit of background:
First, I'm a fan of make. I know, it shows my age. But, it's really quite good at handling building tasks with a minimum of tricky requirements on the build system. It can also be turned to organising installations and running jobs, so keeping related issues in one version-controllable place. It is also in a format that's quite user friendly, even when ssh-ed into a server and fixing stuff with vi. So, while I've also used Ant, Ansible and Maven to do some of these things, it remains a reliable standby.

One of the things I use Docker for is setting up groups of images which are related to each other: django; django with a different config; django with that config, but setup for running tests rather than running the server - and so on. Which leads to dependencies.

Most of the time when I use Docker, its in a local environment, and I just build and run images on one machine - without using Docker Hub. I like having the tool chain version controlled, and this approach fits.

Make is good at handling dependencies, but the relationship chains in Docker image definitions don't expose themselves in file names (unless you're very organised with naming schemes). The quick first approach is to make them with explicit pointers in the make file as well as in the Dockerfile. But eventually that level of duplication will irk. So, in a spare couple of hours to polish my build scripts, I refactored the duplication out of the Makefile.

The key parts of the makefile are below - stripped of my builds to show the principle:

DIRS := $(shell find . -mindepth 1 -maxdepth 1 -type d)
DOCKERFILES := $(addsuffix /Dockerfile,$(DIRS))
IMAGES := $(subst /,,$(subst ./,,$(dir $(DOCKERFILES))))
FLAG_FILES := $(addprefix ., $(addsuffix .docker, $(IMAGES)))
PWD := $(shell pwd)


# Docker images can depend on each other.
# A changed base image ought to trigger a rebuild of its children.
define image_dep_search
@echo "checking dependencies of $1"
@for d in $(IMAGES); do \
 from=`grep FROM $$d/Dockerfile | cut -d ' ' -f 2`; \
 if [ $1 = $$from ]; then \
  echo "dependent image $$d"; \
  touch $$d; \
  make .$$d.docker; \
 fi\
done
endef


all: images 


# Consider all docker image directories for building
images: $(FLAG_FILES)
 @echo "Done making images."

# Build images where the directory or contents have changed since flag last set
.%.docker: % %/* 
 $(eval IMAGE = $(subst .,,$(basename $@)))
 $(eval BASE = $(word 2,$(shell grep FROM $(addsuffix /Dockerfile,$(IMAGE)))))
 $(eval HAS_DEP = $(filter $(BASE),$(IMAGES)))
 @echo "building $(IMAGE)"
 @cd $(IMAGE) && docker build -t $(IMAGE) .
 @touch $@
 $(call image_dep_search,$(IMAGE))


# Utility make targets for creating containers from images 
.PHONY: run_java_bash
run_java_bash: .java_base.docker java_bash_container

.PHONY: java_bash_container
java_bash_container:
 docker run --rm -v=$(PWD)/..:/project -it --name java_bash java_base bash

clean:
 @rm -f $(FLAG_FILES)

In order, this contains:
  1. Some definitions, which find directories that contain Dockerfiles. Files called ".<imgname>.docker" will be created to mark the latest build.
  2. A definition to use later, that finds the FROM line in the Dockerfile; extracts the argument; sees whether it is one of our images; fiddles the need for a build and calls make on that image.
  3. The standard make stuff, to run a build on each image directory which has changes. Once built any dependent images are found and built using the routine defined above.
  4. A phony target to run the container, to illustrate the point. It double checks the image, in case we're forgetful about running "make all" first. This provides the project root (the parent of the docker directory) as a mounted volume - which may or may not be a good thing, depending on your use case.
  5. A clean target, that gets rid of the build flag files. 

There's a couple of assumptions here:
  1. A directory of docker images within the project. I usually call it "docker". This is the set of docker images that will be considered for dependencies. The rest are just assumed to exist. The makefile is in this docker directory.
  2. The FROM and the image name in the Dockerfile are separated by a space.
  3. Images are flat directories. I think that structure is probably better put elsewhere, and built in an archive format than copying lots of files one by one in the Docker build process - so this hasn't been an issue for me.
  4. If one image depends on having another already built then another makefile rule is needed to force correct order. I'll update this when I've got an update that automates this as well. Simple "list of target" rules may well be needed to build specific sub-sets anyway.
And that's it: build the images same as I build the code, with a minimum of drag on my effort. Easy to call from Jenkins, easy to call from the command line.

Addendum: With a bit of tidying up, and some example Dockerfiles, this is now on GitHub at https://github.com/danchalmers/MakeingDocker