CI with Jenkins, ant, git and Assembla

I've got a little continuous integration (CI) box at home. For what is, at the moment, a one person project CI might be seen as overkill. My excuses are:

1. If/when I'm employing anyone else then things are taking off and CI will be a good idea but likely to be left behind in distraction; 
2. It keeps me honest, by making sure there's no silly dependencies to my laptop and running stats on the code; and 
3. It keeps my hand in with sensible tools. In a similar vein I use Assembla for managing tasks and as a git repository. 

The little box at home has some disadvantages, not least being switched off quite often. With another project wanting CI I've spun up an Amazon Web Services (AWS) EC2 instance to do the job. The basics are there, after a bit of to-ing and fro-ing. So, here's a record of how it was put together, for other newcomers wanting a how-to, but also for my benefit when I need to reinstall as anything! Metrics and graphs will be a separate post some other time.

AWS Config

First, I brought up an AWS EC2 instance. I chose the Amazon 32bit model. Why 32bit? - because I want to run this in a t1.micro (free) instance. I added an EC2 disk, of 2GB, to hold the builds and Jenkins data, that might want to be persistent; and gave it an imaginative name.

 An alarm by email was added to alert me to problems.

 The firewall just allows ssh and a selection of the usual web ports.

 A key pair was created, downloaded and used.

After a couple of minutes the server was running. (Well, the first one was dead on arrival, but this is what you want to see!)

Linux Config

Using the AWS AMI the usual install and post-install tweaking are much reduced needed, which is nice. The steps were:
Use the key generated above to login with ssh, do the initial update and mount the extra disk:

dan-laptop$ cp ~/Downloads/awskey.pem ~/.ssh/
dan-laptop$ cd .ssh/
dan-laptop$ chmod 600 awskey.pem
dan-laptop$ ssh -i ~/.ssh/awskey.pem ec2-user@ec2-blah.eu-west-1.compute.amazonaws.com

       __|  __|_  )
       _|  (     /   Amazon Linux AMI
      ___|\___|___|

https://aws.amazon.com/amazon-linux-ami/2013.03-release-notes/
There are 5 security update(s) out of 10 total update(s) available
Run "sudo yum update" to apply all updates.
[ec2-user ~]$ sudo yum update
Loaded plugins: priorities, security, update-motd, upgrade-helper
amzn-main                                                | 2.1 kB     00:00     
amzn-updates                                             | 2.3 kB     00:00     
Setting up Update Process
Resolving Dependencies
--> Running transaction check
---> Package aws-apitools-ec2.noarch 0:1.6.7.3-1.0.amzn1 will be updated
...
[ec2-user ~]$ df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/xvda1             8256952   1059968   7113124  13% /
tmpfs                   307704         0    307704   0% /dev/shm
[ec2-user etc]$ sudo mkfs -t ext4 /dev/sdb
mke2fs 1.42.3 (14-May-2012)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=0 blocks, Stripe width=0 blocks
131072 inodes, 524288 blocks
26214 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=536870912
16 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks: 
 32768, 98304, 163840, 229376, 294912

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done 

[ec2-user ~]$ cd /etc
[ec2-user etc]$ sudo cp fstab fstab.install
[ec2-user etc]$ sudo vi fstab
[ec2-user ci]$ diff /etc/fstab.install /etc/fstab
2a3
> /dev/sdb    /data/ci    ext4  defaults 1   2
[ec2-user etc]$ sudo mount /dev/sdb /data/ci
[ec2-user etc]$ cd /data/ci
[ec2-user etc]$ sudo ln -sf /usr/share/zoneinfo/Europe/London /etc/localtime

Install various packages that we need to integrate with, and accepting all dependancies:

sudo yum install erlang
wget http://www.rabbitmq.com/releases/rabbitmq-server/v3.1.3/rabbitmq-server-3.1.3-1.noarch.rpm
sudo yum install rabbitmq-server-3.1.3-1.noarch.rpm
sudo chkconfig --levels 235 rabbitmq-server on
sudo service rabbitmq-server start

sudo yum install mysql mysql-server mysql-libs
sudo service mysqld start
/usr/bin/mysqladmin -u root password 'new-password'
sudo chkconfig --levels 235 mysqld on
sudo yum install mysql-connector-java
cd /data/ci
sudo mkdir mysql
sudo /etc/init.d/mysqld stop
sudo cp -R -p /var/lib/mysql/ /data/ci/mysql/
sudo cp /etc/my.cnf /etc/my.cnf.install
sudo vi /etc/my.cnf 
diff /etc/my.cnf /etc/my.cnf.install 
2c2
< datadir=/data/ci/mysql
---
> datadir=/var/lib/mysql
sudo /etc/init.d/mysqld start

cd /etc/yum.repos.d
sudo vi 10gen.repo
cat /etc/yum.repos.d/10gen.repo 
[10gen]
name=10gen Repository
baseurl=http://downloads-distro.mongodb.org/repo/redhat/os/i686
gpgcheck=0
enabled=1
sudo yum install mongo-10gen mongo-10gen-server
sudo chkconfig --levels 235 mongod on
sudo service mongod start

yum search java | grep 'java-'
sudo yum install java-1.7.0-openjdk.i686
sudo alternatives --config java

wget http://downloads.typesafe.com/play/2.1.2/play-2.1.2.zip
cd /usr/local
unzip ~/play-2.1.2.zip
sudo chown -R root:root play-2.1.2/

yum search tomcat | grep tomcat7
sudo yum install tomcat7-lib.noarch tomcat7-servlet-3.0-api.noarch tomcat7.noarch

sudo yum install ant
sudo yum install git

To support selenium for web testing I installed Xvfb and firefox, to run headless. Firefox doesn't install on an AWS AMI so easily, but the script at Joe Killer's blog seems to do the job after a couple of hours of compiling stuff (although it doesn't like a bz2 compression at the end and for my 32 bit install the URL to get firefox from at the end needs changing to http://releases.mozilla.org/pub/mozilla.org/firefox/releases/latest/linux-i686/en-GB/). Xvfb is then

sudo yum install Xvfb
Xvfb :99 -screen 0 800x600x16&

The starting of X can also be done in the Jenkins job later, but doesn't seem to exit cleanly.

Jenkins

I'm using Jenkins as a continuous integration server. It seems to do things I want of it - I haven't been moved to try the alternatives yet.
Configuring Jenkins:

sudo wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins-ci.org/redhat/jenkins.repo
sudo rpm --import http://pkg.jenkins-ci.org/redhat/jenkins-ci.org.key
sudo yum install jenkinscd /data/ci
sudo mkdir jenkins
sudo chown -R jenkins:jenkins jenkins
sudo cp /etc/sysconfig/jenkins /etc/sysconfig/jenkins.install
sudo vi /etc/sysconfig/jenkins
sudo diff /etc/sysconfig/jenkins /etc/sysconfig/jenkins.install
10c10
< JENKINS_HOME="/data/ci/jenkins/"
---
> JENKINS_HOME="/var/lib/jenkins"
80c80
< JENKINS_HANDLER_MAX="20"
---
> JENKINS_HANDLER_MAX="100"
88c88
< JENKINS_HANDLER_IDLE="10"
---
> JENKINS_HANDLER_IDLE="20"
sudo /etc/init.d/jenkins start
sudo chkconfig --levels 235 jenkins on

Then the Jenkins config can happen via the web interface. Manage Jenkins is the start point, where it will encourage securing the install. Take care not to lock yourself out: switch on security, but let users register and do anything. Then create an admin user, and then close registrations!

Add plugins, git and assembla auth for now:

Assembla

I use Assembla for task organisation and as a git repository. The obvious thing to do is to get Jenkins to get code from git when I push a change. I have an integration branch in git just for this. (Most) development happens in a branch off integration and gets merged back in to integration when it passes unit tests. Live deployment, when it happens, will be by merging from integration into a main branch when integration (and happy testing) pass.

For Jenkins to use git get updates from Assembla we need a key with an empty passphrase, which will just be given read permission. The key in ~jenkins/.ssh/id_rsa.pub is then added to Assembla's keys. The sudo trick gets round the fact that the jenkins user isn't designed to be logged-in.

sudo su - -s /bin/bash jenkins
ssh-keygen -t rsa -C "jenkins@aws"
git ls-remote -h git@git.assembla.com:projectname.git HEAD

Now, the key needs to be registered with Assembla:

At this point it should be possible to create a Jenkins job, have it use git to get the code, build and test.

Finally, assembla can be swapped in as an authorisation mechanism. The assembla auth plugin allows the assembla user name to be used to log into Jenkins (not to be the git key). This saves managing another set of user ids. As for the ssh key, this is a step that needs changes on both sides. Additional users are added by creating their own API key and adding them to the list in the Jenkins security page.

Build for CI

For me one of the key benefits of CI is that you're regularly building on a not-a-development-system. Ideally on something somewhat like a deployment system. This means that build scripts, like code, need to have any system specific stuff (paths, ports etc) grouped into a few config files. For me that means ant and properties files. The first couple of builds on the new CI server will fail while you tweak out database users that don't exist, paths that have changed, correct passwords etc. Build config may get its own post sometime later.

To Do

• Get jobs to stop and start their own services (mysql, mongo, xvfb, rabbitmq, tomcat...) as a micro instance doesn't have enough memory for all this lot at once! Indeed, this will probably need a slave with more memory, on demand, for some tests.
• Make an AMI from the system and take a snapshot of the /data/ci filesystem, to speed up reuse.
• Metrics, graphs, alerts...
• Test deployment on the CI box
• Tests which exercise the system via http.