Monday, 12 August 2013

Jenkins Slave on EC2

Our Jenkins master running on an AWS t1.micro has run out of memory for running the more complex integration tests e.g. play server, mongodb, headless X and firefox. Rather than have the cost of a larger instance all the time when pushes to version control are occasional (cough) we will run a slave for these tests that only costs when there's a test to run. A simple JUnit run through can still work on the master. The Jenkins Amazon EC2 plugin makes this easy, and handles temporary instances.
This install a lot like the earlier post on creating the master, so I'll skip most of the screenshots, but fix a few things too:

  1. On the master, stop all the extra services (databases, X etc) running with service and chkconfig. That leaves more memory for basic compile / unit test cycles and fewer potential holes for a site that needs to be up all the time.
  2. Create a new 64bit instance which deletes its root filesystem on termination. Auto-delete is much cheaper than tidying up manually. I went with a 10GB filesystem.The 64bit install means that there are some x86_64 in place of earlier i686 choices. The firewall need only open ssh as this is what Jenkins uses.
  3. Do the package installations and a first start / stop pass for each and a manual git list, to get the first time prompts out of the way. Generally I'm accepting all the defaults here and using the default filesystem layout. I try to keep the package order the same as on the master, so that UIDs match between the two and make sure ec2-user and/or jenkins own the necessary files.
    sudo yum update
    sudo ln -sf /usr/share/zoneinfo/Europe/London /etc/localtime
    sudo yum install erlang
    sudo yum install rabbitmq-server-3.1.3-1.noarch.rpm
    sudo service rabbitmq-server start
    sudo service rabbitmq-server stop
    sudo yum install mysql mysql-server mysql-libs
    sudo service mysqld start
    sudo service mysqld stop
    sudo yum install mysql-connector-java
    yum search java | grep 'java-'
    sudo yum install java-1.7.0-openjdk.x86_64
    sudo alternatives --config java 
    sudo yum install tomcat7-lib.noarch tomcat7-servlet-3.0-api.noarch tomcat7.noarch
    sudo wget -O /etc/yum.repos.d/jenkins.repo
    sudo rpm --import
    sudo yum install jenkins
    sudo mkdir /var/jenkins-slave
    sudo chown ec2-user:jenkins /var/jenkins-slave/
    cd /etc/yum.repos.d
    sudo vi 10gen.repo
    cat 10gen.repo
    cat 10gen.repo
    name=10gen Repository
    sudo yum install mongo-10gen mongo-10gen-server
    sudo service mongo start
    sudo service mongo stop
    cd /usr/local
    sudo unzip ~/
    sudo chown -R ec2-user:jenkins /usr/local/play-2.1.2/
    /usr/local/play-2.1.2/play test
    sudo yum install ant ant-junit
    sudo yum install git
    ssh-keygen -t rsa -C "jenkins@aws"
    cat .ssh/
    // add the key to Assembla's list of keys in your profile
    git ls-remote -h HEAD
    sudo yum install Xvfb
    vi ./gtk-firefox
    insert the code from
    chmod 755 ./gtk-firefox
    sudo ./gtk-firefox
    // do something else for a bit  
  4. Stop the EC2 instance and create an AMI of it.
Now, the slave can be set up in Jenkins. First, install the Amazon EC2 plugin. Then use the global configuration to set up the slave's properties. The idle termination time of 30 means it has time to boot, run some tests and get shut down within a one hour billing time. I also use this for starting services on the slave:
Then, in the job configuration use the " to add the label for that slave. Jobs requiring different services can have additional AMIs defined under the Jenkins global config. Note that you can have multiple AMIs under Jenkins point to the same AMI image on AWS, which may reduce the number of EC2 configurations to be maintained.

If you're at all unsure about the config it doesn't hurt to do a test cycle by hand: pull from git, compile, test and tidy up.


  1. The wget of firefox in the gtk-firefox script seems to have broken since I wrote this. However, the page it attempts to download is still correct and links to the tar file. So the simple solution is a manual download and comment out the relevant line.

  2. Particular versions of Selenium are designed to run against particular versions of Firefox. I'm using FluentLenium which requires Selenium 2.42.2. At the time of writing the latest firefox doesn't work against this dependency. (It also requires a couple of other packages to be installed beyond the ones in the script above.) Version 28 does work. But as the CDN breaks the wget in Joe Killer's script selecting the right one is easy (once you know which is the correct version).