So we've had a pretty decent Continuous Integration setup here at Isotope11 for the past four years or so. We should have had it for longer, but them's the breaks. At any rate, some of our most popular blog posts, by a huge margin, have been related to our Jenkins work, whether it be our post on Monitoring Your Continuous Integration Server With Traffic Lights and an Arduino , our post on Styling Your Jenkins Continuous Integration Server, or my last post on our first, less-appropriate means of Continuous Deployment with Capistrano and Jenkins. The point is, we love Jenkins, we do fun stuff with it, and The Internets seems to like what we do. With that in mind, I'm going to explore the new Continuous Deployment strategy here at Isotope11 for 2015.

The problems with the existing setup

So our existing Continuous Deployment setup, detailed in the above post, has gotten us pretty far with a lot of success. We've never had a situation where a deployment caused us pain. However, there was a pretty big known bug with it the entire time. Here's the 'algorithm' it followed, in a nutshell:

Every time any build passes on any branch, deploy whatever is at the head of master at the moment.

This is stupid, but it was the easiest thing to implement, and: master should always be deployable, and deploys shouldn't cause downtime anyway. If those two cases are true, then there should be no problems. So in general it was a reasonable but stupid way to implement CD, and the whole Internet seemed to like it fine enough so I didn't lose sleep over the downsides.

However, it is a horrible strategy and I wasn't willing to leave it like this forever. So here's the new deal:

The proper thing to do

The above strategy works horribly for also automatically deploying ongoing work to a staging server (or multiple subdomains, one-per-branch, or whatever). This is a desirable feature for us to gain, and January's kind of slow, so I went for it. Here's the new 'algorithm' in a nutshell:

When a build passes on any branch, deploy that particular sha. If the branch that passed was master, deploy to production; otherwise, deploy to staging.

This is not 100% perfect, as it'd be ideal to have a subdomain per branch, teardown of merged branches' deploys, etc., but it's a pretty good setup. It's also a far cry less naive than the previous strategy. Let me show you how we ended up implementing it.



So our project that we used as a testbed for this strategy was itself. It's low-risk, and we're immediately eating our own dogfood rather than doing this on a small project and then not living with the consequences five months later. It's a rails site, and it's deployed with Capistrano, with the deployment happening directly from the site's repo. (For newer projects, we tend to have a separate deployment repo, as that can house things like ansible scripts and whatnot, but this is just a sidebar)

If you're unfamiliar with capistrano and multistage projects, our deploys look like one of two things:

cap production deploy
cap beta deploy

Each of those deploys to different servers, for either production or staging purposes. Inside of config/deploy/beta.rb we have the branch to deploy specified as:

set :branch, ENV["BRANCH"] || "development"

So if there's a BRANCH env var, use that. Otherwise, use development. We also have a little function we typically used for deploying "whatever branch I'm on right now" but given our continuous deployment changes I figured it was legitimate to take that feature away.


So now let's have a look at Jenkins. I won't go into this part, but of course Jenkins is set up to run our tests after every commit to the GitHub repo is pushed up. What I want to look at is the job that we added to manage the deployment. The idea is to set up a parametrized build that allows you to specify which branch you wish to deploy, so it can be used pushbutton style. The next move is to make sure it can also handle being triggered from a Jenkins job that runs the tests, so that in the event that a Jenkins variable specifying the sha that the tests were run on is set, it will deploy that sha specifically. Let's look at the shell script that our deploy job runs:

source /usr/local/share/chruby/

# Setup Local Environment
export RAILS_ENV=development
export BUILD_DATE=`date +%s`

chruby 2.1.0
(gem list | grep bundler) || gem install bundler

git checkout $BRANCH
git reset --hard $BRANCH

# Install the bundle of gems
bundle install --path .bundle/bundle

if [ $BRANCH != origin/master ] && [ $BRANCH != master ]
    export STAGE=beta
    export STAGE=production

eval `ssh-agent -s`
ssh-add -D
ssh-add ~/.ssh/

if [ -z "$GIT_COMMIT" ]
    export SHA=`git rev-list HEAD --max-count=1`
    export SHA=$GIT_COMMIT

BRANCH=$SHA bundle exec cap $STAGE deploy
ssh-agent -k

Some things to note, or an English-language walkthrough of the above:

  • We're forcing our shell to bash, which is helpful with the chruby bits when compared to sh.
  • We're specifying our ruby with chruby, installing bundler, and installing the bundle locally.
  • If the branch is master, we're setting the STAGE env var to production, otherwise it's beta. This lines up with our capistrano discussion above. The point here is that commits to master will deploy production, while commits anywhere else will deploy beta.
  • We're loading up our ssh identity.
  • If we have a GIT_COMMIT environment variable, we're setting the SHA env var to it. Otherwise we're setting it to the latest commit in the local branch.
  • Finally, we set the BRANCH var to the sha and we deploy to the STAGE specified.
  • Oh yeah, kill the ssh agent :)

So this is a pretty solid way to do this. One of the things that's worth pointing out is that we had to install the Parametrized Trigger plugin and then in our main repo job we added a Post-build action to trigger this parametrized build when stable, and specify a predefined parameter such that the local GIT_BRANCH jenkins variable is used for the BRANCH variable that's passed to the parametrized build. With that, we're done, and everything works exactly as I'd hoped.

jenkins parametrized trigger setup

It took me a little while to think through how I would set all of this up, so I figured it was worth sharing with the world. Feel free to leave comments if you have any questions. Thanks!

Josh Adams is a developer and architect with over eleven years of professional experience building production-quality software and managing projects. Josh is isotope|eleven's lead architect, and is responsible for overseeing architectural decisions and translating customer requirements into working software. Josh graduated from the University of Alabama at Birmingham (UAB) with Bachelor of Science degrees in both Mathematics and Philosophy. He runs the ElixirSips screencast series, teaching hundreds of developers Elixir. He also occasionally provides Technical Review for Apress Publishing, specifically regarding Arduino microprocessors. When he's not working, Josh enjoys spending time with his family.