So we've had a pretty decent Continuous Integration setup here at Isotope11 for the past four years or so. We should have had it for longer, but them's the breaks. At any rate, some of our most popular blog posts, by a huge margin, have been related to our Jenkins work, whether it be our post on Monitoring Your Continuous Integration Server With Traffic Lights and an Arduino , our post on Styling Your Jenkins Continuous Integration Server, or my last post on our first, less-appropriate means of Continuous Deployment with Capistrano and Jenkins. The point is, we love Jenkins, we do fun stuff with it, and The Internets seems to like what we do. With that in mind, I'm going to explore the new Continuous Deployment strategy here at Isotope11 for 2015.
The problems with the existing setup
So our existing Continuous Deployment setup, detailed in the above post, has gotten us pretty far with a lot of success. We've never had a situation where a deployment caused us pain. However, there was a pretty big known bug with it the entire time. Here's the 'algorithm' it followed, in a nutshell:
Every time any build passes on any branch, deploy whatever is at the head of master at the moment.
This is stupid, but it was the easiest thing to implement, and: master should always be deployable, and deploys shouldn't cause downtime anyway. If those two cases are true, then there should be no problems. So in general it was a reasonable but stupid way to implement CD, and the whole Internet seemed to like it fine enough so I didn't lose sleep over the downsides.
However, it is a horrible strategy and I wasn't willing to leave it like this forever. So here's the new deal:
The proper thing to do
The above strategy works horribly for also automatically deploying ongoing work to a staging server (or multiple subdomains, one-per-branch, or whatever). This is a desirable feature for us to gain, and January's kind of slow, so I went for it. Here's the new 'algorithm' in a nutshell:
When a build passes on any branch, deploy that particular sha. If the branch that passed was master, deploy to production; otherwise, deploy to staging.
This is not 100% perfect, as it'd be ideal to have a subdomain per branch, teardown of merged branches' deploys, etc., but it's a pretty good setup. It's also a far cry less naive than the previous strategy. Let me show you how we ended up implementing it.
So our project that we used as a testbed for this strategy was isotope11.com itself. It's low-risk, and we're immediately eating our own dogfood rather than doing this on a small project and then not living with the consequences five months later. It's a rails site, and it's deployed with Capistrano, with the deployment happening directly from the site's repo. (For newer projects, we tend to have a separate deployment repo, as that can house things like ansible scripts and whatnot, but this is just a sidebar)
If you're unfamiliar with capistrano and multistage projects, our deploys look like one of two things:
cap production deploy cap beta deploy
Each of those deploys to different servers, for either production or staging
purposes. Inside of
config/deploy/beta.rb we have the branch to deploy
set :branch, ENV["BRANCH"] || "development"
So if there's a BRANCH env var, use that. Otherwise, use development. We also have a little function we typically used for deploying "whatever branch I'm on right now" but given our continuous deployment changes I figured it was legitimate to take that feature away.
So now let's have a look at Jenkins. I won't go into this part, but of course Jenkins is set up to run our tests after every commit to the GitHub repo is pushed up. What I want to look at is the job that we added to manage the deployment. The idea is to set up a parametrized build that allows you to specify which branch you wish to deploy, so it can be used pushbutton style. The next move is to make sure it can also handle being triggered from a Jenkins job that runs the tests, so that in the event that a Jenkins variable specifying the sha that the tests were run on is set, it will deploy that sha specifically. Let's look at the shell script that our deploy job runs:
#!/bin/bash source /usr/local/share/chruby/chruby.sh # Setup Local Environment export RAILS_ENV=development export BUILD_DATE=`date +%s` chruby 2.1.0 (gem list | grep bundler) || gem install bundler git checkout $BRANCH git reset --hard $BRANCH # Install the bundle of gems bundle install --path .bundle/bundle if [ $BRANCH != origin/master ] && [ $BRANCH != master ] then export STAGE=beta else export STAGE=production fi eval `ssh-agent -s` ssh-add -D ssh-add ~/.ssh/isotope11.com if [ -z "$GIT_COMMIT" ] then export SHA=`git rev-list HEAD --max-count=1` else export SHA=$GIT_COMMIT fi BRANCH=$SHA bundle exec cap $STAGE deploy ssh-agent -k
Some things to note, or an English-language walkthrough of the above:
- We're forcing our shell to
bash, which is helpful with the chruby bits when compared to
- We're specifying our ruby with
bundler, and installing the bundle locally.
- If the branch is master, we're setting the
STAGEenv var to
production, otherwise it's
beta. This lines up with our capistrano discussion above. The point here is that commits to master will deploy production, while commits anywhere else will deploy beta.
- We're loading up our ssh identity.
- If we have a
GIT_COMMITenvironment variable, we're setting the
SHAenv var to it. Otherwise we're setting it to the latest commit in the local branch.
- Finally, we set the
BRANCHvar to the sha and we deploy to the
- Oh yeah, kill the ssh agent :)
So this is a pretty solid way to do this. One of the things that's worth
pointing out is that we had to install the
Parametrized Trigger plugin and
then in our main
isotope11.com repo job we added a Post-build action to
trigger this parametrized build when stable, and specify a predefined parameter
such that the local
GIT_BRANCH jenkins variable is used for the
variable that's passed to the parametrized build. With that, we're done, and
everything works exactly as I'd hoped.
It took me a little while to think through how I would set all of this up, so I figured it was worth sharing with the world. Feel free to leave comments if you have any questions. Thanks!
Josh Adams is a developer and architect with over eleven years of professional experience building production-quality software and managing projects. Josh is isotope|eleven's lead architect, and is responsible for overseeing architectural decisions and translating customer requirements into working software. Josh graduated from the University of Alabama at Birmingham (UAB) with Bachelor of Science degrees in both Mathematics and Philosophy. He runs the ElixirSips screencast series, teaching hundreds of developers Elixir. He also occasionally provides Technical Review for Apress Publishing, specifically regarding Arduino microprocessors. When he's not working, Josh enjoys spending time with his family.