Ever since I started playing around with Unix ~13 years ago, I've been a fan of automating things. What started out as writing little (maybe pointless) shell scripts slowly but surely morphed into infrastructure automation today.
As for my, or maybe anyone's, motivation to do these things, I see three main factors:
- I'm easily bored — because repeating things is dull.
- I'm easily distracted (when I'm bored).
- I'm German: Of course we strive for perfection and excellence. ;-)
Being on Unix (or Linux) it's fairly simple to automate things — add some script-fu to bash or csh (or even better zsh) and off you go wrapping things into a small shell script! Then execute again and again!
Before we decided to moved to AWS (and RightScale) in late 2009 we had half a rack of servers (in a Peer1's POP in NYC) and never did any or much infrastructure automation. We had an image and a set of commands to get a server up and running, but it was far from two mouse-clicks today.
At the time, I had read about cfengine a couple of times, but datacenter-grade infrastructure management along with a rather steep learning wall (at that time anyway) seemed overkill. Add to that, that there is not a lot of time for research Fridays when you work in a small company.
Moving to AWS and RightScale required us to write lots of small shell scripts using bash and Ruby. When we moved from RightScale to Scalarium in late 2010, we went from shell scripts to Chef.
Using Chef meant that we created so-called recipes which are used to bootstrap our servers. Recipes are little Ruby scripts which live in a cookbook — open source projects are sometimes so creative. Before this move I had very little, or next to no, experience with Chef, and Chef being Ruby didn't exactly make me want to try it either.
So what exactly is Chef?
A Chef recipe is a tiny bit of Ruby code — essentially a high(er)-level wrapper around calls such as installing certain packages, creating and parsing configuration files and many other things.
Chef offers a robust abstraction about everything you can do with shell and with a little effort it's also possible to write recipes which run on multiple OS'. Supported are Ubuntu, CentOS, FreeBSD and others. For an intro to Chef see the slides of a talk I gave a couple weeks ago; I briefly blogged about it too.
Our Chef recipes currently include things like installing and configuring PHP (from source and through a launchpad repository), nginx, MySQL, CouchDB, haproxy and many other things. The list was literally growing every day for the first few weeks.
Automation in Ruby
Chef is ruby — especially in the PHP world people seem a little hesitant when I tell them about chef and Scalarium. Afterall, learning Ruby to setup your PHP stack seems funny. But in the end, the imho best tools (chef and puppet) are written in Ruby — so yes, using Chef means that you'll have to write Ruby.
Fear not though, because Chef is just one example, or one tool, to automate your infrastructure — others in this space (or similar) are cfengine (Perl), fabric (Python) or puppet (also Ruby).
I noticed that aside from learning Ruby it seems to be a huge burden for some people to write just about anything to automate a server setup. Even though these kind of tools have been around for a while, a lot of people still plan a day or so to setup another server when need comes around.
Regardless of the flavour, I believe that using just about anything results in a huge boost in productivity. Even if it's a very sophisticated shell script in the end. By how much you see or feel this boost very much depends on how thorough you go about implementing and using the tools.
To EC2 and beyond!
When we started off on EC2, we went with RightScale. RightScale essentially required us to
create so-called ServerTemplates. These ServerTemplates were a collection of scripts (bash or
/usr/bin/env ruby) which were either run on demand or on these three events:
When we moved to Scalarium the basics seemed similar, with one event added: Configure. Configure is basically an event send to all nodes of the cluster whenever the cluster changes.
Configure in a nutshell
A very simple cluster would be made of nodes with different roles:
- application server
- database server
Whenever a new application server is added, the configure event is sent to all nodes of the cluster to make everyone aware of the new application server node.
Awareness in this case would translate to adding the new node to the load balancer so it receives traffic and maybe adding an entry to MySQL's privileges so the new node can connect to the database.
Scalarium vs. RightScale
I don't really want to go into too many juicy details here. I've shared my opinion on RightScale before and it basically got to a point for us where we didn't want to touch it anymore.
And while I write this of course I am aware of the fact that this is not just RightScale's fault, but expectations we had from using RightScale might just not be what the average RightScale customer wants.
Scalarium in this respect is a little different. On one hand, these folks are local to me (they are based in Berlin), but they also offer great support in general. Don't take my word for it: contact them for references.
The bottom line is: Scalarium embraces Chef. Chef is open source as well and seems a better fit in the long run because we can use it for more than just bootstrapping on EC2.
Which brings me to Vagrant.
Vagrant is a really neat piece of software to build and manage VMs. The software underneath is (Oracle's) Virtualbox and the provisioning process is powered by Chef or Puppet. We'll use Chef in this example to bring my point across.
So in order to to do Vagrant, you need Ruby gems and Ruby installed. On Ubuntu Ruby is a little painful — then again, what isn't? To ease the process, I suggest you download a .deb with the Ruby Enterprise edition.
Setting up on Ubuntu sometimes entails limited access to bleeding edge software. So in this case, Vagrant's 0.7.x branch requires Virtualbox 4.0, but so far I found 3.2 only:
$ sudo aptitude install virtualbox-3.2
I checked out all published vagrant gems versions online, and decided to go with the latest of the 0.6-branch:
$ sudo gem install vagrant -v 0.6.9
The following steps setup a small VM:
$ vagrant box add base http://files.vagrantup.com/lucid32.box $ mkdir myproject/ $ cd myproject/ $ vagrant init base $ vagrant up $ vagrant ssh
And then you should be inside the box:
[email protected]:~$ uname -a Linux vagrantup 2.6.32-21-generic #32-Ubuntu SMP Fri Apr 16 08:10:02 UTC 2010 i686 GNU/Linux
CTRL+d or type
exit to leave it. ;-)
So in my case, I wanted setup a replica of our application server. My cookbooks are checked out in
/home/till/cookbooks, so I left the VM and edited the
config.vm.provisioner = :chef_solo config.chef.cookbooks_path = "/home/till/cookbooks" config.chef.add_recipe "easybib-base::setup" config.chef.add_recipe "php-fpm::install-apt" config.chef.add_recipe "php-fpm::pear"
Then, restart the provisioning process:
$ vagrant provision
So awesome and so simple!
Going from here
Assuming everything worked so far, you got yourself the tools to setup an identical setup on your local machine to your setup in the cloud. I noticed that I had to improve some of my recipes to make them less EC2 specific. But this kind of refactoring was overdue anyway.
If you happen to work with a team of developers, here are some ideas to make your life even easier.
The first and obvious step is to make everyone use Vagrant. This allows everyone to work in an identical environment. None of this, "But it works for me!".
As you can tell from this blog entry: it's far from rocket science.
Package it up!
However, telling people to do something and them actually getting around to it, is two pair of shoes. So the easiest way to make sure the setup is exactly what you want is to package your setup into a custom box and distribute your box to the developers.
$ cd myproject/ $ vagrant package --include Vagrantfile
The result is:
The developer then has to do these steps:
$ vagrant box add myproject myproject.box $ mkdir myproject/ $ cd myproject/ $ vagrant init myproject $ vagrant up $ vagrant ssh
When things go wrong
So sometimes, the provision process failed due to whatever. By default, vagrant (or essentially chef-solo) is not very verbose about it.
The following SSH command responded with a non-zero exit status. Vagrant assumes that this means the command failed! cd /tmp/vagrant-chef && sudo -E chef-solo -c solo.rb -j dna.json
So first off: to run the above command you have to
vagrant ssh first. Secondly: you should add
-l debug to the end off it because otherwise the error message won't improve.
All in all this two stepper seems rather elaborate to me. To streamline the debugging I actually want Vagrant to run chef-solo with debug logging always. To do this, I added the following to my
config.chef.log_level = :debug
That's all, in the blog post I demo'd how to re-use your chef recipes for local development goodness. I sure hope this gets everyone started. For feedback, please email me or comment.