Till's blog

I'm sure most of you noticed that with the recent upgrade of PHPUnit to version 3.6, a lot of breakage was introduced in various projects.

And for example Zend Framework 1.x won't update to the latest version either. When I ranted on twitter someone send me Christer Edvartsen's blog post on how to setup multiple versions of PHPUnit. It's really neat since it walks you through the setup step by step and you learn about things such as --installroot on the way. --installroot in particular is something I never ever saw before and I've been using PEAR for more than a few years now. So kudos to Christer for introducing myself to it.

The only thing to add from my side would be, Why are you guys not aggregated on planet-php?.

Cooking with Chef

Another reason why I decided to write this blog entry was that I created a chef-recipe based on Christer's blog entry.

If you follow my blog for a while, you might have noticed that I'm a huge fan of automation. I just moved one of our development servers the other day and had one of these moments where something just paid off. Taking for granted that I can spin up fully operational EC2 instances in minutes, I also had our development stack installed and configured in an instant.

My recipe basically follows Christer's instructions and because I distribute phpunit's command along with it, editing of the file is no longer required: when the chef run completes, phpunit34 is installed and ready to be used.

Get started

I'm doing the following commands as root — my setup is in /root/chef-setup.

Install chef(-solo) and clone my cookbooks

shell# gem install --no-ri --no-rdoc chef
... 
shell# git clone git://github.com/till/easybib-cookbooks.git
...

Chef configuration

Then setup a node.json file which chef-solo will need to run:

{
  "run_list": [
    "recipe[phpunit]"
  ]
}

Then create a solo.rb:

file_cache_path "/var/chef-solo"
cookbook_path ["/root/chef-setup/easybib-cookbooks"]

Chef run

Finally we start chef-solo with following command:

shell# chef-solo -c /root/chef-setup/solo.rb -j /root/chef-setup/node.json -l debug
...

The command runs chef-solo (which is part of the gem we installed) and reads the basic configuration from the solo.rb-file. This file contains the location of the cookbooks (remember git clone ...) and a path to cache files. You don't need to create anything, it should be all taken care of.

The node.json-part allows us to set node-specific values. The prime example is the run-list, but it allows you to set attributes as well. Attributes contain values for variables used in recipes, but are not used in this example.

Last but not least: -l debug — a lot of useful output, but we usually run with -l warn. And if this is interesting enough for you, I suggest the other blog entries I wrote on this topic.

Did it work?

Depending on the location of your pear setup — usually /usr/bin/pear or /usr/local/bin/pear — the phpunit34 script is created in the same path:

shell# which phpunit34
/usr/local/bin/phpunit34

Yay!

Fin

This feels like hitting two birds with one stone. Though just by figure of speech! I object to violence against birds.

It might be overkill to setup chef to just install phpunit 3.4 by itself, but I think this serves as a stellar example of how you can leverage the power of chef to get more done. Writing a couple more recipes to install and configure the rest of your stack shouldn't be too hard.

If you'd like to see anything in particular: I'll take requests via email, Twitter or in the comments.

If you followed my blog, you probably know about chef and vagrant.

So the other day I managed to upgrade to VirtualBox 4.0. The upgrade just happened by accident, so to speak. I noticed that Virtualbox 4.0 had moved from nonfree to contrib on Oracle's repository which is why I had previously missed it. With 4.0, I am now able to run the latest and greatest Vagrant — and with Vagrant being pre-1.0 it's safe to assume that newer is indeed better.

Anyway — the VirtualBox update fubar'd all of my boxes. They didn't outright refuse to start, but provisioning failed nine out of ten times and other random issues seemed to pile up.

I noticed this warning about the Guest Additions inside the box and how they are different from the VirtualBox installed on the host:

[default] The guest additions on this VM do not match the install version of
VirtualBox! This may cause things such as forwarded ports, shared
folders, and more to not work properly. If any of those things fail on
this machine, please update the guest additions and repackage the
box.

Guest Additions Version: 3.y.z
VirtualBox Version: 4.0.8

To cut to the chase — once I updated them and the box itself, they all continued to work.

So here's how!

Boot a blank VM

So first off, you take the image of your choice and boot a blank virtual machine. Blank because no chef-recipes should be configured, etc.. This VM should literally boot in a minute or so.

Now enter the VM and update the Guest Additions, and while we're at it, feel free to update the OS etc. as well.

Update the base

On my Ubuntu Karmic box, this is what I did (inside the VM) first:

vagrantup:~$ sudo aptitude update
...
vagrantup:~$ sudo aptitude upgrade
...
vagrantup:~$ sudo gem update --no-rdoc --no-test --no-ri
...
vagrantup:~$ sudo aptitude install portmap nfs-common
...

Pro-tip: The last step is necessary in case you run into the various issues with VirtualBox shared folders. The official website lists performance as the #1 one reason to use NFS shares instead of the VirtualBox shares, my #1 reason was issue #351.

Update Guest Additions

It took me a lot of Google, to actually find the download — regardless this is what seems to work. I extracted these steps from the blog post I found. For this to work, you are still inside the VM:

vagrantup:~$ wget -c \
http://download.virtualbox.org/virtualbox/4.0.8/VBoxGuestAdditions_4.0.8.iso \
-O VBoxGuestAdditions_4.0.8.iso
...
vagrantup:~$ sudo mount VBoxGuestAdditions_4.0.8.iso -o loop /mnt
vagrantup:~$ sudo sh /mnt/VBoxLinuxAdditions.run --nox11
...
vagrantup:~$ rm *.iso

There's an error about something failing to install (Windows-related), but I ignored it since I don't run the GUI anyway.

Pro-tip: Installing the latest vagrant along with Virtualbox 4.0.8 made the GUI mode fail consistently. Not that I really mind, but I feel compelled to share.

Repackage

Once these steps are completed, exit the VM (ctrl+d), halt the VM and up to test. If the VM starts up and no warnings show up, you got yourself a winner.

To repackage the VM, do the following:

~/karmic-test$ vagrant package

This creates a package.box file — I renamed mine to karmic-nfs.box. Import the box with:

~/karmic-test$ vagrant box add karmic-nfs karmic-nfs.box 
[vagrant] Downloading with Vagrant::Downloaders::File...
[vagrant] Copying box to temporary location...
[vagrant] Extracting box...
[vagrant] Verifying box...
[vagrant] Cleaning up downloaded box...

Done!

Fin

That's all, in summary the biggest obstacles are as follows:

1. VirtualBox on the host and VirtualBox Guest Additions inside the box need to match (closely).
2. The box needs to be kept up to date — more or less recent software inside the box makes bootstrapping and provisioning easier and faster.

So the other day, I wanted to quickly check something in BigCouch and thanks to Vagrant, chef(-solo) and a couple cookbooks — courtesy of Cloudant — this was exceptionally easy.

As a matter of fact, I had BigCouch running and setup within literally minutes.

Here's how.

Requirements

You'll need git, Ruby, gems and Vagrant (along with Virtualbox) installed. If you need help with those items, I suggest you check out my previous blog post called Getting the most out of Chef with Scalarium and vagrant.

For operating system to use, I suggest you get a Ubuntu 10.04 box (aka Lucid).

Vagrant (along with Ruby and Virtualbox) is a one time setup which you can use and abuse for all kinds of things, so don't worry about the extra steps.

Setup

Clone the cookbooks in $HOME:

$ git clone http://github.com/cloudant/cloudant_cookbooks

Create a vagrant environement:

$ mkdir ~/bigcouch-test
$ cd ~/bigcouch-test
$ vagrant init

Setup ~/bigcouch-test/Vagrantfile:

Vagrant::Config.run do |config|
  config.vm.box = "base"
  config.vm.box_url = "http://files.vagrantup.com/lucid32.box"

  # Forward a port from the guest to the host, which allows for outside
  # computers to access the VM, whereas host only networking does not.
  # config.vm.forward_port "http", 80, 8080

  config.vm.provisioner = :chef_solo
  config.chef.cookbooks_path = "~/cloudant_cookbooks"
  config.chef.add_recipe "bigcouch::default"
end

Start the vm:

$ vagrant up

Use BigCouch

$ vagrant ssh
$ sudo /etc/init.d/bigcouch start
$ ps aux|grep [b]igcouch

Done. (You should see processes located in /opt/bigcouch.)

Fin

That's all — for an added bonus you could open BigCouch's ports on the VM use it from your host system because otherwise this is all a matter of localhost. See config.vm.forward_port in your Vagrantfile.

Ever since I started playing around with Unix ~13 years ago, I've been a fan of automating things. What started out as writing little (maybe pointless) shell scripts slowly but surely morphed into infrastructure automation today.

As for my, or maybe anyone's, motivation to do these things, I see three main factors:

• I'm easily bored — because repeating things is dull.
• I'm easily distracted (when I'm bored).
• I'm German: Of course we strive for perfection and excellence. ;-)

Being on Unix (or Linux) it's fairly simple to automate things — add some script-fu to bash or csh (or even better zsh) and off you go wrapping things into a small shell script! Then execute again and again!

Before we decided to moved to AWS (and RightScale) in late 2009 we had half a rack of servers (in a Peer1's POP in NYC) and never did any or much infrastructure automation. We had an image and a set of commands to get a server up and running, but it was far from two mouse-clicks today.

At the time, I had read about cfengine a couple of times, but datacenter-grade infrastructure management along with a rather steep learning wall (at that time anyway) seemed overkill. Add to that, that there is not a lot of time for research Fridays when you work in a small company.

Moving to AWS and RightScale required us to write lots of small shell scripts using bash and Ruby. When we moved from RightScale to Scalarium in late 2010, we went from shell scripts to Chef.

Using Chef meant that we created so-called recipes which are used to bootstrap our servers. Recipes are little Ruby scripts which live in a cookbook — open source projects are sometimes so creative. Before this move I had very little, or next to no, experience with Chef, and Chef being Ruby didn't exactly make me want to try it either.

So what exactly is Chef?

A Chef recipe is a tiny bit of Ruby code — essentially a high(er)-level wrapper around calls such as installing certain packages, creating and parsing configuration files and many other things.

Chef offers a robust abstraction about everything you can do with shell and with a little effort it's also possible to write recipes which run on multiple OS'. Supported are Ubuntu, CentOS, FreeBSD and others. For an intro to Chef see the slides of a talk I gave a couple weeks ago; I briefly blogged about it too.

Our Chef recipes currently include things like installing and configuring PHP (from source and through a launchpad repository), nginx, MySQL, CouchDB, haproxy and many other things. The list was literally growing every day for the first few weeks.