Tuesday, April 16. 2013

At the last meetup of the Berlin PHP Usergroup, Christoph gave a talk about Vagrant.

Good enough of a reason to write down or re-cap some things I've learned with or about Vagrant over the last two years.

Base boxes

There are lots of base boxes available, but don't be tempted to rely on them (e.g. via config.vm.box_url).

  1. Vagrantbox.es doesn't actually mirror images and that is a huge pain.
  2. Available base boxes tend to be outdated. (Think kernel, packages, etc.) Running updates each time you provision is painful.
  3. Available base boxes use U.S. mirrors only/mostly/always — because we all live in the U.S. of A..
  4. Your VirtualBox guest additions may not match with your system and this may create random issues.

Your best bet is to create your own base box and establish a workflow e.g. using veewee or bento.

At EasyBib, we use bento and we created a definition which replaces the sources with Ubuntu's nifty mirror syntax (since we're pretty distributed at times, everyone appreciates this) and upgrades the base system. Either one of these tools introduce more Ruby into your organization and you may think "WTF — why do I need this?!", but the clear advantage is that no one has to write down a lot of steps how to recreate these boxes and anyone can do it.

In bento's case, the requirement is Ruby 1.9.1+ (getting this installed is IMO the hardest) and bundler. bundle install in your bento-clone gets you everything needed and then the three commands require build, validate and export a box which is ready to use. Ensure to put whatever you need into the definition — for example in the update.sh. Avoid too many manual steps before you export because the next person will have to know and repeat them. Bento serves as documentation as well.

I version our boxes with like easybib-something-10.04.4_vbox-4.1.8_0.1.box and upload them to an S3 bucket. The first number is the Ubuntu release and the second is the version of the VirtualBox guest additions. Simple. The third version is our internal iteration — typically a base box isn't perfect from the beginning while e.g. the Ubuntu and VirtualBox part are settled, there might be other improvements. With an extra version you avoid conflicts and extra work like vagrant box delete etc. and ensure the latest box is always used.

Also — in case the software stack is very different across your projects, it also helps to to create different boxes which come with different software pre-installed.

Standardize on versions

Vagrant and VirtualBox have frequent releases. I suggest to standardize on one so members of your team don't have random issues at hand and fires to fight.

Even for a small team of up to ten developers this makes a lot of sense. Because people tend to add a lot of randomness anyway — different hardware, operating systems and so on. Fight only the battles you want to fight, and deploy otherwise.

Vagrant also recently went from being a rubygem to providing installers. I haven't had the time to roll this out yet, but I expect this to help as well since at least as far as ruby is concerned all the dependencies are bundled.

This of course still implies that testing is required so you and your team don't walk into a stupid little regression and waste away the day trying to figure out what went wrong. And of course even if Vagrant is smoother, it still leaves you with VirtualBox and tools like bento and plenty of potential breakage.

Chef Versions

On a side-note — Chef 10 and 11 may also introduce a lot of breakage in recipes. It helps to roll your base box with a specific version as well. With bento the work-around was pretty straight forward: I replaced the chef-client.sh and installed Chef 10 (instead of 11 — or whatever the latest is).

VirtualBox and guest additions

In theory, it's alright to run with different guest additions in a box than the version of VirtualBox you have installed on the host. It should at least match the main release — for example: 4.1.8 guest additions and 4.1.12 VirtualBox should do fine. That's a big should though, because it also may cause random issues like crashes and hangs.

If you don't want or cannot rebuild the base box for some reason, you can also use vbguest which is a Vagrant plugin to update the guest additions when you start the virtual machine. Keep in mind that this adds a couple minutes to the bootstrapping.

Learn some Ruby

There are little things where it helps to know a little Ruby. And by Ruby, I don't mean Rails. A Vagrantfile itself is Ruby code — this implies that it is fully customizable.

An example of something we as a team couldn't agree on is the location of where projects (and essentially cookbooks) are located on your local disk. Every other team member has a different preference:

case ENV['USER']
when 'till'
  local_cookbook_dir = "~/Documents/workspaces/easybib-cookbooks"
when 'someonelse'
  local_cookbook_dir = "~/dev/till/easybib-cookbooks"
  local_cookbook_dir = "~/Sites/easybib/cookbooks"

if not File.directory?(File.expand_path(local_cookbook_dir))
  raise "You need to checkout your cookbooks into #{local_cookbook_dir}"

# ...

web_config.vm.provision :chef_solo do |chef|
  chef.cookbooks_path = local_cookbook_dir
  chef.add_recipe "ohai"
  # ...

It's as simple as that.

Another example — setting VirtualBox options for everyone but a certain user:

web_config.vm.boot_mode = :gui unless ENV['USER'] == 'mr_I_dont_run_X'

Bonus tip: Once you made changes, make sure to at least re-provision. Commit and push after!

Learn Chef or Puppet

I often see projects where developers end up writing a lot of shell script to bootstrap VMs, but learning Chef or Puppet is not really that hard.

I find it harder to validate exit codes (again and again and again) in bash than using a DSL (which is what Chef and Puppet essentially are). The code in your cookbooks (Chef) or manifests (Puppet) is certainly not faster than a shell script but a lot easier to read and more maintainable in the end.

Bash-scripting is not hard either, but in order to produce a set of scripts which can be ran again and again (not just to bootstrap a fresh VM but e.g. also to run updates on one that is running), defensive coding is paramount. And while that is certainly not impossible, it's often a waste of time when frameworks like Chef or Puppet have that covered.

But let's skip on the benefits of using identical tools to bootstrap Vagrant, staging and production because I find them more than obvious.

Learn some Linux

Every once in a while you will run into weird issues with the VMs. These may include one of your VMs losing connectivity (sudo restart networking to the rescue) or weird behavior like assets not refreshing (sendfile off; in nginx). Take it as an opportunity to learn some about the system that is run in production.

In the end all required configuration changes will go back into your provisioning and make sure to share your experience with at least the people on your team.

Backup everything

Whatever you find and use — make a copy of it and put it on Amazon S3 or the local network. With larger teams even a local Ubuntu mirror (or whatever you use) can come in handy.

This includes base boxes, packages, etc.. Nothing is more annoying than waking up and not being able to bootstrap your VMs because someone decided to remove something in order to force you to upgrade.

Don't dumb it down!

Typically, PHP applications are developed on a single host — Apache, PHP and MySQL on localhost. With Vagrant it becomes surprisingly easy to mimic production.

Not to say that I have to run 20 virtual machines to copy my cluster of application servers, but it's perfectly acceptable to set up an environment with four VMs where one is a loadbalancer, two are application servers and then a database server.

Networking and port forwards

Unless you regulary let others use your VMs, don't add port forwards — or at least install a firewall.

For networking, I suggest you either use static IPs (and keep track of them in a sheet) or DHCP. I prefer static IPs though since that makes configuration (e.g. of an application to connect to the database) easier.

It also doesn't hurt to assign names, so you know which VM you're dealing with when GUI is enabled:

    db_config.vm.customize [
      "modifyvm", :id,
      "--name", "DB",


It doesn't hurt to have lots of CPU and RAM, but also configure the VMs accordingly. I run up to four virtual machines on a Macbook Air — usually configured with 256 to 512 MB. I imagine this would go smoother with VMWare Fusion, but since our team contains Mac and Linux as well, we haven't moved on this.

Here's an example how to give 512 MB RAM to a virtual machine:

    db_config.vm.customize [
      "modifyvm", :id,
      "--memory", "512"


That's all I can think of right now. Happy development!

Vagrant sans Ruby

Tuesday, June 5. 2012

Development, testing, staging and production — this is how most people devide up different environments for application development.

Maintenance and setup of these environments is often not a trivial goal to achieve. Having worked with a couple different code bases and setups over the last decade, I often noticed things like environment specific hacks (if ($env == 'testing') { ... }) in application code and service configurations and a lot of manual labour all around. It's still very common that code deployed to staging does not work — but it worked for you right?

And it's also too common that members of a team do not know how something works because whoever set it up is currently out sick or on vacation.

My opinion is that the best setup currently available is something like: chef(-solo) on the server and Vagrant on the desktop. Something like because aside from chef, there is also puppet, cfengine and a couple others. Leaving specific projects aside, it really just boils down to automation (within reason).

Automation in my opinion is not just the easiest but the only viable way to setup development, testing, staging and production environments. Without some automation in place it's required that all team members know how all of it works when maybe that is not yet important and the end goal is that all environments actually resemble each other.

Enter Vagrant

Vagrant is a very nifty toolkit to bootstrap Virtualbox images. Bootstrapping means installing your application stack into one or multiple virtual machines in order to resemble production a lot better.

So a lot of times when I rave about how useful Vagrant and chef are, I get the crazy eye from PHP developers:

You want me to learn Ruby to setup my local development environment?

My response:

  1. Do not fear the Ruby.
  2. You don't have to. (Well, not a whole lot!)

Real talk

First off, of course you need Virtualbox — get a download from their website or use your package manager.

Then, it's not possible to avoid Ruby a 100% — after all Vagrant is written in Ruby.

When you're on a Linux or a Mac, this is usually enough (you may need sudo unless you RVM):

$ sudo gem install vagrant

For a sudo-less install, use RVM:

$ sudo gem install rvm
$ rvm install
$ rvm use 1.8.7

When set up, this is what you do:

$ gem install vagrant

RVM allows me to run multiple versions of ruby side by side and also leverage local gem installs — per Ruby version. Think of it as a nifty way to run multiple PHP versions and have a PEAR install per version — similar projects exist for PHP today:

To learn more about RVM, visit their website.

Once vagrant is installed, we can continue!

ShellProvisioner for fun and profit

So, in case you are more comfortable writing some shell script for the time being — Vagrant got you covered!

First off, create a new Vagrantfile for your project:

$ mkdir -p Documents/workspaces/blog-example/
$ cd Documents/workspaces/blog-example
$ vagrant init lucid64
A `Vagrantfile` has been placed in this directory. You are now
ready to `vagrant up` your first virtual environment! Please read
the comments in the Vagrantfile as well as documentation on
`vagrantup.com` for more information on using Vagrant.

lucid64 (Ubuntu Lucid 10.04, 64bit) is the name of one of my local box files for Vagrant. In case you haven't got a box yet, head over to vagrantbox.es. There are a couple images you (aka box files) you can download. To get the lucid64 box, use the following command:

$ vagrant box add lucid64 http://files.vagrantup.com/lucid64.box

Once you made it past vagrant init, you should have a Vagrantfile with a lot of stuff in there.

While it's not important for my tutorial, I recommend you review it some (other time). The created Vagrantfile contains examples for all the provisioners (puppet, chef and shell) and a couple other configuration options, etc..

Let's skip over this and get to the ShellProvisioner.

This is all you need

First off, let's create a shell script setup.sh in the same directory and put something like this in it:


apt-get update

apt-get install -y php5 php5-cli php-pear
hash -r

pear upgrade-all
pear install -f HTTP_Request2

What does it do?

  1. Update local package sources.
  2. Install php5, cli interpreter and PEAR installer
  3. reload environment
  4. upgrade all installed PEAR packages
  5. install PEAR HTTP_Request2

When you're done editing, make sure to chmod +x it.

Simple as that.

Putting your shell script to use

Replace your Vagrantfile with the following:

Vagrant::Config.run do |config|

  config.vm.box = "lucid64"
  config.vm.provision :shell, :path => "./setup.sh"


Then, run vagrant up and watch it provision. :)

Enter your virtual machine with vagrant ssh and verify HTTP_Request2 is installed:

$ vagrant ssh                                                                                                                                                               [14:16:08]
Linux lucid64 2.6.32-33-server #70-Ubuntu SMP Thu Jul 7 22:28:30 UTC 2011 x86_64 GNU/Linux
Ubuntu 10.04.3 LTS

Welcome to the Ubuntu Server!
 * Documentation:  http://www.ubuntu.com/server/doc
Last login: Thu Jul 21 14:08:15 2011 from
[email protected]:~$ pear list -c pear
Installed packages, channel pear.php.net:
Package          Version State
Archive_Tar      1.3.10  stable
Console_Getopt   1.3.1   stable
HTTP_Request2    2.1.1   stable
Net_URL2         2.0.0   stable
PEAR             1.9.4   stable
Structures_Graph 1.0.4   stable
XML_Util         1.2.1   stable

Yay — your first steps into automation!


That's all for today — Vagrant with almost no Ruby. I hope this gets many more started!

apt-repair-sources on Ubuntu

Wednesday, November 23. 2011

When I ran our setup on an instance the other day, I noticed how it failed with a "package not found" (or similar) error. After debugging this a bit, we discovered that Karmic moved from "archive.ubuntu.com" to "old-releases.ubuntu.com" (Probably diskspace or something — but who knows? :-)). And because the sources pointed to the former, it broke the bootstrap process on new and existing EC2 instances and Vagrant VMs for us. A truely consistent experience!

Whenever apt-get update is run in a chef-recipe and it exists with a non-zero status, the process is stopped. Of course there are ways to work around it (for example: ignore_failure true), but then again, most of these workarounds are hacks and not suitable for a production environment (IMHO, of course): we often discover new sources from launchpad PPAs and so on and it's paramount to want to know if discovery failed. You cannot assume that all went well

Scalarium fixed their AMI already and updated the sources to point to "old-releases". Running instances are of course still broken.

Enter apt-repair-sources

apt-repair-sources is a small (opinionated) tool written in Ruby.

It offers:

  • --dry-run (-d), which is the default
  • --fix-it-for-me (-f), which attempts to correct all problems

The reason why apt-repair-sources was written in Ruby is, that I wanted a tool to run with only the most basic setup (on Scalarium). Since Ruby comes installed by default, it was my weapon of choice (vs. Python or PHP). Another advantage was that I had an opportunity to check out more Ruby (aside from cooking with chef) and used this project to learn more anything about testing in Ruby (using Test::Unit).

Dry run

A dry run can be used to essentially debug the sources on a system.

Here's the output of a dry-run, and all is well:

[email protected]:~/apt-repair-sources/bin$ ./apt-repair-sources 
There are no errors in /etc/apt/sources.list
There are no errors in /etc/apt/sources.list.d/chris-lea-node.js-lucid.list
There are no errors in /etc/apt/sources.list.d/node.list
There are no errors in /etc/apt/sources.list.d/chris-lea-redis-server.list
There are no errors in /etc/apt/sources.list.d/silverline.list

Here's the output of a system, where sources are currently broken:

[email protected]:~/apt-repair-sources/bin$ ./apt-repair-sources 
/etc/apt/sources.list: http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/karmic/main/binary-amd64/Packages.gz
/etc/apt/sources.list: http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/karmic/main/source/Sources.gz
/etc/apt/sources.list: http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/karmic-updates/main/binary-amd64/Packages.gz
/etc/apt/sources.list: http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/karmic-updates/main/source/Sources.gz
/etc/apt/sources.list: http://security.ubuntu.com/ubuntu/dists/karmic-security/main/binary-amd64/Packages.gz
/etc/apt/sources.list: http://security.ubuntu.com/ubuntu/dists/karmic-security/main/source/Sources.gz
There are no errors in /etc/apt/sources.list.d/gearman-developers-ppa-karmic.list
/etc/apt/sources.list.d/karmic-multiverse.list: http://archive.ubuntu.com/ubuntu/dists/karmic/multiverse/binary-amd64/Packages.gz
/etc/apt/sources.list.d/karmic-multiverse.list: http://archive.ubuntu.com/ubuntu/dists/karmic/multiverse/source/Sources.gz
/etc/apt/sources.list.d/karmic-multiverse.list: http://archive.ubuntu.com/ubuntu/dists/karmic-updates/multiverse/binary-amd64/Packages.gz
/etc/apt/sources.list.d/karmic-multiverse.list: http://archive.ubuntu.com/ubuntu/dists/karmic-updates/multiverse/source/Sources.gz
/etc/apt/sources.list.d/karmic-multiverse.list: http://security.ubuntu.com/ubuntu/dists/karmic-security/multiverse/binary-amd64/Packages.gz
/etc/apt/sources.list.d/karmic-multiverse.list: http://security.ubuntu.com/ubuntu/dists/karmic-security/multiverse/source/Sources.gz


Fix it for me

Fix it for me attempts to correct the sources like this:

  • sources with *.releases.ubuntu.com are moved to archive.ubuntu.com
  • sources with *.archive.ubuntu.com are moved to old-releases.ubuntu.com
  • sources with security.ubuntu.com are moved to old-releases.ubuntu.com

On top of these things, it will check Launchpad and third-party PPAs as well, if an issue is found, it'll just disable the entry in the sources file (by commenting it out: #).

Future releases will probably re-check commented out entries and also attempt to do some kind of sanity-checking of entries using the release name, etc.. These things are hard though and it might be the wrong approach to be opinionated here because e.g. Lucid packages sometimes also work on Karmic. Disabling these might break other things, etc..

Here's a run:

[email protected]:~/apt-repair-sources/bin$ sudo ./apt-repair-sources -f
[email protected]:~/apt-repair-sources/bin$ echo $?
[email protected]:~/apt-repair-sources/bin$ ./apt-repair-sources
There are no errors in /etc/apt/sources.list
There are no errors in /etc/apt/sources.list.d/gearman-developers-ppa-karmic.list
There are no errors in /etc/apt/sources.list.d/karmic-multiverse.list

Great success!


Both modes usually exit with zero (0), which makes it easy to include them for bootstrap processes, general trouble-shooting or periodic cronjobs etc..

Reason to not exit with 0:

  • attempt to run apt-repair-sources on another distro than Ubuntu
  • old-releases.ubuntu.com is down
  • you run with -d and -f (which of course makes no sense :-))
  • trollop (a rubygem i use for CLI option parsing is not found)



# sudo gem install apt-repair-sources


  • install Ruby Enterprise Edition (steal Karmic here; this should be your default anyway)
  • sudo gem install trollop (don't use what is in apt)
  • clone my repo: git clone git://github.com/lagged/apt-repair-sources.git
  • cd ./apt-repair/sources/bin && ./apt-repair-sources


  • create a gem
  • add support for Debian
  • improve my Ruby


Sure hope it's useful for someone else out there.

The code is on github, and I take pull-requests: https://github.com/lagged/apt-repair-sources

Just add blame

Sunday, April 5. 2009

I swore myself to only post meaningful stuff to this blog, so basically, no rants, ever. But!

The discussion revolving around Twitter and Rails (versus Scala) did remind me of something: If you've doing PHP for a while and know people who are programming not in PHP, you probably heard it all before. PHP guys (and girls) are being look down on and mocked by people of other programming and scripting languages. And that is despite ...

  1. ... the overall acceptance and distribution of the language
  2. ... the millions of free lines of code in open source projects and other code repositories
  3. ... the countless great examples of PHP in the enterprise
  4. and the amazingly short time to market when you develop with PHP

Those are a few good things to say about this fine language called PHP. Unfortunately they are never taken into consideration when you bash people who use it.

Instead you tell them about the short comings of several pieces of software such as phpBB, Mambo/Joomla and Wordpress who have had a lot of issues in the past but never the less are more popular than virtually any of their open source counterparts in the other languages.

Now, because a lot of Ruby people have a strong dis-like for PHP, you may think they deserve this. I don't think they, or Ruby, do.

I am perplexed how Alex Payne handles the current Scala/Ruby shoot out over at Twitter.

  • I'm amazed that someone with his experience and knowledge oversees the obvious short comings of several people who worked on the Twitter code base over the past years.

  • I'm amazed that he oversees all the crap they implemented.

  • I'm amazed that he oversees how they were and still are suffering from the most classic NIH.

And all to blame it on the language — Ruby.

Anyway, there may be plenty to argue about Ruby (or maybe just Rails). But blaming the language because of the inability of a developer (or multiple) is not the way to do it. It just makes you look very ignorant.

I sure hope Scala lives up to Alex' expectations, but if it doesn't — just blame it on Scala!