Bento and VirtualBox

Tuesday, April 23. 2013
Comments

Last week I blogged some Vagrant tips and pretty much jinxed the run I had in the past months.

Here's how:

  • I decided to upgrade to Vagrant 1.1, which broke bento: the current bento master is incompatible with Vagrant 1.1. But selecting the right rbenv env and installing the latest available Vagrant gem (inside the rbenv environment) fixed it.

  • My base box build, but for some reason, the guest addition setup broke and while it worked on Mac OSX, it broke the image completely on Ubuntu. Don't ask me why.

Especially the last bit reminded me to share another small tip, or work-around.

Fixing up base boxes

So every once in a while something doesn't work as expected. In my case, the guest additions were installed but not loaded and also failed to load when I started a VM. This in itself wasn't so bad, but it broke the vboxfs shares and while there may be cases where you don't need that (e.g. for a database VM), a VM without your code mounted into it is pretty useless.

The fix wasn't too hard:

  • I created a blank Vagrantfile (no recipes, just a simple box definition).
  • vagrant up and wait for the box to fail.
  • vagrant ssh to enter the box, or start with the GUI option and login through it.
  • execute sudo /etc/init.d/vboxadd setup

Then, exit the VM and execute the following: vagrant package vm_name --output box_name.box.

Import the box again with vagrant box add etc., update your Vagrantfile and test — then distribute.

Thoughs

Let me use this to reiterate on how critical a fixed set of versions are.

VirtualBox 4.2.x and bento (or veewee) seem to be incompatible somewhere and even though an error doesn't surface while the box is build or validated, it's still happening. It's paramount that there's always a rollback of some kind before you end up breaking too many things and stall your team for long.

Fin

That would be all for today.

Vagrant sans Ruby

Tuesday, June 5. 2012
Comments

Development, testing, staging and production — this is how most people devide up different environments for application development.

Maintenance and setup of these environments is often not a trivial goal to achieve. Having worked with a couple different code bases and setups over the last decade, I often noticed things like environment specific hacks (if ($env == 'testing') { ... }) in application code and service configurations and a lot of manual labour all around. It's still very common that code deployed to staging does not work — but it worked for you right?

And it's also too common that members of a team do not know how something works because whoever set it up is currently out sick or on vacation.

My opinion is that the best setup currently available is something like: chef(-solo) on the server and Vagrant on the desktop. Something like because aside from chef, there is also puppet, cfengine and a couple others. Leaving specific projects aside, it really just boils down to automation (within reason).

Automation in my opinion is not just the easiest but the only viable way to setup development, testing, staging and production environments. Without some automation in place it's required that all team members know how all of it works when maybe that is not yet important and the end goal is that all environments actually resemble each other.

Enter Vagrant

Vagrant is a very nifty toolkit to bootstrap Virtualbox images. Bootstrapping means installing your application stack into one or multiple virtual machines in order to resemble production a lot better.

So a lot of times when I rave about how useful Vagrant and chef are, I get the crazy eye from PHP developers:

You want me to learn Ruby to setup my local development environment?

My response:

  1. Do not fear the Ruby.
  2. You don't have to. (Well, not a whole lot!)

Real talk

First off, of course you need Virtualbox — get a download from their website or use your package manager.

Then, it's not possible to avoid Ruby a 100% — after all Vagrant is written in Ruby.

When you're on a Linux or a Mac, this is usually enough (you may need sudo unless you RVM):

$ sudo gem install vagrant

For a sudo-less install, use RVM:

$ sudo gem install rvm
...
$ rvm install
...
$ rvm use 1.8.7
...

When set up, this is what you do:

$ gem install vagrant
...

RVM allows me to run multiple versions of ruby side by side and also leverage local gem installs — per Ruby version. Think of it as a nifty way to run multiple PHP versions and have a PEAR install per version — similar projects exist for PHP today:

To learn more about RVM, visit their website.

Once vagrant is installed, we can continue!

ShellProvisioner for fun and profit

So, in case you are more comfortable writing some shell script for the time being — Vagrant got you covered!

First off, create a new Vagrantfile for your project:

$ mkdir -p Documents/workspaces/blog-example/
$ cd Documents/workspaces/blog-example
$ vagrant init lucid64
A `Vagrantfile` has been placed in this directory. You are now
ready to `vagrant up` your first virtual environment! Please read
the comments in the Vagrantfile as well as documentation on
`vagrantup.com` for more information on using Vagrant.

lucid64 (Ubuntu Lucid 10.04, 64bit) is the name of one of my local box files for Vagrant. In case you haven't got a box yet, head over to vagrantbox.es. There are a couple images you (aka box files) you can download. To get the lucid64 box, use the following command:

$ vagrant box add lucid64 http://files.vagrantup.com/lucid64.box

Once you made it past vagrant init, you should have a Vagrantfile with a lot of stuff in there.

While it's not important for my tutorial, I recommend you review it some (other time). The created Vagrantfile contains examples for all the provisioners (puppet, chef and shell) and a couple other configuration options, etc..

Let's skip over this and get to the ShellProvisioner.

This is all you need

First off, let's create a shell script setup.sh in the same directory and put something like this in it:

#!/bin/sh

apt-get update

apt-get install -y php5 php5-cli php-pear
hash -r

pear upgrade-all
pear install -f HTTP_Request2

What does it do?

  1. Update local package sources.
  2. Install php5, cli interpreter and PEAR installer
  3. reload environment
  4. upgrade all installed PEAR packages
  5. install PEAR HTTP_Request2

When you're done editing, make sure to chmod +x it.

Simple as that.

Putting your shell script to use

Replace your Vagrantfile with the following:

Vagrant::Config.run do |config|

  config.vm.box = "lucid64"
  config.vm.provision :shell, :path => "./setup.sh"

end

Then, run vagrant up and watch it provision. :)

Enter your virtual machine with vagrant ssh and verify HTTP_Request2 is installed:

$ vagrant ssh                                                                                                                                                               [14:16:08]
Linux lucid64 2.6.32-33-server #70-Ubuntu SMP Thu Jul 7 22:28:30 UTC 2011 x86_64 GNU/Linux
Ubuntu 10.04.3 LTS

Welcome to the Ubuntu Server!
 * Documentation:  http://www.ubuntu.com/server/doc
Last login: Thu Jul 21 14:08:15 2011 from 10.0.2.2
[email protected]:~$ pear list -c pear
Installed packages, channel pear.php.net:
=========================================
Package          Version State
Archive_Tar      1.3.10  stable
Console_Getopt   1.3.1   stable
HTTP_Request2    2.1.1   stable
Net_URL2         2.0.0   stable
PEAR             1.9.4   stable
Structures_Graph 1.0.4   stable
XML_Util         1.2.1   stable

Yay — your first steps into automation!

Fin

That's all for today — Vagrant with almost no Ruby. I hope this gets many more started!

VirtualBox Guest Additions and vagrant

Wednesday, May 25. 2011
Comments

If you followed my blog, you probably know about chef and vagrant.

So the other day I managed to upgrade to VirtualBox 4.0. The upgrade just happened by accident, so to speak. I noticed that Virtualbox 4.0 had moved from nonfree to contrib on Oracle's repository which is why I had previously missed it. With 4.0, I am now able to run the latest and greatest Vagrant — and with Vagrant being pre-1.0 it's safe to assume that newer is indeed better.

Anyway — the VirtualBox update fubar'd all of my boxes. They didn't outright refuse to start, but provisioning failed nine out of ten times and other random issues seemed to pile up.

I noticed this warning about the Guest Additions inside the box and how they are different from the VirtualBox installed on the host:

[default] The guest additions on this VM do not match the install version of
VirtualBox! This may cause things such as forwarded ports, shared
folders, and more to not work properly. If any of those things fail on
this machine, please update the guest additions and repackage the
box.

Guest Additions Version: 3.y.z
VirtualBox Version: 4.0.8

To cut to the chase — once I updated them and the box itself, they all continued to work.

So here's how!

Boot a blank VM

So first off, you take the image of your choice and boot a blank virtual machine. Blank because no chef-recipes should be configured, etc.. This VM should literally boot in a minute or so.

Now enter the VM and update the Guest Additions, and while we're at it, feel free to update the OS etc. as well.

Update the base

On my Ubuntu Karmic box, this is what I did (inside the VM) first:

vagrantup:~$ sudo aptitude update
...
vagrantup:~$ sudo aptitude upgrade
...
vagrantup:~$ sudo gem update --no-rdoc --no-test --no-ri
...
vagrantup:~$ sudo aptitude install portmap nfs-common
...

Pro-tip: The last step is necessary in case you run into the various issues with VirtualBox shared folders. The official website lists performance as the #1 one reason to use NFS shares instead of the VirtualBox shares, my #1 reason was issue #351.

Update Guest Additions

It took me a lot of Google, to actually find the download — regardless this is what seems to work. I extracted these steps from the blog post I found. For this to work, you are still inside the VM:

vagrantup:~$ wget -c \
http://download.virtualbox.org/virtualbox/4.0.8/VBoxGuestAdditions_4.0.8.iso \
-O VBoxGuestAdditions_4.0.8.iso
...
vagrantup:~$ sudo mount VBoxGuestAdditions_4.0.8.iso -o loop /mnt
vagrantup:~$ sudo sh /mnt/VBoxLinuxAdditions.run --nox11
...
vagrantup:~$ rm *.iso

There's an error about something failing to install (Windows-related), but I ignored it since I don't run the GUI anyway.

Pro-tip: Installing the latest vagrant along with Virtualbox 4.0.8 made the GUI mode fail consistently. Not that I really mind, but I feel compelled to share.

Repackage

Once these steps are completed, exit the VM (ctrl+d), halt the VM and up to test. If the VM starts up and no warnings show up, you got yourself a winner.

To repackage the VM, do the following:

~/karmic-test$ vagrant package

This creates a package.box file — I renamed mine to karmic-nfs.box. Import the box with:

~/karmic-test$ vagrant box add karmic-nfs karmic-nfs.box 
[vagrant] Downloading with Vagrant::Downloaders::File...
[vagrant] Copying box to temporary location...
[vagrant] Extracting box...
[vagrant] Verifying box...
[vagrant] Cleaning up downloaded box...

Done!

Fin

That's all, in summary the biggest obstacles are as follows:

  1. VirtualBox on the host and VirtualBox Guest Additions inside the box need to match (closely).
  2. The box needs to be kept up to date — more or less recent software inside the box makes bootstrapping and provisioning easier and faster.

Getting the most out of Chef with Scalarium and vagrant

Wednesday, March 9. 2011
Comments

Ever since I started playing around with Unix ~13 years ago, I've been a fan of automating things. What started out as writing little (maybe pointless) shell scripts slowly but surely morphed into infrastructure automation today.

As for my, or maybe anyone's, motivation to do these things, I see three main factors:

  • I'm easily bored — because repeating things is dull.
  • I'm easily distracted (when I'm bored).
  • I'm German: Of course we strive for perfection and excellence. ;-)

Being on Unix (or Linux) it's fairly simple to automate things — add some script-fu to bash or csh (or even better zsh) and off you go wrapping things into a small shell script! Then execute again and again!

Before we decided to moved to AWS (and RightScale) in late 2009 we had half a rack of servers (in a Peer1's POP in NYC) and never did any or much infrastructure automation. We had an image and a set of commands to get a server up and running, but it was far from two mouse-clicks today.

At the time, I had read about cfengine a couple of times, but datacenter-grade infrastructure management along with a rather steep learning wall (at that time anyway) seemed overkill. Add to that, that there is not a lot of time for research Fridays when you work in a small company.

Moving to AWS and RightScale required us to write lots of small shell scripts using bash and Ruby. When we moved from RightScale to Scalarium in late 2010, we went from shell scripts to Chef.

Using Chef meant that we created so-called recipes which are used to bootstrap our servers. Recipes are little Ruby scripts which live in a cookbook — open source projects are sometimes so creative. Before this move I had very little, or next to no, experience with Chef, and Chef being Ruby didn't exactly make me want to try it either.

So what exactly is Chef?

A Chef recipe is a tiny bit of Ruby code — essentially a high(er)-level wrapper around calls such as installing certain packages, creating and parsing configuration files and many other things.

Chef offers a robust abstraction about everything you can do with shell and with a little effort it's also possible to write recipes which run on multiple OS'. Supported are Ubuntu, CentOS, FreeBSD and others. For an intro to Chef see the slides of a talk I gave a couple weeks ago; I briefly blogged about it too.

Our Chef recipes currently include things like installing and configuring PHP (from source and through a launchpad repository), nginx, MySQL, CouchDB, haproxy and many other things. The list was literally growing every day for the first few weeks.


Continue reading "Getting the most out of Chef with Scalarium and vagrant"