Getting the most out of Chef with Scalarium and vagrant

Wednesday, March 9. 2011
Comments

Ever since I started playing around with Unix ~13 years ago, I've been a fan of automating things. What started out as writing little (maybe pointless) shell scripts slowly but surely morphed into infrastructure automation today.

As for my, or maybe anyone's, motivation to do these things, I see three main factors:

  • I'm easily bored — because repeating things is dull.
  • I'm easily distracted (when I'm bored).
  • I'm German: Of course we strive for perfection and excellence. ;-)

Being on Unix (or Linux) it's fairly simple to automate things — add some script-fu to bash or csh (or even better zsh) and off you go wrapping things into a small shell script! Then execute again and again!

Before we decided to moved to AWS (and RightScale) in late 2009 we had half a rack of servers (in a Peer1's POP in NYC) and never did any or much infrastructure automation. We had an image and a set of commands to get a server up and running, but it was far from two mouse-clicks today.

At the time, I had read about cfengine a couple of times, but datacenter-grade infrastructure management along with a rather steep learning wall (at that time anyway) seemed overkill. Add to that, that there is not a lot of time for research Fridays when you work in a small company.

Moving to AWS and RightScale required us to write lots of small shell scripts using bash and Ruby. When we moved from RightScale to Scalarium in late 2010, we went from shell scripts to Chef.

Using Chef meant that we created so-called recipes which are used to bootstrap our servers. Recipes are little Ruby scripts which live in a cookbook — open source projects are sometimes so creative. Before this move I had very little, or next to no, experience with Chef, and Chef being Ruby didn't exactly make me want to try it either.

So what exactly is Chef?

A Chef recipe is a tiny bit of Ruby code — essentially a high(er)-level wrapper around calls such as installing certain packages, creating and parsing configuration files and many other things.

Chef offers a robust abstraction about everything you can do with shell and with a little effort it's also possible to write recipes which run on multiple OS'. Supported are Ubuntu, CentOS, FreeBSD and others. For an intro to Chef see the slides of a talk I gave a couple weeks ago; I briefly blogged about it too.

Our Chef recipes currently include things like installing and configuring PHP (from source and through a launchpad repository), nginx, MySQL, CouchDB, haproxy and many other things. The list was literally growing every day for the first few weeks.


Continue reading "Getting the most out of Chef with Scalarium and vagrant"

Giganews VPN on Ubuntu

Sunday, February 20. 2011
Comments

This article briefly describes how you can setup the Giganews VPN (PPTP) on Ubuntu. I'm gonna assume 10.04 (that's what I tested with) and the gnome desktop.

Does this sound familiar: The VPN connection 'xxxxx' failed because there were no valid VPN secrets.

If so then I'm hoping this article will help you.

What is a VPN for?

The use-case for a VPN — and this is for basically any VPN (not just Giganews' service) — is security. For example, for online banking when you use public WIFI at airports, train stations or your favorite coffee place.

Unless the VPN service itself is blocked, a VPN also provides real and unfiltered Internet access. No bogus blocks or censorships because your goverment thought they need to protect you.

From a development perspective using different VPNs also allow me to easily test location-based services. And it doesn't stop there — for example since I work for people in the U.S. we often run into issues with services which we use which do things different to me because of my location. A VPN in the U.S. fixes that for me.

And there are more and more use-cases once you think about it.


Continue reading "Giganews VPN on Ubuntu"

Find space hogs and prettify output using AWK

Saturday, September 18. 2010
Comments

I really love awk.

You might disagree and call me crazy, but while awk might be a royal brainfuck at first, here's a very simple example of its power which should explain my endorsement.

Figuring out space hogs

Every once in a while I run out of diskspace on /home. Even though I am the only user on this laptop I'm always puzzled as of why and I start running du trying to figure out which install or program stole my diskspace.

Here's a example of how I start it off in $HOME: du -h --max-depth 1

If I run the above line in my $HOME directory, I get a pretty list of lies — and thanks to -h this list is including more or less useful SI units, e.g. G(B), M(B) and K(B).

However, since I have a gazillion folders in my $HOME directory, the list is too long to figure out the biggest offenders, so naturally, I pipe my du command to sort -n. This doesn't work for the following reason:

[email protected]:~$ du -h --max-depth 1|sort -n
(...)
2.5M    ./net_gearman
2.6M    ./logs
2.7M    ./.gconf
2.8M    ./.openoffice.org2
3.3G    ./.config
3.3M    ./ubuntu
(...)

The order of the files is a little screwed up. As you see .config ate 3.3 GB and listed before ubuntu, which is only 3.3 MB in size. The reason is that sort -n (-n is numeric sort) doesn't take the unit into account. It compares the string and all of the sudden it makes sense why 3.3G is listed before 3.3M.

This is what I tried to fix this: du --max-depth 1|sort -n

The above command omits the human readable SI units (-h), and the list is sorted. Yay. Case closed?

AWK to the rescue

In the end, I'm still human, and therefor I want to see those SI units to make sense of the output and I want to see them in the correct order:

du --max-depth 1|sort -n|awk '{ $1=$1/1024; printf "%.2f MB: %s\n",$1,$2 }'

In detail

Let me explain the awk command:

  • Whenever you pipe output to awk, it breaks the line into multiple variables. This is incredible useful as you can avoid grep'ing and parsing the hell out of simple strings. $0 is the entire line, then $1, $2, etc. — awk magically divided the string by _whitespace. As an example, "Hello World" piped to awk would be $0 equals "Hello World", $1 equals "Hello" and $2 equals "World".
[email protected]:~$ echo "Hello World" |awk '{ print $0 }'
Hello World
[email protected]:~$ echo "Hello World" |awk '{ print $1 }'
Hello
[email protected]ll-laptop:~$ echo "Hello World" |awk '{ print $2 }'
World
  • My awk command uses $1 (which contains the size in raw kilobytes) and devides it by 1024 to receive megabytes. No rocket science!
  • printf outputs the string and while outputting we round the number (to two decimals: %.2f) and display the name of the folder which is still in $2.

All of the above is not just simple, but it should look somewhat familiar when you have a development background. Even shell allows you to divide a number and offers a printf function for formatting purposes.

Fin

Tada!

I hope awk is a little less confusing now. For further reading, I recommend the GNU AWK User Guide. (Or maybe just keep it open next time you think you can put awk to good use.)

Installing Varnish on Ubuntu Hardy

Tuesday, September 14. 2010
Comments

This is a quick and dirty rundown on how to install Varnish 2.1.x on Ubuntu Hardy (8.04 LTS).

Get sources setup

Add the repository to /etc/apt/sources.list:

deb http://repo.varnish-cache.org/ubuntu/ hardy Varnish-2.1 

Import the key for the new repository:

gpg --keyserver wwwkeys.eu.pgp.net --recv-keys 60E7C096C4DEFFEB
gpg --armor --export 60E7C096C4DEFFEB | apt-key add -

Installation

Update sources list and install varnish:

apt-get update
apt-get install varnish

Files of importance:

/etc/default/varnish
/etc/varnish/default.vcl
/etc/init.d/varnish

Double-check:

[email protected]:~# varnishd -V
varnishd (varnish-2.1.2 SVN )
Copyright (c) 2006-2009 Linpro AS / Verdens Gang AS

Further reading

I recommend a mix of the following websites/links:

Fin

That's all!

Selenium & Saucelenium: installation and dbus-xorg-woes

Tuesday, September 7. 2010
Comments

We're about to launch a new product, and this time it's pretty client-side-intense. The application is powered by a lot of JavaScript(-mvc) and jQuery, which do xhr calls to a ZF/CouchDB powered backend. While js-mvc has unit-testing sort of covetred, I was also looking for some integration testing, multiple browsers and all that.

Selenium vs. Saucelenium

I can't really say if you want one or the other. Revisiting Selenium in general, it's IMHO the only viable and suitable thing for a PHP shop. Primarily of course because all those nifty test cases will integrate into our existing suite of PHPUnit/phpt tests. And while I use Zend_Test already or find projects like SimpleTest's browser or even Perl's www::mechanize very appealing, neither of those executes JavaScript like a browser.

Selenium and Saucelenium have the same root — in fact Saucelenium is a Selenium fork. While the Selenium project seems to focus on 2.x currently, stable 1.x development seems to really happen at Saucelabs. That is if you call a commit from January 22nd of this year active development.

In the process of selecting one or the other, more people recommended that I'd use the Saucelabs distribution than the original one, and so I forked it on github.

Installation

Along with a script to start the damn thing, my fork also contains a README.md. Said README covers the installation part in detail, so I won't have to repeat myself here. All of this is pretty Ubuntu-centric and has been tested on Karmic Koala. I expect things to work just as well on Lucid, or on any other distribution if you get the installation right.

One thing that took me a while to figure out was the following error message:

[email protected]:/usr/src/saucelenium$ sudo ./start.sh 
[...]
[config/dbus] couldn't take over org.x.config: org.freedesktop.DBus.Error.AccessDenied (Connection ":1.17" is not allowed to own the service "org.x.config.display0" due to security     policies in the configuration file)
(EE) config/hal: couldn't initialise context: unknown error (null)
13:17:23.166 INFO - Writing debug logs to /var/log/selenium.log
13:17:23.166 INFO - Java: Sun Microsystems Inc. 16.0-b13
13:17:23.166 INFO - OS: Linux 2.6.32.1-rscloud amd64
13:17:23.176 INFO - v1.0.1 [exported], with Core [email protected]@ [@[email protected]]
13:17:23.296 INFO - Version Jetty/5.1.x
13:17:23.306 INFO - Started HttpContext[/selenium-server/driver,/selenium-server/driver]
13:17:23.306 INFO - Started HttpContext[/selenium-server,/selenium-server]
13:17:23.316 INFO - Started HttpContext[/,/]
13:17:23.326 INFO - Started SocketListener on 0.0.0.0:4444
13:17:23.326 INFO - Started [email protected]
^C13:18:09.796 INFO - Shutting down...

After Google'ing for a bit, I came to the conclusion that the above means that I didn't have the xserver installed.

The fix was rather simple: aptitude install xserver-xorg.

Example test

The following is an example test case. It'll open http://www.php.net/ and make sure it finds "What is PHP?" somewhere on that page.

Then it will continue to /downloads.php (by clicking on that link) and will make sure it finds "Binaries for other systems" on that page.

To run this test, execute: phpunit ExampleTestCase.php.

class ExampleTestCase extends PHPUnit_Extensions_SeleniumTestCase
{
    protected function setUp()
    {
        $this->setHost('localhost');
        $this->setPort(4444);
        $this->setBrowser("*chrome");
        $this->setBrowserUrl("http://www.php.net/");
    }

    public function testPHP()
    {
        $this->open('/');
        $this->waitForPageToLoad('30000');
        $this->assertTextPresent('What is PHP?');

        $this->click('//a[@href="/downloads.php"]');
        $this->waitForPageToLoad('30000');
        $this->assertTextPresent('Binaries for other systems');

        // check this out, especially useful for debugging:
        $this->assertEquals('http://www.php.net/downloads.php', $this->drivers[0]->getLocation());
    }
}

That's all.

Thanks for reading, and until next time.