Till's blog

Ever since I started playing around with Unix ~13 years ago, I've been a fan of automating things. What started out as writing little (maybe pointless) shell scripts slowly but surely morphed into infrastructure automation today.

As for my, or maybe anyone's, motivation to do these things, I see three main factors:

• I'm easily bored — because repeating things is dull.
• I'm easily distracted (when I'm bored).
• I'm German: Of course we strive for perfection and excellence. ;-)

Being on Unix (or Linux) it's fairly simple to automate things — add some script-fu to bash or csh (or even better zsh) and off you go wrapping things into a small shell script! Then execute again and again!

Before we decided to moved to AWS (and RightScale) in late 2009 we had half a rack of servers (in a Peer1's POP in NYC) and never did any or much infrastructure automation. We had an image and a set of commands to get a server up and running, but it was far from two mouse-clicks today.

At the time, I had read about cfengine a couple of times, but datacenter-grade infrastructure management along with a rather steep learning wall (at that time anyway) seemed overkill. Add to that, that there is not a lot of time for research Fridays when you work in a small company.

Moving to AWS and RightScale required us to write lots of small shell scripts using bash and Ruby. When we moved from RightScale to Scalarium in late 2010, we went from shell scripts to Chef.

Using Chef meant that we created so-called recipes which are used to bootstrap our servers. Recipes are little Ruby scripts which live in a cookbook — open source projects are sometimes so creative. Before this move I had very little, or next to no, experience with Chef, and Chef being Ruby didn't exactly make me want to try it either.

So what exactly is Chef?

A Chef recipe is a tiny bit of Ruby code — essentially a high(er)-level wrapper around calls such as installing certain packages, creating and parsing configuration files and many other things.

Chef offers a robust abstraction about everything you can do with shell and with a little effort it's also possible to write recipes which run on multiple OS'. Supported are Ubuntu, CentOS, FreeBSD and others. For an intro to Chef see the slides of a talk I gave a couple weeks ago; I briefly blogged about it too.

Our Chef recipes currently include things like installing and configuring PHP (from source and through a launchpad repository), nginx, MySQL, CouchDB, haproxy and many other things. The list was literally growing every day for the first few weeks.

This is a quick and dirty rundown on how to install Varnish 2.1.x on Ubuntu Hardy (8.04 LTS).

Get sources setup

Add the repository to /etc/apt/sources.list:

deb http://repo.varnish-cache.org/ubuntu/ hardy Varnish-2.1 

Import the key for the new repository:

gpg --keyserver wwwkeys.eu.pgp.net --recv-keys 60E7C096C4DEFFEB
gpg --armor --export 60E7C096C4DEFFEB | apt-key add -

Installation

Update sources list and install varnish:

apt-get update
apt-get install varnish

Files of importance:

/etc/default/varnish
/etc/varnish/default.vcl
/etc/init.d/varnish

Double-check:

root@server:~# varnishd -V
varnishd (varnish-2.1.2 SVN )
Copyright (c) 2006-2009 Linpro AS / Verdens Gang AS

Further reading

I recommend a mix of the following websites/links:

• Varnish 2.1 documentation
• Wiki with lots of tutorials and examples
• Team Soundcloud's varnishtop (code on Github)

Fin

That's all!

I updated my mother's Mac Mini from Tiger to Snow Leopard last night and I did not enounter many issues during the one-hour-long installation (or update) process.

In advance — in order to be save than sorry — we had bought an external hard drive and did a backup of her home folder (6 GB — I wish my own backup was that size ;-)).

The one thing I ran into was that after the installation finished, I tried to reconnect to our wifi — but if failed. It wouldn't even show the network.

Since I knew that hooking up the Mini using a cable required me to move everything from her desk in one room to another room, I gave up last night and had myself a beer instead.

A solution

This morning, I briefly researched the issue on Google and found a thread on apple.com where people suggested to do the following:

• Go into System preferences > Network
• Click on Airport
• Delete the "Automatic location" profile
• Add your own
• Disable/enable Airport
• Done

Well, that almost worked for me.

I also re-added our wifi to the preferred network list and we were instantly reconnected.

Anything else?

Aside from the wifi issues almost everything migrated without issues.

This is still a pre-Intel Mac Mini. All her email, pictures and documents are in place. The only thing I need to reinstall is something called Kodak Easy Share since whatever she has installed doesn't seem to work right now.

And as a bonus, Snow Leopard feels faster even on this aging little piece of technology.

Anyway, I just thought I'd share.

So, every once in a while I find myself in need of trying out newer, maybe, not-yet-released features in PHP. For example, recently, I wanted to test RoundCube PHP6 — this is how I did it.

On a side note, the same setup would also work for testing code with previous versions of PHP.

Toolbox

I used nginx and the PHP source with a little bit of ./configure and make — for kicks!

My O.S. of choice is FreeBSD and therefor the installation steps covered are tailored to it. With a small amount of Linux/Unix-fu, anyone should make it work on another distribution regardless.

Install nginx

First off, install nginx. On FreeBSD, this should be all:

• cd /usr/ports/www/nginx-devel && make install distclean

On other systems, this maybe a:

• apt-get install nginx
• emerg nginx
• rpm -shoot-myself nginx

The next step includes the infamous spawn-fcgi which many people use to control the php-cgi processes. A lot of tutorials on the net suggest to install lighttpd because it's a part of it, but on FreeBSD, you may skip that and do the following instead:

• cd /usr/ports/www/spawn-cgi && make install distclean

Pretty cool, huh?

So once this is done, the usual tasks need to be completed — in no particular order:

• edit the nginx config and enable fastcgi in nginx (/usr/local/etc/nginx/nginx.conf)
• enable nginx itself in /etc/rc.conf (nginx_enable="YES")
• get another nifty start script (see Shell Script) to wrap spawn-cgi

... and done!

Since part one and two were uber-successful, here's an update on my Zend Framework PHP performance situation. I've also had this post sitting around since beginning of May and I figured if I don't post it now, I never will.

Disclaimer: All numbers (aka pseudo benchmarks) were not taken on a full moon and are (of course) very relative to our server hardware (e.g. DELL 1950, 8 GB RAM) and environment. The application we run is Zend Framework-based and currently handles between 150,000 and 200,000 visitors per day.

Why switch at all?

In January of this year (2009), we started investigating the 2.2 branch of the Apache webserver. Because we used Apache 1.3 for forever, we never had the need to upgrade to Apache 2.0, or 2.2. After all, you're probably familiar with the don't fix it, if it's not broken-approach.

Late last year we ran into a couple (maybe) rather FreeBSD-specific issues with PHP and its opcode cache APC. I am by no means an expert on the entire situation, but from reading mailing lists and investigating on the server, this seemed to be expected behavior — in a nutshell: Apache 1.3 and a large opcode cache on a newer versions of FreeBSD (7) were bound to fail with larger amounts of traffic.

We tried bumping up a few settings (pv entries), but we just ran into the same issue again and again.

Because the architecture of Apache 2.2 and 1.3 is so different from one another (and upgrading to 2.2 was the proposed solution), I went on to explore this upgrade to Apache 2.2. And once I completed the switch to Apache 2.2, my issues went away.

So far, so good!

Performance?

On the performance side we experienced rather mediocre results.

While we benched that a static file could be read at around 300 requests per second (that is a pretty standard Apache 2.2 install, sans a couple unnecessary modules), PHP (mod_php) performed at a fraction of that, averaging between 20 and 23 requests per second.

Myth: Hardware is cheap(, developer time is not)!

Before some people yell at me for trying to optimize my web server, one needs to take the costs of scaling (to a 100 requests per seconds) into account.

One of those servers currently runs at 2,600.00 USD. The price tag adds up to an additional 10,400.00 USD in order to scale to a 100 (lousy) requests per seconds. Chances are of course, that the hardware is slightly less expensive since DELL gives great rebates — but the 8 GB of (server) RAM and the SAS disks by themselves are melting budgets away.

And on top of all hardware costs, you need add setup, maintenance and running costs (rack space, electricity) for an additional four servers — suddenly, developer time is cheap. ;-)

So what do we do? Nginx to the rescue?!