Till's blog

One of my older hosting servers runs into space issues constantly. And most of the time it's a 24 GB log file (error_log) from a really old Joomla-based website.

Why it gets so huge? Well, when I tail the logfile, I see messages about functions being deprecated in PHP, warnings, notices and a whole lot more literally racing by. I'm not sure if Joomla changed in recent years, but most of the code-base is a great example of how PHP applications should not be build. Part of the problem in this particular case is that the system is hacked together and no one wants to update it. I'd argue another short-coming of the system since extensions actually require you to edit files and upgrade paths are not very clear.

Since it's a friend who's running the website, I haven't been very pressing on him to move to something more updated.

No space left on device

The usual quick fix is:

rm error_log && touch error_log && chown www:www error_log

Nice, and easy.

The only problem is that the system will usually take a while until recognizes that there is free space indeed.

Soft-updates?

FreeBSD recommends soft-updates — they have a lot of advantages. The only disadvantage is that e.g. the inodes will claim the address space until the write is completed. In case of a log file, this is usually when the process writing it is shut down.

Here's a system with soft-updates enabled:

server# mount
/dev/ad8s1a on / (ufs, local)
devfs on /dev (devfs, local)
/dev/ad8s1e on /tmp (ufs, local, soft-updates)
/dev/ad8s1g on /usr (ufs, local, soft-updates)
/dev/ad8s1d on /var (ufs, local, soft-updates)
/dev/ad8s1f on /web (ufs, local, soft-updates)
procfs on /proc (procfs, local)

The fix to making the system recognize the space is simple:

1. Restart your webserver, e.g.: /usr/local/etc/apache restart
2. Or run sync.

Fin

I hope I never ever forget this. Ever.

When I ran our setup on an instance the other day, I noticed how it failed with a "package not found" (or similar) error. After debugging this a bit, we discovered that Karmic moved from "archive.ubuntu.com" to "old-releases.ubuntu.com" (Probably diskspace or something — but who knows? :-)). And because the sources pointed to the former, it broke the bootstrap process on new and existing EC2 instances and Vagrant VMs for us. A truely consistent experience!

Whenever apt-get update is run in a chef-recipe and it exists with a non-zero status, the process is stopped. Of course there are ways to work around it (for example: ignore_failure true), but then again, most of these workarounds are hacks and not suitable for a production environment (IMHO, of course): we often discover new sources from launchpad PPAs and so on and it's paramount to want to know if discovery failed. You cannot assume that all went well

Scalarium fixed their AMI already and updated the sources to point to "old-releases". Running instances are of course still broken.

Enter apt-repair-sources

apt-repair-sources is a small (opinionated) tool written in Ruby.

It offers:

• --dry-run (-d), which is the default
• --fix-it-for-me (-f), which attempts to correct all problems

The reason why apt-repair-sources was written in Ruby is, that I wanted a tool to run with only the most basic setup (on Scalarium). Since Ruby comes installed by default, it was my weapon of choice (vs. Python or PHP). Another advantage was that I had an opportunity to check out more Ruby (aside from cooking with chef) and used this project to learn more anything about testing in Ruby (using Test::Unit).

Dry run

A dry run can be used to essentially debug the sources on a system.

Here's the output of a dry-run, and all is well:

till@dev:~/apt-repair-sources/bin$ ./apt-repair-sources 
There are no errors in /etc/apt/sources.list
There are no errors in /etc/apt/sources.list.d/chris-lea-node.js-lucid.list
There are no errors in /etc/apt/sources.list.d/node.list
There are no errors in /etc/apt/sources.list.d/chris-lea-redis-server.list
There are no errors in /etc/apt/sources.list.d/silverline.list

Here's the output of a system, where sources are currently broken:

tillklampaeckel@ulic:~/apt-repair-sources/bin$ ./apt-repair-sources 
/etc/apt/sources.list: http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/karmic/main/binary-amd64/Packages.gz
/etc/apt/sources.list: http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/karmic/main/source/Sources.gz
/etc/apt/sources.list: http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/karmic-updates/main/binary-amd64/Packages.gz
/etc/apt/sources.list: http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/karmic-updates/main/source/Sources.gz
/etc/apt/sources.list: http://security.ubuntu.com/ubuntu/dists/karmic-security/main/binary-amd64/Packages.gz
/etc/apt/sources.list: http://security.ubuntu.com/ubuntu/dists/karmic-security/main/source/Sources.gz
There are no errors in /etc/apt/sources.list.d/gearman-developers-ppa-karmic.list
/etc/apt/sources.list.d/karmic-multiverse.list: http://archive.ubuntu.com/ubuntu/dists/karmic/multiverse/binary-amd64/Packages.gz
/etc/apt/sources.list.d/karmic-multiverse.list: http://archive.ubuntu.com/ubuntu/dists/karmic/multiverse/source/Sources.gz
/etc/apt/sources.list.d/karmic-multiverse.list: http://archive.ubuntu.com/ubuntu/dists/karmic-updates/multiverse/binary-amd64/Packages.gz
/etc/apt/sources.list.d/karmic-multiverse.list: http://archive.ubuntu.com/ubuntu/dists/karmic-updates/multiverse/source/Sources.gz
/etc/apt/sources.list.d/karmic-multiverse.list: http://security.ubuntu.com/ubuntu/dists/karmic-security/multiverse/binary-amd64/Packages.gz
/etc/apt/sources.list.d/karmic-multiverse.list: http://security.ubuntu.com/ubuntu/dists/karmic-security/multiverse/source/Sources.gz

Problem?

Fix it for me

Fix it for me attempts to correct the sources like this:

• sources with *.releases.ubuntu.com are moved to archive.ubuntu.com
• sources with *.archive.ubuntu.com are moved to old-releases.ubuntu.com
• sources with security.ubuntu.com are moved to old-releases.ubuntu.com

On top of these things, it will check Launchpad and third-party PPAs as well, if an issue is found, it'll just disable the entry in the sources file (by commenting it out: #).

Future releases will probably re-check commented out entries and also attempt to do some kind of sanity-checking of entries using the release name, etc.. These things are hard though and it might be the wrong approach to be opinionated here because e.g. Lucid packages sometimes also work on Karmic. Disabling these might break other things, etc..

Here's a run:

tillklampaeckel@ulic:~/apt-repair-sources/bin$ sudo ./apt-repair-sources -f
tillklampaeckel@ulic:~/apt-repair-sources/bin$ echo $?
0
tillklampaeckel@ulic:~/apt-repair-sources/bin$ ./apt-repair-sources
There are no errors in /etc/apt/sources.list
There are no errors in /etc/apt/sources.list.d/gearman-developers-ppa-karmic.list
There are no errors in /etc/apt/sources.list.d/karmic-multiverse.list

Great success!

Automation

Both modes usually exit with zero (0), which makes it easy to include them for bootstrap processes, general trouble-shooting or periodic cronjobs etc..

Reason to not exit with 0:

• attempt to run apt-repair-sources on another distro than Ubuntu
• old-releases.ubuntu.com is down
• you run with -d and -f (which of course makes no sense :-))
• trollop (a rubygem i use for CLI option parsing is not found)

Setup

Gems!

# sudo gem install apt-repair-sources

Manually

• install Ruby Enterprise Edition (steal Karmic here; this should be your default anyway)
• sudo gem install trollop (don't use what is in apt)
• clone my repo: git clone git://github.com/lagged/apt-repair-sources.git
• cd ./apt-repair/sources/bin && ./apt-repair-sources

Todo

• create a gem
• add support for Debian
• improve my Ruby

Fin

Sure hope it's useful for someone else out there.

The code is on github, and I take pull-requests: https://github.com/lagged/apt-repair-sources

So, every once in a while I find myself in need of trying out newer, maybe, not-yet-released features in PHP. For example, recently, I wanted to test RoundCube PHP6 — this is how I did it.

On a side note, the same setup would also work for testing code with previous versions of PHP.

Toolbox

I used nginx and the PHP source with a little bit of ./configure and make — for kicks!

My O.S. of choice is FreeBSD and therefor the installation steps covered are tailored to it. With a small amount of Linux/Unix-fu, anyone should make it work on another distribution regardless.

Install nginx

First off, install nginx. On FreeBSD, this should be all:

• cd /usr/ports/www/nginx-devel && make install distclean

On other systems, this maybe a:

• apt-get install nginx
• emerg nginx
• rpm -shoot-myself nginx

The next step includes the infamous spawn-fcgi which many people use to control the php-cgi processes. A lot of tutorials on the net suggest to install lighttpd because it's a part of it, but on FreeBSD, you may skip that and do the following instead:

• cd /usr/ports/www/spawn-cgi && make install distclean

Pretty cool, huh?

So once this is done, the usual tasks need to be completed — in no particular order:

• edit the nginx config and enable fastcgi in nginx (/usr/local/etc/nginx/nginx.conf)
• enable nginx itself in /etc/rc.conf (nginx_enable="YES")
• get another nifty start script (see Shell Script) to wrap spawn-cgi

... and done!

Frequently, you do client work and if you are fortunate enough, you can setup a development environment on your own server or your laptop (or whatever), tinker with the files, and templates, and so on — until it's all done.

And whenever you are done, it's time to move files.

Sounds easy? It sort of is!

Checklist

Here's a small check list of things to keep in mind when you move an installation.

1. You may need to fix up the configuration file in order to adjust all database related settings. It's located in app/etc/local.xml.

2. Certain directories will need to be made writable. Writable in this case means that the webserver has to be able to write into it. Since most of the standard PHP setups are with mod_php, this will be the user www, apache or www-run (in most cases). If you (or your provider) runs php-cgi, chances are that this is not necessary.

   You may have to ask an administrator to set this up. If the administrator is not available, you can always chmod 777 these, but for obvious reasons this may be risky in a shared environment.

   The following is a list of directories (and contents) which need to be made writable:

   • app/etc/
   • var/
   • media/

   And in case you want to use MagentoConnect, there are additional directories to make writable. Please see this blog post for more info.

3. URLs, URLs, URLs. You were working on localhost all this time and now you think you can move it over just like that? Wrong!!! ;-) But all you need to do is edit the following keys in the core_config_data table:

   • web/unsecure/base_url
   • web/secure/base_url

That's all folks!

If you have any additions, please leave a comment!

Since part one and two were uber-successful, here's an update on my Zend Framework PHP performance situation. I've also had this post sitting around since beginning of May and I figured if I don't post it now, I never will.

Disclaimer: All numbers (aka pseudo benchmarks) were not taken on a full moon and are (of course) very relative to our server hardware (e.g. DELL 1950, 8 GB RAM) and environment. The application we run is Zend Framework-based and currently handles between 150,000 and 200,000 visitors per day.

Why switch at all?

In January of this year (2009), we started investigating the 2.2 branch of the Apache webserver. Because we used Apache 1.3 for forever, we never had the need to upgrade to Apache 2.0, or 2.2. After all, you're probably familiar with the don't fix it, if it's not broken-approach.

Late last year we ran into a couple (maybe) rather FreeBSD-specific issues with PHP and its opcode cache APC. I am by no means an expert on the entire situation, but from reading mailing lists and investigating on the server, this seemed to be expected behavior — in a nutshell: Apache 1.3 and a large opcode cache on a newer versions of FreeBSD (7) were bound to fail with larger amounts of traffic.

We tried bumping up a few settings (pv entries), but we just ran into the same issue again and again.

Because the architecture of Apache 2.2 and 1.3 is so different from one another (and upgrading to 2.2 was the proposed solution), I went on to explore this upgrade to Apache 2.2. And once I completed the switch to Apache 2.2, my issues went away.

So far, so good!

Performance?

On the performance side we experienced rather mediocre results.

While we benched that a static file could be read at around 300 requests per second (that is a pretty standard Apache 2.2 install, sans a couple unnecessary modules), PHP (mod_php) performed at a fraction of that, averaging between 20 and 23 requests per second.

Myth: Hardware is cheap(, developer time is not)!

Before some people yell at me for trying to optimize my web server, one needs to take the costs of scaling (to a 100 requests per seconds) into account.

One of those servers currently runs at 2,600.00 USD. The price tag adds up to an additional 10,400.00 USD in order to scale to a 100 (lousy) requests per seconds. Chances are of course, that the hardware is slightly less expensive since DELL gives great rebates — but the 8 GB of (server) RAM and the SAS disks by themselves are melting budgets away.

And on top of all hardware costs, you need add setup, maintenance and running costs (rack space, electricity) for an additional four servers — suddenly, developer time is cheap. ;-)

So what do we do? Nginx to the rescue?!