till's blog

The scope of this blog entry is to give you a quick and dirty demo for start-stop-daemon together with a short use case on Gearman (all on Ubuntu). In this example, I'm using the start-stop-daemon to handle my Gearman workers through an init.d script.

Gearman

Gearman is a queue! But unlike for example most of the backends to Zend_Queue, Gearman provides a little more than just a message queue to send — well — messages from sender to receiver. With Gearman it's trivial to register functions (tasks) on the server to make in order to start a job and to get stuff done.

For me the biggest advantages of Gearman are that it's easy to scale (add a server, start more workers) and that you can get work done in another language without building an API of some sort in between. Gearman is that API.

Back to start-stop-daemon

start-stop-daemon is a facility to start and stop programs on system start and shutdown. On recent Ubuntus most of the scripts located in /etc/init.d/ make use of it already. It provides a simple high-level API to system calls — such as stopping a process, starting it in a background, running it under a user and the glue, such as writing a pid file.

My gearman start script

Once adjusted, register it with the rc-system: update-rc.d script defaults. This will take care of the script being run during the boot process and before shutdown is completed.

A little more detail

The script may be called with /etc/init.d/script start|stop|restart (the pipes designated "or").

Upon start, we write a pidfile to /var/run and start the process. The same pidfile is used on stop — simple as that. The rest of it is hidden behind start-stop-daemon which takes care of the ugly rest for us.

Caching is one of the things recommended whenever something is slow — "Your [database, website, server]? Well, duh! You need a cache!".

All things aside, it's not always easy to cache stuff. I find myself often in situations where I can't cache at all or where a caching solution is complex as hell to implement. All of the sudden you need to dissect your page and lazy load half of it with Ajax — bah. And that's just because those users never want to wait for a cache to expire or invalidate. They just expect the most recent ever! :-)

Sometimes however, caching can be so trivial, it even hurts.

Bypass the application server

There are lots of different techniques and strategies to employ when you start to think about caching. The fastest albeit not always available method is to bypass your app server stack completely. And here's how. :-)

An example

My example is a pet project of mine where I display screenshots of different news outlets which are taken every 3 (three) hours — 0:00, 3:00 AM, 6:00 AM, 9:00 AM, 12:00 PM, 3:00 PM, 6:00 PM, 9:00 PM and so on. In between those fixed dates, the page basically never changes and why would I need PHP to render a page if it didn't change since the last request?

Correct, I don't! :)

And here's what I did to setup my cache:

• Apache 1.3.x (!) and mod_php5
• my homepage: docroot/home.php
• httpd.conf: DirectoryIndex index.html home.php
• a cronjob: */10 * * * * fetch -q -o docroot/index.html http://example.org/home.php

In greater detail

Homepage

The home.php does all the PHP funkyness whenever I need a fresh rendered version of my website to track an error, or adjust something.

DirectoryIndex

If I ever delete my cache (index.html), my website will still be available. The DirectoryIndex will use home.php next and the request will be a little slower and also more expensive on my side, but the website will continue to work.

Cronjob

The cronjob will issue a HTTP request (GET) using fetch against my homepage and save the result to my index.html. It's really so simple, it hurts. Currently, this cronjob is executed every 10 minutes so I can fiddle with the design and deploy a change more easily, but I could run that cronjob every hour or every few hours as well.

If you don't have fetch, try the following wget command:

wget -q -O docroot/index.html http://example.org/home.php

Fin

That's all, a really simple cache which bypasses your application server. Enjoy!

If you're in for another nifty solution, I suggest you read Brandon Savage's article on caching with the Zend Framework and/or take a look at nginx+Memcached.

So I posted something on Twitter today, which wasn't exactly news to me. I was more or less pointing out the obvious.

From a couple follow-up tweets, I guess, I need to explain more.

The idea

My thesis is that there's a gain in page rendering time when I disable automatic view rendering and use explicit render calls ($this->render('foo');) inside my controllers. And to cut to the chase, there is. On our app, I measured a 12% improvement using Xdebug's profiler — simple before-after-style.

General setup

I've blogged about Zend Framework performance before (1, 2, 3). Our setup is not the average Zend Framework quickstart application. We utilize a custom (much faster) loader (my public open source work in progress), no Zend_Application and explicit (vs. implicit) view rendering. The framework code is at 1.10.2. On the server-side, the application servers are nginx+php(-cgi).

I don't feel like repeating myself and while a lot of issues were already addressed in new releases of the framework, or are going to be addressed in 2.0, the above links still hold a lot of truth or at least inside and pointers if you're interested in general PHP performance (in German).

Code

IMHO, it doesn't really matter how the rest of your application looks like. Of course all applications are different and that's why I didn't say, "OMG my page rendered in 100 ms", but instead I said something like, "we got a 10+% boost". The bottom line is that everyone wants to serve fast pages and get the most out of their hardware but since applications tend to carry different features there really is no holy grail or number to adhere to.

Proposal

I urge everyone to double-check my claim. After all, it's pretty simple:

1. Setup Xdebug
2. Profile the page
3. Restart PHP app server/processes (in case you use APC and/or php-cgi)
4. Disable automatic view rendering: $this->_helper->viewRenderer->setNoRender(true);
5. Add render() call: $this->render('foo');
6. Profile again

... simple as that.

Conclusion

All in all this thing doesn't require too much to follow.

Automatics — such as an automatic view renderer — add convenience which results in rapid development and hopefully shorter time to market. But they do so almost always (give it nine Erlang nines ;-)) at the expense of performance.

Update, 2010-03-20 21:37: As Rob pointed out, there's even more to gain by bypassing the helper entirely. Use the code snippet below, or consider something like the following:

Padraic also blogged extensively on Zend_Controller_Action_Helper_ViewRenderer, I recommend reading Having a bad ViewRenderer day in your ZF app?.

A small howto to get the latest redis-server and a webinterface on Ubuntu.

Installation

wget http://ftp.de.debian.org/debian/pool/main/r/redis/redis-server_1.2.5-1_amd64.deb
sudo dpkg -i redis-server_1.2.5-1_amd64.deb
/etc/init.d/redis-server start

... redis should listen on localhost:6379.

You may need to get i386 instead of amd64 if you run 32bit.

Tweaks

You may need to add the following to /etc/sysctl.conf:

vm.overcommit_memory = 1

... that is, especially if you run in a VE (e.g. inside xen).

All other configs are in /etc/redis/redis.conf.

Web

Because web interfaces are so simple, I decided to get redweb.

Dependencies

wget http://ftp.us.debian.org/debian/pool/main/p/python-support/python-support_1.0.7_all.deb
dpkg -i python-support_1.0.7_all.deb
wget http://ftp.us.debian.org/debian/pool/main/p/python-redis/python-redis_1.34.1-1_all.deb
dpkg -i python-redis_1.34.1-1_all.deb

So, on Ubuntu, python-support is at 0.8.4 currently, but we'll need something equal or greater than 0.9.0. This is why I update python-support from Debian.

Installation

git clone http://github.com/tnm/redweb ./redweb-git

Patch redweb-git/redweb/redweb.py with:

index e79a062..e278fca 100644
--- a/redweb/redweb.py
+++ b/redweb/redweb.py
@@ -15,6 +15,8 @@ __author__ = 'Ted Nyman'
 __version__ = '0.2.2'
 __license__ = 'MIT'

+import sys
+sys.path.append('/path/to/redweb-git/')

 from bottle import route, request, response, view, send_file, run
 import redis

Run!

cd redweb-git/redweb/
python redweb.py

... this is a bit annoying. If you do python redweb/redweb.py, it'll complain about missing files.

Then browse to http://127.0.0.1:8080.

Fin

So this is my redis-server howto — nice and simple.

And once you have Redis up and running, feel free to browse over to Rediska and use their session handling for Zend Framework. Setup is pretty simple and it works like a charm. :-) I'd suggest you use their trunk code, which is hosted on Github as it will contain a few improvements and a small bugfix which I did.

For more on Rediska, watch this space. ;-)

So for whatever reason, it took me a while to figure this out earlier today:

2010/03/15 15:44:16 [info] 22274#0: *148224 client closed prematurely connection, so upstream connection is closed too while sending request to upstream, client: a.a.a.a, server: localhost, request: "GET / HTTP/1.1", upstream: "fastcgi://unix:/tmp/.fastcgi.till/socket:", host: "localhost"
2010/03/15 15:44:16 [info] 22274#0: *148207 client closed prematurely connection, so upstream connection is closed too while sending request to upstream, client: a.a.a.a, server: localhost, request: "GET / HTTP/1.1", upstream: "fastcgi://unix:/tmp/.fastcgi.till/socket:", host: "localhost"

The issue was a PHP parse error which I overlooked when I added a new file. The weird thing is, I had nothing in the logs (E_ALL, display_errors is off, but all logs are enabled and I tailed them using multitail) and nginx only displayed a blank page. The errors above were in nginx's own log file.