Ubuntu: setting up redis-server (reloaded)

Monday, November 28. 2011

I blogged about setting up redis before, here's a small update to the situation:

If you're running Ubuntu 9.10 or 10.04.x, the most recent release available through aptitude is 1.x. Version 1.x doesn't include many things (pubsub, etc.) which are available in 2.x. So this leaves you with two options!


Compiling redis-server is simple (aka, make && sudo make install), a few things were missing (last time I checked):

Using a PPA!

The PPA (Personal Package Archive) concept is great. In a nutshell, it allows anyone to publish packages for various (Ubuntu) distributions. You upload a definition (sort of) and leverage Canonical's build farm, etc.. There are a few downsides to Launchpad: it's confusing, your PPA only holds the latest version of your build and overall publishing packages could be a lot easier. Which is why I'm so thankful for everyone who takes the time to publish.

Chris Lea seems to build most of the software I need. And he put all packages online in various Launchpad PPAs! Kudos to him!

Here's how you get setup!

Discover the Launchpad PPA:

server# sudo add-apt-repository ppa:chris-lea/redis-server

Update sources and install redis!

server# sudo apt-get update
server# sudo apt-get install redis-server

Done! ;-) All things (start script, configs) included!


In case that's too much type or you're looking for some automation, I also have a chef-recipe to get this done!

nginx configuration gotchas

Tuesday, April 5. 2011

After running away screaming from Zend_XmlRpc we migrated of our internal webservices are RESTful nowadays — which implies that we make heavy use of HTTP status codes and so on.

On the PHP side of things we implemented almost all of those webservices using the Zend Framework where some parts are replaced by in-house replacements (mostly stripped-down and optimized versions equivalents of Zend_Foo) and a couple nifty PEAR packages.

RESTful — how does it work?

Building a RESTful API means to adhere to the HTTP standard. URLs are resources and the appropriate methods (DELETE, GET, POST, PUT) are used on them. Add status codes in the mix and ready you are.

To keep it simple for this blog post the following status codes are more or less relevant for a read-only (GET) API:

  • 200: it's A-OK
  • 400: a bad request, e.g. a parameter missing
  • 401: unauthorized
  • 404: nothing was found

... and this is just the beginning — check out a complete list of HTTP status codes.


To serve PHP almost all of our application servers are setup like the following:

  1. nginx in the front
  2. php (fpm) processes in the back

Nginx and PHP are glued together using fastcgi (and unix-domain sockets).

For an indepth example of our setup check out the nginx-app and php-fpm recipes (along with our launchpad repository).


The other day, I noticed that for some reason whenever our API returned an error — e.g. a 404, for an empty result — it would display a standard nginx error page and not the actual response.


Digging around in /etc/nginx/fastcgi_params, I discovered the following:

fastcgi_intercept_errors on;

So what this does is that it intercepts any errors from the PHP backends and attempts to display an nginx error page. All errors may include the various PHP parse errors but apparently also a PHP generated page with a 404 status code.

So for example, the following code served by a PHP backend triggers the nginx page:

header("HTTP/1.1 404 Not Found);

The obvious fix seems simple:

fastcgi_intercept_errors off;

Sidenote: I think a similar issue might be in nginx's proxy_intercept_errors.

For both directives the manual suggests that they will intercept any status code higher than 400 — or 4xx and 5xx. But that's not all.

Tell me why?!

Reviewing the manual, I noticed that nginx will only act on fastcgi_intercept_errors on; when an error_page is (previously) defined. Checking out the rest of my sites configuration, the following part is to blame:

location / {
    error_page 404 /index.php;

    include /etc/nginx/fastcgi_params;

    fastcgi_pass  phpfpm;
    fastcgi_index index.php;

    fastcgi_param SCRIPT_FILENAME /var/www/current/www/index.php;

    index  index.php index.html index.htm;

So indeed the error_page 404 /index.php is what set it all off to begin with. And that's what I ended up removing, though it doesn't hurt to understand the implications of fastcgi_intercept_errors.

I think historically we used the 404 error handler as a cheap excuse for a rewrite rule since we only serve dynamically generated pages (and super-SEO-friendly URLs) to begin with. But that doesn't seem to be necessary — testing will be required.


The moral of the story is: nginx is really never to blame. ;-)

This is precisly what happens when you copy/paste configurations from the Internetz and don't review each and every single line to understand the full scope. In the end this was more or less a picnic on my part but I wanted to share it anyway because it was one of those WTF-moments for me.

Giganews VPN on Ubuntu

Sunday, February 20. 2011

This article briefly describes how you can setup the Giganews VPN (PPTP) on Ubuntu. I'm gonna assume 10.04 (that's what I tested with) and the gnome desktop.

Does this sound familiar: The VPN connection 'xxxxx' failed because there were no valid VPN secrets.

If so then I'm hoping this article will help you.

What is a VPN for?

The use-case for a VPN — and this is for basically any VPN (not just Giganews' service) — is security. For example, for online banking when you use public WIFI at airports, train stations or your favorite coffee place.

Unless the VPN service itself is blocked, a VPN also provides real and unfiltered Internet access. No bogus blocks or censorships because your goverment thought they need to protect you.

From a development perspective using different VPNs also allow me to easily test location-based services. And it doesn't stop there — for example since I work for people in the U.S. we often run into issues with services which we use which do things different to me because of my location. A VPN in the U.S. fixes that for me.

And there are more and more use-cases once you think about it.

Continue reading "Giganews VPN on Ubuntu"

EC2 security group owner ID

Sunday, May 9. 2010

I recently had the pleasure to setup an RDS instance and it took me a while to figure out what the --ec2-security-group-owner-id parameter needs to be populated with when you want to allow access to your RDS instance from instances with a certain security group.

To cut to the chase, you need to log into AWS and then click the following link — done.

A toolchain for CouchDB Lounge

Friday, February 26. 2010

One of our biggest issues with CouchDB is currently the lack of compaction of our database, and by lack of, I don't mean that CouchDB doesn't support it, I mean that we are unable to actually run it.

Compaction in a nutshell

Compaction in a nutshell is pretty cool.

As you know, CouchDB is not very space-efficient. For once, CouchDB saves revisions of all documents. Which means, whenever you update a document a new revision is saved. You can rollback any time, or expose it as a nifty feature in your application — regardless, those revisions are kept around until your database is compacted.

Think about it in terms of IMAP - emails are not deleted until you hit that magic "compact" button which 99% of all people who use IMAP don't know what it's for anyway.

Another thing is that whenever new documents are written to CouchDB and bulk mode is not used, it'll save them in a way which is not very efficient either. In terms of actual storage and indexing (so rumour has it).

Compaction woes

Since everything is simple with CouchDB, compaction is a simple process in CouchDB too. Yay!

When compaction is started, CouchDB will create a new database file where it stores the data in a very optimized way (I will not detail on this, go read a science book or google if you are really interested in this!). When the compaction process finished, CouchDB will exchange your old database file with the new database file.

The woes start with that e.g. when you have 700 GB uncompacted data, you will probably need another 400 GB for compaction to finish because it will create a second database file.

The second issue is that when you have constant writing on your database, the compaction process will actually never finish. It kind of sucks and for those people who aim to provide close to 100% availability, this is extremely painful to learn.

Continue reading "A toolchain for CouchDB Lounge"