Ubuntu: setting up redis-server (reloaded)

Monday, November 28. 2011
Comments

I blogged about setting up redis before, here's a small update to the situation:

If you're running Ubuntu 9.10 or 10.04.x, the most recent release available through aptitude is 1.x. Version 1.x doesn't include many things (pubsub, etc.) which are available in 2.x. So this leaves you with two options!

Compile!

Compiling redis-server is simple (aka, make && sudo make install), a few things were missing (last time I checked):

Using a PPA!

The PPA (Personal Package Archive) concept is great. In a nutshell, it allows anyone to publish packages for various (Ubuntu) distributions. You upload a definition (sort of) and leverage Canonical's build farm, etc.. There are a few downsides to Launchpad: it's confusing, your PPA only holds the latest version of your build and overall publishing packages could be a lot easier. Which is why I'm so thankful for everyone who takes the time to publish.

Chris Lea seems to build most of the software I need. And he put all packages online in various Launchpad PPAs! Kudos to him!

Here's how you get setup!

Discover the Launchpad PPA:

server# sudo add-apt-repository ppa:chris-lea/redis-server
...

Update sources and install redis!

server# sudo apt-get update
...
server# sudo apt-get install redis-server
...

Done! ;-) All things (start script, configs) included!

Fin

In case that's too much type or you're looking for some automation, I also have a chef-recipe to get this done!

FreeBSD + softupdates + no space left on device

Friday, November 25. 2011
Comments

One of my older hosting servers runs into space issues constantly. And most of the time it's a 24 GB log file (error_log) from a really old Joomla-based website.

Why it gets so huge? Well, when I tail the logfile, I see messages about functions being deprecated in PHP, warnings, notices and a whole lot more literally racing by. I'm not sure if Joomla changed in recent years, but most of the code-base is a great example of how PHP applications should not be build. Part of the problem in this particular case is that the system is hacked together and no one wants to update it. I'd argue another short-coming of the system since extensions actually require you to edit files and upgrade paths are not very clear.

Since it's a friend who's running the website, I haven't been very pressing on him to move to something more updated.

No space left on device

The usual quick fix is:

rm error_log && touch error_log && chown www:www error_log

Nice, and easy.

The only problem is that the system will usually take a while until recognizes that there is free space indeed.

Soft-updates?

FreeBSD recommends soft-updates — they have a lot of advantages. The only disadvantage is that e.g. the inodes will claim the address space until the write is completed. In case of a log file, this is usually when the process writing it is shut down.

Here's a system with soft-updates enabled:

server# mount
/dev/ad8s1a on / (ufs, local)
devfs on /dev (devfs, local)
/dev/ad8s1e on /tmp (ufs, local, soft-updates)
/dev/ad8s1g on /usr (ufs, local, soft-updates)
/dev/ad8s1d on /var (ufs, local, soft-updates)
/dev/ad8s1f on /web (ufs, local, soft-updates)
procfs on /proc (procfs, local)

The fix to making the system recognize the space is simple:

  1. Restart your webserver, e.g.: /usr/local/etc/apache restart
  2. Or run sync.

Fin

I hope I never ever forget this. Ever.

apt-repair-sources on Ubuntu

Wednesday, November 23. 2011
Comments

When I ran our setup on an instance the other day, I noticed how it failed with a "package not found" (or similar) error. After debugging this a bit, we discovered that Karmic moved from "archive.ubuntu.com" to "old-releases.ubuntu.com" (Probably diskspace or something — but who knows? :-)). And because the sources pointed to the former, it broke the bootstrap process on new and existing EC2 instances and Vagrant VMs for us. A truely consistent experience!

Whenever apt-get update is run in a chef-recipe and it exists with a non-zero status, the process is stopped. Of course there are ways to work around it (for example: ignore_failure true), but then again, most of these workarounds are hacks and not suitable for a production environment (IMHO, of course): we often discover new sources from launchpad PPAs and so on and it's paramount to want to know if discovery failed. You cannot assume that all went well

Scalarium fixed their AMI already and updated the sources to point to "old-releases". Running instances are of course still broken.

Enter apt-repair-sources

apt-repair-sources is a small (opinionated) tool written in Ruby.

It offers:

  • --dry-run (-d), which is the default
  • --fix-it-for-me (-f), which attempts to correct all problems

The reason why apt-repair-sources was written in Ruby is, that I wanted a tool to run with only the most basic setup (on Scalarium). Since Ruby comes installed by default, it was my weapon of choice (vs. Python or PHP). Another advantage was that I had an opportunity to check out more Ruby (aside from cooking with chef) and used this project to learn more anything about testing in Ruby (using Test::Unit).

Dry run

A dry run can be used to essentially debug the sources on a system.

Here's the output of a dry-run, and all is well:

till@dev:~/apt-repair-sources/bin$ ./apt-repair-sources 
There are no errors in /etc/apt/sources.list
There are no errors in /etc/apt/sources.list.d/chris-lea-node.js-lucid.list
There are no errors in /etc/apt/sources.list.d/node.list
There are no errors in /etc/apt/sources.list.d/chris-lea-redis-server.list
There are no errors in /etc/apt/sources.list.d/silverline.list

Here's the output of a system, where sources are currently broken:

tillklampaeckel@ulic:~/apt-repair-sources/bin$ ./apt-repair-sources 
/etc/apt/sources.list: http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/karmic/main/binary-amd64/Packages.gz
/etc/apt/sources.list: http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/karmic/main/source/Sources.gz
/etc/apt/sources.list: http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/karmic-updates/main/binary-amd64/Packages.gz
/etc/apt/sources.list: http://us-east-1.ec2.archive.ubuntu.com/ubuntu/dists/karmic-updates/main/source/Sources.gz
/etc/apt/sources.list: http://security.ubuntu.com/ubuntu/dists/karmic-security/main/binary-amd64/Packages.gz
/etc/apt/sources.list: http://security.ubuntu.com/ubuntu/dists/karmic-security/main/source/Sources.gz
There are no errors in /etc/apt/sources.list.d/gearman-developers-ppa-karmic.list
/etc/apt/sources.list.d/karmic-multiverse.list: http://archive.ubuntu.com/ubuntu/dists/karmic/multiverse/binary-amd64/Packages.gz
/etc/apt/sources.list.d/karmic-multiverse.list: http://archive.ubuntu.com/ubuntu/dists/karmic/multiverse/source/Sources.gz
/etc/apt/sources.list.d/karmic-multiverse.list: http://archive.ubuntu.com/ubuntu/dists/karmic-updates/multiverse/binary-amd64/Packages.gz
/etc/apt/sources.list.d/karmic-multiverse.list: http://archive.ubuntu.com/ubuntu/dists/karmic-updates/multiverse/source/Sources.gz
/etc/apt/sources.list.d/karmic-multiverse.list: http://security.ubuntu.com/ubuntu/dists/karmic-security/multiverse/binary-amd64/Packages.gz
/etc/apt/sources.list.d/karmic-multiverse.list: http://security.ubuntu.com/ubuntu/dists/karmic-security/multiverse/source/Sources.gz

Problem?

Fix it for me

Fix it for me attempts to correct the sources like this:

  • sources with *.releases.ubuntu.com are moved to archive.ubuntu.com
  • sources with *.archive.ubuntu.com are moved to old-releases.ubuntu.com
  • sources with security.ubuntu.com are moved to old-releases.ubuntu.com

On top of these things, it will check Launchpad and third-party PPAs as well, if an issue is found, it'll just disable the entry in the sources file (by commenting it out: #).

Future releases will probably re-check commented out entries and also attempt to do some kind of sanity-checking of entries using the release name, etc.. These things are hard though and it might be the wrong approach to be opinionated here because e.g. Lucid packages sometimes also work on Karmic. Disabling these might break other things, etc..

Here's a run:

tillklampaeckel@ulic:~/apt-repair-sources/bin$ sudo ./apt-repair-sources -f
tillklampaeckel@ulic:~/apt-repair-sources/bin$ echo $?
0
tillklampaeckel@ulic:~/apt-repair-sources/bin$ ./apt-repair-sources
There are no errors in /etc/apt/sources.list
There are no errors in /etc/apt/sources.list.d/gearman-developers-ppa-karmic.list
There are no errors in /etc/apt/sources.list.d/karmic-multiverse.list

Great success!

Automation

Both modes usually exit with zero (0), which makes it easy to include them for bootstrap processes, general trouble-shooting or periodic cronjobs etc..

Reason to not exit with 0:

  • attempt to run apt-repair-sources on another distro than Ubuntu
  • old-releases.ubuntu.com is down
  • you run with -d and -f (which of course makes no sense :-))
  • trollop (a rubygem i use for CLI option parsing is not found)

Setup

Gems!

# sudo gem install apt-repair-sources

Manually

  • install Ruby Enterprise Edition (steal Karmic here; this should be your default anyway)
  • sudo gem install trollop (don't use what is in apt)
  • clone my repo: git clone git://github.com/lagged/apt-repair-sources.git
  • cd ./apt-repair/sources/bin && ./apt-repair-sources

Todo

  • create a gem
  • add support for Debian
  • improve my Ruby

Fin

Sure hope it's useful for someone else out there.

The code is on github, and I take pull-requests: https://github.com/lagged/apt-repair-sources

PHAR and FreeBSD

Sunday, November 6. 2011
Comments

I noticed that archivers/pecl-phar vanished from the ports tree on one of my FreeBSD servers.

Problem?

Reasons to remove the port were:

  • the port is unmaintained
  • the port was based on the outdated phar extension from pecl
  • phar (in pecl) contains open security issues

The simple solution is to create a new port which of course will use the phar which is bundled in PHP's core. And I will get to that (but feel free to beat me at it ;-)).

Solution

In the meantime, here's a simple solution to get phar on FreeBSD.

At first, make sure you're running PHP 5.3.8 (installed from lang/php5). Then fetch the PHP source and compile the phar extension only.

foo# fetch http://de2.php.net/get/php-5.3.8.tar.gz/from/de.php.net/mirror
foo# tar zxvf php-5.3.8.tar.gz
...
foo# cd php-5.3.8/
foo# ./configure --disable-all --enable-phar=shared
...
foo# make
...

If all goes well the modules/ directory contains a phar.so file (among other phar-things).

The remaining steps are to copy phar.so to your extension_dir and load it in extension.ini:

foo# cp modules/phar.so /usr/local/lib/php/20090626/

Feel free to double-check the location of directory:

foo# php -r 'echo ini_get("extension_dir");'

Finally, make sure PHP loads far by adding the following to /usr/local/etc/php/extensions.ini:

extension=phar.so

Fin

That's all. Once you restart php, or your webserver, phar should show up in phpinfo().

And as soon as a new php5-phar port is added (probably in archivers), you should make sure to delete phar.so and also remove the line added to the .ini.