Contributing to PEAR: Taking over packages

Friday, February 18. 2011
Comments

One of the more frequent questions I see on the mailing lists and IRC is, "How do I take over a package?".

Very often people start to use a PEAR package and then at some point encounter either a bug or they miss a certain feature. The package's state however is inactive or flat unmaintained.

Offer help

The first step is to offer your help. And the easiest way to help is to report bugs or open feature requests.

A great bug report or feature request includes:

  1. Accurate problem description.
  2. A proposed solution.
  3. A patch (unified diff, aka diff -up or svn diff)
  4. A PHPUnit test case so this bug stays fixed.

I realize that these are a lot of requirements, but I also said a great bug report and no one said it was easy.

Getting fixes in

Often people persued these first steps and went the whole nine yards. But the bugs are still not fixed, the feature isn't implemented either and no release was pushed out.

How do you move forward now?

  1. Email the maintainer and cc [email protected].
  2. Ping PEAR QA again after reasonable period of time passed (e.g. 1 or 2 weeks tops)

If the package is generally unmaintained or the maintainer did not respond, PEAR QA will be happy to apply patches for you and do a new release. If the maintainer replied, work with them to get the issues fixed and a new release shipped.

Do you want to maintain?

To become the maintainer of any package we need to see your contributions first.

Contributions include:

  • patches for bugs and features
  • tests
  • bug reports
  • (maybe) helping people on the IRC channel

The fastest way to take over an orphaned package is to provide patches to any open issues so the QA team can push out a new release.

Contributing code is not just the preferred way but a hard requirement. It's the only viable way for others to assess your skills.

Another way to maintainership is of course when the current maintainer asked you if you want to contribute more since you showed them an interest through your work.

Find a package

Here's a list of currently unmaintained PEAR packages: http://pear.php.net/qa/packages_orphan.php.

Fin

I hope I summarized the process a little to make it more transparent. In the end it's pretty simple and straight forward to contribute to PEAR — and lots of people will be tankful for your contributions too.

PHP SDK for Amazon Web Services

Wednesday, September 29. 2010
Comments

Yesterday, Jeff Barr announced Amazon's own PHP SDK for their web services — own, because AWS hired CloudFusion's lead developer earlier this year (in March) and I guess after a while they decided it was time to incorporate his open source efforts into the company. The full story is on getcloudfusion.com.

So what?

What's more than just pretty interesting about all of this, is that not only is the AWS PHP SDK hosted on Github (bonus points for sure), but since it implements almost the entire API of all infrastructural services and is backed by the API provider, this library currently presents the most feasible way for PHP developers to work with AWS. And to add to that, the library is fully documented as well.

Having worked myself on various small wrappers for the EC2 and SNS web services, I'm really somewhat glad that I can stop working on them now and continue implementing the web services.

PEAR

Amazon's move is also another victory for PEAR (and of course Pirum) because it brings more acceptance to the distribution of PHP libraries using a PEAR Channel.

The SDK's channel is the following: http://pear.amazonwebservices.com/

Setup

pear channel-discover pear.amazonwebservices.com
pear install aws/sdk

The flipside

There are absolutely no unit tests included anywhere. But since I'm assuming that they exist indeed, I hope they will be open sourced before not too long. Or in case they don't (What's up with that?), that pull requests will be accepted so the community will be able to contribute some.

Debugging Zend_Test

Monday, September 20. 2010
Comments

Sometimes, I have to debug unit tests and usually this is a situation I'm trying to avoid.

If I have to spend too much time debugging a test it's usually a bad test. Which usually means that it's too complex. However, with Zend_Test_PHPUnit_ControllerTestCase, it's often not the actual test, but the framework. This is not just tedious for myself, it's also not the most supportive fact when I ask my developers to write tests.

An example

The unit test fails with something like:

Failed asserting last module used <"error"> was "default".

Translated, this means the following:

  • The obvious: an error occurred.
  • The error was caught by our ErrorController.
  • Things I need to find out:
    • What error actually occurred?
    • Why did it occur?
    • Where did the error occur?

The last three questions are especially tricky and drive me nuts on a regular basis because a unit test should never withhold these things from you. After all, we use these tests to catch bugs to begin with. Why make it harder for the developer fix them?

In my example an error occurred, but debugging Zend_Test also kicks in when things supposedly go according to plan. Follow me to the real life example.

Real life example

I have an Api_IndexController where requests to my API are validated in its preDispatch().

Whenever a request is not validated, I will issue "HTTP/1.1 401 Unauthorized". For the sake of this example, this is exactly what happens.

class ApiController extends Zend_Controller_Action
{
    protected $authorized = false;

    public function preDispatch()
    {
        // authorize the request
        // ...
    }
    public function profileAction()
    {
        if ($this->authorized === false) {
            $this->getResponse()->setRawHeader('HTTP/1.1 401 Unauthorized');
        }
        // ...
    }
}

Here's the relevant test case:

class Api_IndexControllerTest ...

    public function testUnAuthorizedHeader()
    {
        $this->dispatch('/api/profile'); // unauthorized
        $this->assertResponseCode(401);
    }
}

The result:

1) Api_IndexControllerTest::testUnAuthorizedHeader
Failed asserting response code "401"

/project/library/Zend/Test/PHPUnit/Constraint/ResponseHeader.php:230
/project/library/Zend/Test/PHPUnit/ControllerTestCase.php:773
/project/tests/controllers/api/IndexControllerTest.php:58

Not very useful, eh?

Debugging

Before you step through your application with print, echo and an occasional var_dump, here's a much better way of see what went wrong.

I'm using a custom Listener for PHPUnit, which works sort of like an observer. This allows me to see where I made a mistake without hacking around in Zend_Test.

Here is how it works

Discover my PEAR channel:

sudo pear channel-discover till.pearfarm.org

Install:

[email protected]:~/ sudo pear install till.pearfarm.org/Lagged_Test_PHPUnit_ControllerTestCase_Listener-alpha
downloading Lagged_Test_PHPUnit_ControllerTestCase_Listener-0.1.0.tgz ...
Starting to download Lagged_Test_PHPUnit_ControllerTestCase_Listener-0.1.0.tgz (2,493 bytes)
....done: 2,493 bytes
install ok: channel://till.pearfarm.org/Lagged_Test_PHPUnit_ControllerTestCase_Listener-0.1.0

If you happen to not like PEAR (What's wrong with you? ;-)), the code is also on github.

Usage

This is my phpunit.xml:

<?xml version="1.0" encoding="utf-8"?>
<phpunit bootstrap="./TestInit.php" colors="true" syntaxCheck="true">
    <testsuites>
    ...
    </testsuites>
    <listeners>
        <listener class="Lagged_Test_PHPUnit_ControllerTestCase_Listener" file="Lagged/Test/PHPUnit/ControllerTestCase/Listener.php" />
    </listeners>
</phpunit>

Output

Whenever I run my test suite and a test fails, it will add something like this to the output of PHPUnit:

PHPUnit 3.4.15 by Sebastian Bergmann.

..Test 'testUnAuthorizedHeader' failed.
RESPONSE

Status Code: 200

Headers:

     Cache-Control - public, max-age=120 (replace: 1)
     Content-Type - application/json (replace: 1)
     X-Ohai - WADDAP (replace: false)

Body:

{"status":"error","msg":"Not authorized"}

F..

Time: 5 seconds, Memory: 20.50Mb

There was 1 failure:

1) Api_IndexControllerTest::testUnAuthorizedHeader
Failed asserting response code "401"

/project/library/Zend/Test/PHPUnit/Constraint/ResponseHeader.php:230
/project/library/Zend/Test/PHPUnit/ControllerTestCase.php:773
/project/tests/controllers/api/IndexControllerTest.php:58

FAILURES!
Tests: 5, Assertions: 12, Failures: 1.

Analyze

Analyzing the output, I realize that my status code was never set. Even though I used a setRawHeader() call to set it. Turns out setRawHeader() is not parsed so the status code in Zend_Controller_Response_Abstract is not updated.

IMHO this is also a bug and a limitation of the framework or Zend_Test.

The quickfix is to do the following in my action:

$this->getResponse()->setHttpResponseCode(401);

Fin

That's all. Quick, but not so dirty. If you noticed, I got away without hacking Zend_Test or PHPUnit.

The listener pattern provides us with very powerful methods to hook into our test suite. If you see the source code it also contains methods for skipped tests, errors, test suite start and end.

start-stop-daemon, Gearman and a little PHP

Thursday, April 22. 2010
Comments

The scope of this blog entry is to give you a quick and dirty demo for start-stop-daemon together with a short use case on Gearman (all on Ubuntu). In this example, I'm using the start-stop-daemon to handle my Gearman workers through an init.d script.

Gearman

Gearman is a queue! But unlike for example most of the backends to Zend_Queue, Gearman provides a little more than just a message queue to send — well — messages from sender to receiver. With Gearman it's trivial to register functions (tasks) on the server to make in order to start a job and to get stuff done.

For me the biggest advantages of Gearman are that it's easy to scale (add a server, start more workers) and that you can get work done in another language without building an API of some sort in between. Gearman is that API.

Back to start-stop-daemon

start-stop-daemon is a facility to start and stop programs on system start and shutdown. On recent Ubuntus most of the scripts located in /etc/init.d/ make use of it already. It provides a simple high-level API to system calls — such as stopping a process, starting it in a background, running it under a user and the glue, such as writing a pid file.

My gearman start script

Once adjusted, register it with the rc-system: update-rc.d script defaults. This will take care of the script being run during the boot process and before shutdown is completed.

A little more detail

The script may be called with /etc/init.d/script start|stop|restart (the pipes designated "or").

Upon start, we write a pidfile to /var/run and start the process. The same pidfile is used on stop — simple as that. The rest of it is hidden behind start-stop-daemon which takes care of the ugly rest for us.


Continue reading "start-stop-daemon, Gearman and a little PHP"

DB_CouchDB_Replicator

Wednesday, March 3. 2010
Comments

Update, 2010-03-04: I just rolled a 0.0.2 release. In case you had 0.0.1 installed, just use pear upgrade-all to get it automatically. This release is trying to fix a random hang while reading documents from the source server.

I also opened a repository on Github.

---

As some may have guessed from a previous blog post we are currently running a test setup with CouchDB lounge. My current objective is to migrate our 200 million documents to it, and this is where I am essentially stuck this week.

No replication, no bulk docs

The lounge currently does not support replication (to it) or saving documents via bulk requests, so in essence migrating a lot of data into it is slow and tedious.

I have yet to figure out if there is a faster way (Maybe parallelization?), but DB_CouchDB_Replicator is the result of my current efforts.

I think I gave up on parallelization for now because it looked like hammering the lounge with a single worker was already enough, but generally I didn't have time to experiment much with it. It could have been my network connection too. Feedback in this area is very, very appreciated.

DB_CouchDB_Replicator

DB_CouchDB_Replicator is a small PHP script which takes two arguments, --source and --target. Both accept values in style of http://username:[email protected]:port/db and attempt to move all documents from source to target.

Since long running operations on the Internet are bound to fail, I also added a --resume switch, and while it's running it outputs a progress bar, so it should be fairly easy to resume. And you also get an idea of where it's currently at and how much more time it will eat up.

These switches may change, and I may add more — so keep an eye on --help. Also, keep in mind, that this is very alpha and I give no guarantees.

Installation

Installation is simple! :-)

apt-get install php-pear
pear config-set preferred_state alpha
pear channel-discover till.pearfarm.org
pear install till.pearfarm.org/DB_CouchDB_Replicator

Once installed, the replicator resides in /usr/local/bin or /usr/bin and is called couchdb-replicator.

Fin

The code is not yet on github, but will eventually end up there. All feedback is welcome!