What's wrong with composer and your .travis.yml?

Sunday, August 24. 2014
Comments

I'm a huge advocate of CI and one service in particular called Travis-Ci.

Travis-CI runs a continuous integration platform for both open source and commercial products. In a nutshell: Travis-CI listens for a commit to a Github repository and runs your test suite. Simple as that, no Jenkins required.

At Imagine Easy we happily take advantage of both. :)

So what's wrong?

For some reason, every other open source project (and probably a lot of closed source projects), use Travis-CI wrong in a way, that it will eventually break your builds.

When exactly? Whenever Travis-CI promotes composer to be a first-class citizen on the platform and attempts to run composer install automatically for you.

There may be breakage, but there may also be slowdown because by then you may end up with not one, but two composer install runs before your tests actually run.

Here's what needs fixing

A lot of projects use composer like so:

language: php
before_script: composer install --dev
script: phpunit

Here's what you have to adjust

language: php
install: composer install --dev
script: phpunit

install vs. before_script

I had never seen the install target either. Not consciously at least. And since I don't do a lot of CI for Ruby projects, I wasn't exposed to it either. On a Ruby build, Travis-CI will automatically run bundler for you, using said install target.

order of execution

In a nutshell, here are the relevant targets and how the execute:

  1. before_install
  2. install
  3. before_script
  4. script

The future

The future is that Travis-CI will do the following:

  1. before_install will self-update the composer(.phar)
  2. install will run composer install
  3. There is also the rumour of a composer_opts (or similar) setting so you can provide something like --prefer-source to the install target, without having to add an install target

Fin

Is any of this your fault? I don't think so, since the documentation leaves a lot to be desired. Scanning it while writing this blog post, I can't find a mention of install target on the pages related to building PHP products.

Long story short: go update your configurations now! ;)

I've started with doctrine/cache and doctrine/dbal, and will make it a habit to send a PR each time I see a configuration which is not what it should be.

Continuous Integration: Automated database setup with Doctrine on Travis-CI

Monday, August 13. 2012
Comments

Testing is important — most people understand that by now. A lot of people write tests for their open source code already, but in-house testing is still hard. For example, many of us had an encounter with Jenkins: it runs well to a point where it becomes harder to maintain the Jenkins than it is to write tests.

Another obstacle is test setup and environments: When I write and run tests, there is sometimes only so much I can do to mock and avoid actual calls to my storage backend. While I prefer to run my database tests against a SQLite in memory database, there are these edge cases, where I work with multiple database or I write a direct query (and by-pass the ORM-magic).

In these cases I need to have that database server available in my test environment!

The following blog posts explains how to solve these things with Travis-CI. I will walk you through the setup on Travis-CI's business service. But most of this applies to their open source offering as well.

Step by step

I'll try to break it up into small steps.

Travis-CI

The first step is to login at http://travis-ci.com and add the repository tests should be run for. To be able to add repositories of an organization, you have to be the owner of the organization. The easiest way to get access to the service right now is donating to these guys or in case you have done that already: email them. ;-)

The second step is setting up a .travis.yml file.

Mine looks like this:

language: php
php:
  - 5.3
  - 5.4
before_script:
  - ./composer.phar -v install --dev
  - psql -c 'create database testdatabase;' -U postgres

Run-down:

  • run the tests against PHP 5.3 and 5.4
  • before_script defines your test setup (outside PHP)
  • I omitted the script stanza because the default (phpunit) works for me

Composer

I am using composer to manage my dependencies and you should too. I don't want to go into details here, but a short example of my composer.json is the following:

{
    "name": "mycompany/project",
    "description": "A super cool project.",
    "require": {
        "doctrine/orm": "2.2.2"
    },
    "autoload": {
        "psr-0": {
            "MyCompany\\Project\\Test": "tests/",
            "MyCompany\\Project": "src/"
        }
    }
}

Side-note: We also currently commit a composer.phar into each repository for two reasons:

  1. To ensure a change in composer won't break our setup.
  2. Downtime of (or connectivity issues to) their website don't break our deployments and test runs.

Test framework setup

There is not a whole lot to setup since Travis-CI installs phpunit already. Just make sure you have a phpunit.xml in the root of your repository and you are good to go.

Database schema

The next step would be to generate your schema and check in some .sql, right? I'm not a huge fan of this, because I hate running through a lot of manual steps when I need to update something. Manual steps means that they might be forgotten or people make a mistake. So the objective is to avoid any manual labour as much as you can.

Instead of maintaining these files, I use Doctrine's SchemaTool. It takes care of this just fine because I annotated all my entities already.

To make use of this, I suggest to add the following to your test case:

<?php 
namespace MyCompany\Project\Test;

use Doctrine\ORM\Tools\Setup;
use Doctrine\ORM\EntityManager;
use Doctrine\Common\Persistence\PersistentObject;
use Doctrine\ORM\Tools\SchemaTool;

class MyTestCase extends \PHPUnit_Framework_Testcase
{
    protected $em, $tool;

    public function setUp()
    {
        $this->setUpDatabase(); // wrap this again
        $this->setUpSchema();
        /* more setup here */
    }

    public function tearDown()
    {
        $classes = array(
            $this->em->getClassMetadata('MyCompany\Project\Entity\SomethingImportant'),
        );
        $this->tool->dropSchema($classes);
        unset($tool);
        unset($em);
    }

    public function setUpDatabase()
    {
        $isDevMode      = true;
        $doctrineConfig = Setup::createAnnotationMetadataConfiguration(
            array('path/to/Entity'),
            $isDevMode
        );

        // database configuration parameters
        $dbConfig = array(
            'host'     => '127.0.0.1',
            'user'     => 'postgres',
            'password' => '',
            'dbname'   => 'testdatabase',
        );

        $this->em = EntityManager::create($dbConfig, $doctrineConfig);
        PersistentObject::setObjectManager($this->em);
    }

    public function setUpSchema()
    {
        $this->tool = new SchemaTool($this->em);
        $classes = array(
            $this->em->getClassMetadata('MyCompany\Project\Entity\SomethingImportant'),
        );
        $this->tool->createSchema($classes);
    }
}

Assumptions made (aka, room for improvement):

  • your entities are in path/to/Entity
  • PostgreSQL is used and runs on 127.0.0.1
  • you're using a database called testdatabase

Tests

Once the setup is complete, I add the usual testFoo() methods into my test case class.

From within the testFoo() methods I have Doctrine's EntityManager available via $this->em. The entity manager allows me to do whatever I need or want within a test run.

After a test completes, the tearDown() method is invoked and destroys the tables in your testdatabase and then setUp() re-creates it. This will take some time but the side-effects of stale data are not to be neglected. Add to that, your tests should not rely on the order they are executed in anyway.

Another benefit of this setup are updated SQL tables each time a commit changes the annotations. No extra .sql files to maintain. :)

Fin

That's really all there is to running your test suite with Travis-CI and while you did all the above, you just added continuous integration to your toolkit because these tests run each time a pul request is opened or commits are pushed. :-)

As I mentioned early on, these steps apply to the open source offering as well — all but the link to login.

If PostgreSQL is not your game, have a look at the list of currently supported databases on Travis-CI. And if your database server is not on the list of supported applications, you might as well install it with a before_script (as long as it runs on Ubuntu). ;-)

Happy testing!

Cooking PHPUnit (and a chef-solo example on top)

Sunday, December 4. 2011
Comments

I'm sure most of you noticed that with the recent upgrade of PHPUnit to version 3.6, a lot of breakage was introduced in various projects.

And for example Zend Framework 1.x won't update to the latest version either. When I ranted on twitter someone send me Christer Edvartsen's blog post on how to setup multiple versions of PHPUnit. It's really neat since it walks you through the setup step by step and you learn about things such as --installroot on the way. --installroot in particular is something I never ever saw before and I've been using PEAR for more than a few years now. So kudos to Christer for introducing myself to it.

The only thing to add from my side would be, Why are you guys not aggregated on planet-php?.

Cooking with Chef

Another reason why I decided to write this blog entry was that I created a chef-recipe based on Christer's blog entry.

If you follow my blog for a while, you might have noticed that I'm a huge fan of automation. I just moved one of our development servers the other day and had one of these moments where something just paid off. Taking for granted that I can spin up fully operational EC2 instances in minutes, I also had our development stack installed and configured in an instant.

My recipe basically follows Christer's instructions and because I distribute phpunit's command along with it, editing of the file is no longer required: when the chef run completes, phpunit34 is installed and ready to be used.

Get started

I'm doing the following commands as root — my setup is in /root/chef-setup.

Install chef(-solo) and clone my cookbooks

shell# gem install --no-ri --no-rdoc chef
... 
shell# git clone git://github.com/till/easybib-cookbooks.git
...

Chef configuration

Then setup a node.json file which chef-solo will need to run:

{
  "run_list": [
    "recipe[phpunit]"
  ]
}

Then create a solo.rb:

file_cache_path "/var/chef-solo"
cookbook_path ["/root/chef-setup/easybib-cookbooks"]

Chef run

Finally we start chef-solo with following command:

shell# chef-solo -c /root/chef-setup/solo.rb -j /root/chef-setup/node.json -l debug
...

The command runs chef-solo (which is part of the gem we installed) and reads the basic configuration from the solo.rb-file. This file contains the location of the cookbooks (remember git clone ...) and a path to cache files. You don't need to create anything, it should be all taken care of.

The node.json-part allows us to set node-specific values. The prime example is the run-list, but it allows you to set attributes as well. Attributes contain values for variables used in recipes, but are not used in this example.

Last but not least: -l debug — a lot of useful output, but we usually run with -l warn. And if this is interesting enough for you, I suggest the other blog entries I wrote on this topic.

Did it work?

Depending on the location of your pear setup — usually /usr/bin/pear or /usr/local/bin/pear — the phpunit34 script is created in the same path:

shell# which phpunit34
/usr/local/bin/phpunit34

Yay!

Fin

This feels like hitting two birds with one stone. Though just by figure of speech! I object to violence against birds.

It might be overkill to setup chef to just install phpunit 3.4 by itself, but I think this serves as a stellar example of how you can leverage the power of chef to get more done. Writing a couple more recipes to install and configure the rest of your stack shouldn't be too hard.

If you'd like to see anything in particular: I'll take requests via email, Twitter or in the comments.

RFC: Mocking protected methods

Wednesday, June 15. 2011
Comments

Update, 2011-06-16, 12:15 AM Thanks for the comments.

(I swear I had something like that before and it didn't work!) Here's the solution:

$userId = 61382;
$docId  = 'CLD2_62e029fc-1dae-4f20-873e-69facb64a21a';
$body   = '{"error":"missing","reason":"problem?"}';

$client = new Zend_Http_Client;
$client->setAdapter(new Zend_Http_Client_Adapter_Test);

$couchdb = $this->getMock(
    'CouchDB',
    array('makeRequest',),
    array($userId, $this->config,)
);
$couchdb->expects($this->once())
    ->method('makeRequest')
    ->will($this->returnValue(new Zend_Http_Response(200, array(), $body)));
$couchdb->setHttpClient($client);
$couchdb->getDocument($docId);

--- Original blog entry ---

I wrote a couple tests for a small CouchDB access wrapper today. But when I wrote the implementation itself, I realized that my class setup depends on an actual CouchDB server being available and here my journey began.

Example code

Consider the following example:

My objective is not to be able to test any of the protected methods directly, but to be able to supply a fixture so we don't have to setup CouchDB to run our testsuite. My fixture would replace makeRequest() and return a JSON string instead.


Continue reading "RFC: Mocking protected methods"

Debugging Zend_Test

Monday, September 20. 2010
Comments

Sometimes, I have to debug unit tests and usually this is a situation I'm trying to avoid.

If I have to spend too much time debugging a test it's usually a bad test. Which usually means that it's too complex. However, with Zend_Test_PHPUnit_ControllerTestCase, it's often not the actual test, but the framework. This is not just tedious for myself, it's also not the most supportive fact when I ask my developers to write tests.

An example

The unit test fails with something like:

Failed asserting last module used <"error"> was "default".

Translated, this means the following:

  • The obvious: an error occurred.
  • The error was caught by our ErrorController.
  • Things I need to find out:
    • What error actually occurred?
    • Why did it occur?
    • Where did the error occur?

The last three questions are especially tricky and drive me nuts on a regular basis because a unit test should never withhold these things from you. After all, we use these tests to catch bugs to begin with. Why make it harder for the developer fix them?

In my example an error occurred, but debugging Zend_Test also kicks in when things supposedly go according to plan. Follow me to the real life example.

Real life example

I have an Api_IndexController where requests to my API are validated in its preDispatch().

Whenever a request is not validated, I will issue "HTTP/1.1 401 Unauthorized". For the sake of this example, this is exactly what happens.

class ApiController extends Zend_Controller_Action
{
    protected $authorized = false;

    public function preDispatch()
    {
        // authorize the request
        // ...
    }
    public function profileAction()
    {
        if ($this->authorized === false) {
            $this->getResponse()->setRawHeader('HTTP/1.1 401 Unauthorized');
        }
        // ...
    }
}

Here's the relevant test case:

class Api_IndexControllerTest ...

    public function testUnAuthorizedHeader()
    {
        $this->dispatch('/api/profile'); // unauthorized
        $this->assertResponseCode(401);
    }
}

The result:

1) Api_IndexControllerTest::testUnAuthorizedHeader
Failed asserting response code "401"

/project/library/Zend/Test/PHPUnit/Constraint/ResponseHeader.php:230
/project/library/Zend/Test/PHPUnit/ControllerTestCase.php:773
/project/tests/controllers/api/IndexControllerTest.php:58

Not very useful, eh?

Debugging

Before you step through your application with print, echo and an occasional var_dump, here's a much better way of see what went wrong.

I'm using a custom Listener for PHPUnit, which works sort of like an observer. This allows me to see where I made a mistake without hacking around in Zend_Test.

Here is how it works

Discover my PEAR channel:

sudo pear channel-discover till.pearfarm.org

Install:

[email protected]:~/ sudo pear install till.pearfarm.org/Lagged_Test_PHPUnit_ControllerTestCase_Listener-alpha
downloading Lagged_Test_PHPUnit_ControllerTestCase_Listener-0.1.0.tgz ...
Starting to download Lagged_Test_PHPUnit_ControllerTestCase_Listener-0.1.0.tgz (2,493 bytes)
....done: 2,493 bytes
install ok: channel://till.pearfarm.org/Lagged_Test_PHPUnit_ControllerTestCase_Listener-0.1.0

If you happen to not like PEAR (What's wrong with you? ;-)), the code is also on github.

Usage

This is my phpunit.xml:

<?xml version="1.0" encoding="utf-8"?>
<phpunit bootstrap="./TestInit.php" colors="true" syntaxCheck="true">
    <testsuites>
    ...
    </testsuites>
    <listeners>
        <listener class="Lagged_Test_PHPUnit_ControllerTestCase_Listener" file="Lagged/Test/PHPUnit/ControllerTestCase/Listener.php" />
    </listeners>
</phpunit>

Output

Whenever I run my test suite and a test fails, it will add something like this to the output of PHPUnit:

PHPUnit 3.4.15 by Sebastian Bergmann.

..Test 'testUnAuthorizedHeader' failed.
RESPONSE

Status Code: 200

Headers:

     Cache-Control - public, max-age=120 (replace: 1)
     Content-Type - application/json (replace: 1)
     X-Ohai - WADDAP (replace: false)

Body:

{"status":"error","msg":"Not authorized"}

F..

Time: 5 seconds, Memory: 20.50Mb

There was 1 failure:

1) Api_IndexControllerTest::testUnAuthorizedHeader
Failed asserting response code "401"

/project/library/Zend/Test/PHPUnit/Constraint/ResponseHeader.php:230
/project/library/Zend/Test/PHPUnit/ControllerTestCase.php:773
/project/tests/controllers/api/IndexControllerTest.php:58

FAILURES!
Tests: 5, Assertions: 12, Failures: 1.

Analyze

Analyzing the output, I realize that my status code was never set. Even though I used a setRawHeader() call to set it. Turns out setRawHeader() is not parsed so the status code in Zend_Controller_Response_Abstract is not updated.

IMHO this is also a bug and a limitation of the framework or Zend_Test.

The quickfix is to do the following in my action:

$this->getResponse()->setHttpResponseCode(401);

Fin

That's all. Quick, but not so dirty. If you noticed, I got away without hacking Zend_Test or PHPUnit.

The listener pattern provides us with very powerful methods to hook into our test suite. If you see the source code it also contains methods for skipped tests, errors, test suite start and end.