Too much abstraction: Doctrine and PHP

Friday, July 6. 2012
Comments

Personally — I feel like I'm really, really late to get on the Doctrine train.

I've tried Doctrine 1 a few years back and found it mostly too complicated for my needs. For all I know, I still find PEAR::DB and PEAR::MDB2 the perfect database abstraction layers (DBAL). Though of course they may or may not be totally up to date in terms of PHP etc.. The overall concept of an ORM never stuck with me.

Of course Doctrine has a DBAL too and it would be unfair to compare its ORM to another DBAL, but it seems like almost no one uses the Doctrine DBAL by itself. I haven't either.

The primary use-case for Doctrine seems to be ORM and here is what I think about it.

Little steps

RTFM is especially hard with Doctrine: Google usually points to older documentation, then there are a lot of out-dated links and the classic has not been written yet.

In case there is documentation I compare the experience to RPM (Redhat Package Manager):

  • I need to learn about X
  • But have to read about Y first.
  • To understand Y I need to dive into Z.

Sounds familiar? Dependency resolution in documentation. However, of course there's always also light at the end of the tunnel — it's when it all works — I conquered a problem and defeated the proxies and caches. ;-)

Entities

Entities can be incredibly simple — take the persistent object:

<?php
use Doctrine\Common\Persistence\PersistentObject;

/**
 * @Entity
 * @Table(
 *   name="db.users"
 * )
 */
class Row extends PersistentObject
{
  /**
   * @Id
   * @Column(type="integer")
   * @GeneratedValue
   */ 
  protected $id
}

So what does this look like:

  • database "db"
  • table: "users"
  • columns: "id" (most likely integer and auto_increment with MySQL)

I feel like that is feature a hidden gem because I could not find much about it in official documentation — but that's basically all you have to do.

Annotations are not my thing and XML or YAML or neither — I can live with either of the three though. What seemed annoying is that usually people start off by writing half a dozen getFoo() and setFoo() methods which of course depends on how many columns in your table are. With the persistent object, this is not necessary.

Abstraction

Abstraction is Doctrine's biggest strength and largest drawback in my opinion. Abstraction is nice within reason, but I think with Doctrine it's way over the top.

My personal opinion on SQL is that SELECT * FROM table WHERE id = 1 never hurt anyone. Not to suggest to do this in the wrong layer of your application, but actually understanding the tools I work with, made me a better developer today. A JOIN does not hurt either because then you and I know how to write one and what it costs to execute the sucker. LEFT JOIN, STRAIGHT JOIN and INNER JOIN — most developers don't know the difference. Doctrine also likes cascades but I bet not many people ever watched a busy database server while it's trying to process all them while a couple 1000 writes happen at the same time.

With Doctrine, SQL is so far away that developers really don't need to have an idea about what is going on and I think that is bad in particular.

My problem is that most of the time, the developer does not see their SQL at all and also doesn't analyze what it is doing. I have yet to figure out how to make the application log all statements produced. And yeah, it seems to be impossible because with the use of prepared statements no SQL is actually produced in Doctrine — it all happens on the server. But sure, prepared statements are pretty awesome because we are all idiots and don't know how to escape data. But I should leave this for another blog post.

About abstraction — we had a similar discussion at this month's PHP Usergroup in Berlin and usually someone will counter my argument that the benefits of faster development outweigh all the potential screw ups. And maybe they are right — for a while. And then they upgrade their database server to have more RAM, right? What if for a while is already past tense?

Example

I just found the following code in one of our projects and I think it's a good example of what I mean.

The basics (broken down):

  • SQL:
    • a user table
    • columns: id, email, password, site_id
    • indices: PRIMARY (on id), UNIQUE (on email and site_id)
  • an entity Doctrine\Model\User class describing the above
  • a repository Doctrine\Model\UserRepository class to add convenience calls to the entity

So here is what I found:

<?php
// in class UserRepository
public function getUserById($id, $site_id = 1)
{
    $qb = $this->_em->createQueryBuilder();
    $qb->select(array('u'))
        ->from('Doctrine\Model\User', 'u')
        ->where('u.id = :id')
        ->andWhere('u.site_id = :site_id')
        ->setParameter('id', $id)
        ->setParameter('site_id', $site_id);

        /* ... */
}

And to the defense of all the people who worked on this (including me), it takes a while to see. :-)

Question to my reader: Take the setup into account, do you see the issue? Leave a comment if you figure out what the problem is.

So despite having an entity which defines the table setup, it's still possible to do something like that and there is no error/warning what so ever to let the developer know what they are doing. Not at run-time and not in the Doctrine CLI either. I'd argue that with the level of abstraction Doctrine adds to hide SQL from the developer, I expect that it would do enough analysis on what the developer is doing.

Or maybe the level of abstraction added here is actually counter-productive?

Fin

Food for thoughts. Hope it was not too ranty.

Disclaimer: This is with no offense meant to the people contributing to Doctrine or using Doctrine. I just had to raise my concerns somewhere.

Vagrant: ShellProvisioner vs. Chef

Wednesday, June 20. 2012
Comments

In my last blog entry, I demo'd how to get started with Vagrant and the ShellProvisioner.

To further illustrate how amazingly simple it is to get started on some Ruby, I'll convert the shell script from my last blog post to a little recipe for chef. Same objective, we install a PEAR package — but it could be anything really.

Follow me.

Shell

This is the shell script from before:

#!/bin/sh

apt-get update

apt-get install -y php5 php5-cli php-pear
hash -r

pear upgrade-all
pear install -f HTTP_Request2

Ruby

Create a cookbooks directory and create structure for your first cookbook in it:

$ mkdir -p my-cookbooks/first/recipes/

Create a default.rb file with the following content:

# my-cookbooks/first/recipes/default.rb
execute "apt-get update"

packages = ["php5", "php5-cli", "php-pear"]

packages.each|p| do
  package p
end

execute "pear upgrade-all"
execute "pear install -f HTTP_Request2"

The recipe is later referred to as first or first::default (name of the recipe directory, name of the .rb).

It's so simple it hurts. ;)

Step by step

  1. I run apt-get update using Chef's execute resource.
  2. I create an array of the packages (Arrays are ordered in Ruby, hashes are not (up until Ruby 1.9.x). Order is important here.)
  3. I run pear upgrade-all using the execute resource.
  4. I run pear install using the execute resource.

Vagrantfile

The Vagrantfile looks slightly different when you provision with chef-solo:

Vagrant::Config.run do |config|

  config.vm.define :web do |web_config|
    web_config.vm.box       = "lucid64"
    web_config.vm.host_name = "web"

    web_config.vm.provision :chef_solo do |chef|
      chef.cookbooks_path = "PATH-TO-YOUR-COOKBOOKS"
      chef.add_recipe "first"
      chef.log_level = :debug
    end
  end

end

The important bit: the path to the location of your cookbooks — could be an /absolute/path or ./../relative/path.

Because a Vagrantfile is essentially Ruby code, anything goes here.

Further reading

Getting started with Chef and Ruby can be intimidating or even frustrating at times. Google "chef cookbooks" and you know what I mean.

These links are what you need:

Fin

Save, and enjoy — vagrant up.

Vagrant sans Ruby

Tuesday, June 5. 2012
Comments

Development, testing, staging and production — this is how most people devide up different environments for application development.

Maintenance and setup of these environments is often not a trivial goal to achieve. Having worked with a couple different code bases and setups over the last decade, I often noticed things like environment specific hacks (if ($env == 'testing') { ... }) in application code and service configurations and a lot of manual labour all around. It's still very common that code deployed to staging does not work — but it worked for you right?

And it's also too common that members of a team do not know how something works because whoever set it up is currently out sick or on vacation.

My opinion is that the best setup currently available is something like: chef(-solo) on the server and Vagrant on the desktop. Something like because aside from chef, there is also puppet, cfengine and a couple others. Leaving specific projects aside, it really just boils down to automation (within reason).

Automation in my opinion is not just the easiest but the only viable way to setup development, testing, staging and production environments. Without some automation in place it's required that all team members know how all of it works when maybe that is not yet important and the end goal is that all environments actually resemble each other.

Enter Vagrant

Vagrant is a very nifty toolkit to bootstrap Virtualbox images. Bootstrapping means installing your application stack into one or multiple virtual machines in order to resemble production a lot better.

So a lot of times when I rave about how useful Vagrant and chef are, I get the crazy eye from PHP developers:

You want me to learn Ruby to setup my local development environment?

My response:

  1. Do not fear the Ruby.
  2. You don't have to. (Well, not a whole lot!)

Real talk

First off, of course you need Virtualbox — get a download from their website or use your package manager.

Then, it's not possible to avoid Ruby a 100% — after all Vagrant is written in Ruby.

When you're on a Linux or a Mac, this is usually enough (you may need sudo unless you RVM):

$ sudo gem install vagrant

For a sudo-less install, use RVM:

$ sudo gem install rvm
...
$ rvm install
...
$ rvm use 1.8.7
...

When set up, this is what you do:

$ gem install vagrant
...

RVM allows me to run multiple versions of ruby side by side and also leverage local gem installs — per Ruby version. Think of it as a nifty way to run multiple PHP versions and have a PEAR install per version — similar projects exist for PHP today:

To learn more about RVM, visit their website.

Once vagrant is installed, we can continue!

ShellProvisioner for fun and profit

So, in case you are more comfortable writing some shell script for the time being — Vagrant got you covered!

First off, create a new Vagrantfile for your project:

$ mkdir -p Documents/workspaces/blog-example/
$ cd Documents/workspaces/blog-example
$ vagrant init lucid64
A `Vagrantfile` has been placed in this directory. You are now
ready to `vagrant up` your first virtual environment! Please read
the comments in the Vagrantfile as well as documentation on
`vagrantup.com` for more information on using Vagrant.

lucid64 (Ubuntu Lucid 10.04, 64bit) is the name of one of my local box files for Vagrant. In case you haven't got a box yet, head over to vagrantbox.es. There are a couple images you (aka box files) you can download. To get the lucid64 box, use the following command:

$ vagrant box add lucid64 http://files.vagrantup.com/lucid64.box

Once you made it past vagrant init, you should have a Vagrantfile with a lot of stuff in there.

While it's not important for my tutorial, I recommend you review it some (other time). The created Vagrantfile contains examples for all the provisioners (puppet, chef and shell) and a couple other configuration options, etc..

Let's skip over this and get to the ShellProvisioner.

This is all you need

First off, let's create a shell script setup.sh in the same directory and put something like this in it:

#!/bin/sh

apt-get update

apt-get install -y php5 php5-cli php-pear
hash -r

pear upgrade-all
pear install -f HTTP_Request2

What does it do?

  1. Update local package sources.
  2. Install php5, cli interpreter and PEAR installer
  3. reload environment
  4. upgrade all installed PEAR packages
  5. install PEAR HTTP_Request2

When you're done editing, make sure to chmod +x it.

Simple as that.

Putting your shell script to use

Replace your Vagrantfile with the following:

Vagrant::Config.run do |config|

  config.vm.box = "lucid64"
  config.vm.provision :shell, :path => "./setup.sh"

end

Then, run vagrant up and watch it provision. :)

Enter your virtual machine with vagrant ssh and verify HTTP_Request2 is installed:

$ vagrant ssh                                                                                                                                                               [14:16:08]
Linux lucid64 2.6.32-33-server #70-Ubuntu SMP Thu Jul 7 22:28:30 UTC 2011 x86_64 GNU/Linux
Ubuntu 10.04.3 LTS

Welcome to the Ubuntu Server!
 * Documentation:  http://www.ubuntu.com/server/doc
Last login: Thu Jul 21 14:08:15 2011 from 10.0.2.2
[email protected]:~$ pear list -c pear
Installed packages, channel pear.php.net:
=========================================
Package          Version State
Archive_Tar      1.3.10  stable
Console_Getopt   1.3.1   stable
HTTP_Request2    2.1.1   stable
Net_URL2         2.0.0   stable
PEAR             1.9.4   stable
Structures_Graph 1.0.4   stable
XML_Util         1.2.1   stable

Yay — your first steps into automation!

Fin

That's all for today — Vagrant with almost no Ruby. I hope this gets many more started!

Composer and chef updates

Sunday, June 3. 2012
Comments

Here are some updates on my chef resource for composer:

It started off with a PR where I was asked to include code that would download (and install) a composer.phar if none existed. Not a bad idea in general, but also not something I want to impose on anyone, so I decided to create a new action (or in chef-lingo goal) instead.

setup

Up until now, my php_composer resource supported :install which runs php composer.phar install when a composer.phar is in the designated directory:

php_composer "/var/www/app" do
  action :install
end

Since an hour or so ago, it also supports :setup. And :setup will download composer for you if requested:

php_composer "/var/www/app" do
  action [:setup, :install]
end

Note: I'm not sure if this is a good idea for a couple of reasons — the most obvious one: getcomposer.org could be unavailable when you run your deployment. And then there may or may not be changes in the latest composer.phar which require you to make adjustments to your local composer.json. I prefer my deployments rock-solid to minimize side-effects! :)

Getting started

My resource will use composer but it's generally a chef-no-no to also install the tools required by the resource.

A small list of items you will need:

  • php 5.3+
  • the Phar extension
  • git-core (and/or subversion)
  • unzip (or PHP's zip extension)

For an idea how to set these up with chef, see my vagrant-test::default recipe.

Roadmap

I haven't really decided on a roadmap, but here are a few things I want to implement in the coming weeks:

  • If you're feeling frisky: support for php composer.phar self-update.
  • Support for php composer.phar update!
  • An attribute to provide the location of a phar (e.g. local or network file-system, CDN, etc.).
  • Support for --install-dir.

I'm open to contributions on any of these items, my requirements are:

  • Code must be robust.
  • Code must be robust.
  • Your additions should work on Ubuntu and Linux in general. :)

Fin

That's all. Check it out on Github!

Zend Framework: CRUD

Monday, May 21. 2012
Comments

Update, 2012-05-24: Fixed some typos and added the screenshot (eyecandy!).

I think it took me (or us) a couple attempts to get this right — let me introduce you to Zf_Crud, a CRUD controller for the Zend Framework.

What's CRUD?

CRUD is an acronym and stands for:

  • Create
  • Read
  • Update
  • Delete

A general purpose for CRUD are administrative interfaces — view records, create them, update them or delete them. Think of phpMyAdmin as a very general purpose CRUD interface. One you need some SQL-fu for.

In my experience, such interfaces are most likely or often the very last item on a project. Not too many people (myself included) like to build these interfaces primarily because we have build them before. They are necessary non-the-less: not everybody on a project is a developer and feels comfortable writing SQL queries into phpMyAdmin to get data out (or in).

Zf_Crud

Zf_Crud aims to provide you with an interface for any table possible — think of it as a phpMyAdmin more tailored towards your data and (thanks to Twitter Bootstrap and the Easybib_Form_Decorator) prettier!

Example

Note: We haven't setup a PEAR package yet. I'll get to it soon, but feel free to submit a PR with a package.xml.

Install

The only technical dependencies are PHP 5.3+ and the Easybib_Form_Decorator:

$ pear channel-discover easybib.github.com/pear
...
$ pear install easybib/Easybib_Form_Decorator
...

The other dependency is a working Zend Framework (1) application and an idea what you're doing.

Setup

To setup Zf_Crud, clone (or export) it into your local vendor library:

Our Zend Framework projects have the following structure:

  • app/modules/<module>/library/vendor

To export with git, just do the following:

$ cd app/modules/foo/library/vendor
$ git clone git://github.com/lagged/Zf_Crud.git ./Zf_Crud
...
$ cd Zf_Crud && rm -rf .git

Then check it into your own VCS.

If you don't have git (and cannot install it), get a download of Zf_Crud from Github.

Composer, you ask?

With the initial public release, we also added composer:

{
  "require": {
    "php": ">=5.3.0",
    "lagged/Zf_Crud": "0.5.1"
  }
}

Code

Working with Zf_Crud should be super-simple and easy!

In your module (e.g. foo), create a controller in app/modules/foo/controllers/AdminController.php.

<?php
use Lagged\Zf\Crud\Autoload as CrudAutoload;
use Lagged\Zf\Crud\Controller as CrudController;

require_once dirname(__DIR__) . '/library/vendor/Zf_Crud/library/Autoload.php';
CrudAutoload::register();

class Foo_AdminController extends CrudController
{
    protected $model = 'Foo_Model_SomeTable';
    protected $title = 'My Interface';
}

Next, create a plain model using Zend_Db_Table_Abstract:

<?php
class Foo_Model_SomeTable extends Zend_Db_Table_Abstract
{
    protected $_name = 'some_table';
}

The (naming convention in ZF1 is not PSR-0 and a little weird. In case it's not obvious: the) model should live in: app/modules/foo/models/SomeTable.php.

Last but not least: Zend_Db_Table means RDBMS. Zf_Crud expects a Zend_Registry-key called dbAdapter to work. If your's is called differently, skip to the "Convention over Configuration" section.

Anyway — once these two files created (and assuming the rest is setup correctly), you should see something like the following:

screenshot-zf_crud

Convention over configuration

I'm a fan of convention over configuration and it's the approach we selected when we build Zf_Crud.

The idea is that it should work out of the box without setting up a huge application.ini or DIC, in case you want to tinker with it when you're up and running, here is how.

Since you're extending from the Lagged\Zf\Crud\Controller, this controller has a few configuration options. Configuration is probably too advanced since these are essentially a bunch of class-properties you can overwrite in your class or via init():

<?php
// setup here
class Foo_AdminController extends CrudController
{
    protected $model = 'Foo_Model_SomeTable';
    protected $dbAdapter = 'db';
}

Some of the gems are:

  • $where: a WHERE-clause for the data query
  • $order: column to order by
  • $hidden: hide these columns from display
  • $count: number of items per page

Fin

We've been using this code for a couple months now for various items. We recently tagged an early 0.5.1 which suggests that this code is still a WIP and a moving target. The configuration bits are not too great elegant yet. So there are a lot of rough edges to be aware of.

The bottom line is that Zf_Crud has been good for us since it allows us to take the pain out of building administrative interfaces. In most cases it's setup in an hour tops and then we can move on to build something more interesting than a couple forms and views to display and edit data.

If you have anything to add — comments and pull-requests welcome!