till's blog

Here's how I literally wasted eight hours of my life. :-)

We signed up for Yahoo! Search Boss last week. The process itself was pretty straight:

1. Sign into your Yahoo! account at https://developer.apps.yahoo.com/
2. Click "New Project", fill out the form.
3. Then click on the project name, activate "Yahoo! Search Boss" by suppling some billing info.

Consumer key rejected?

The above process doesn't even take five minutes, but then I spent eight hours figuring out what oauth_problem=consumer_key_rejected means. Spent a couple hours googling, reading bug reports and even posted to the Yahoo! group associated with Search Boss.

To cut to the chase: When you create a new project, it's not sufficient to just activate "Yahoo! Search Boss" (and provide billing details and so on).

You have to select another (pointless) API along with it. Even if you don't use it. Apparently a consumer key and secret are only put into use when you selected one of their other offerings. For some reason, Yahoo! Search Boss doesn't work by itself.

So once I selected one of their other APIs (I went for Knowledge Plus), I copied my updated consumer secret, it magically worked.

Add to all of the above the constant link screw up in Yahoo!'s developer documentation. Since there's a v1 and a v2 of the API you're bound to find the wrong one for sure — the best way to find it: Google.

Small working example for request tokens?

Sure, so to get a request token, this is what you do with Zend_Oauth:

$consumer = new Zend_Oauth_Consumer($config);
$token    = $consumer->getRequestToken();

$config is an instance of Zend_Config. In addition to the oauth.ini later in the blog post, you'll also need requestTokenUrl to retrieve a request token.

The result ($token) is of Zend_Oauth_Token_Request.

Nice to know: In the process I discovered that in case an access token was needed, I had to make sure that $token contains oauth_verifier. This verifier is only supplied when you supply a callbackUrl (in your ini file). If you supply oob (out of bounds — no callback), the verifier has to be entered by the user manually later on.

Consumers and Tokens

And the best part of this discovery is, that the Search Boss API doesn't use any OAuth tokens at all.

I guess I must have overread that in the process and only figured it out when I started looking at the sample OAuth code, or rather stepped through it all to figure out what it is doing (and what I was doing wrong). (More on that later!)

But anyway, I'm not afraid to learn new things.

Two for the price of one!

So once I mastered the credentials, it lead me to another thing: OAuth implementations in PHP are pretty sucky, and PHP doesn't seem to be alone here either. :-) And even if the implementation is not so sucky, their documentation usually is and real life examples are rather sparse.

So for example, I honestly don't know why developers who build OAuth wrappers rather explain OAuth for the who knows how many time, instead of providing example code. Example code goes far, and for reading yada yada just link to oauth.net. :-)

In case of the Zend Framework (and all things Zend_Oauth), I used the components unit tests to figure out a direction though that didn't take me too far.

Da codez.

In the end, I wrapped OAuth.php into a unit test and wrote some code using Zend_Oauth_Http_Utility until the requests looked the same. Here's what I ended up with!

oauth.ini

[production]
requestMethod   = 'GET'
signatureMethod = 'HMAC-SHA1'
consumerKey     = 'YOUR CONSUMER KEY'
consumerSecret  = 'YOUR CONSUMER SECRET'
version         = '1.0'

For simplicity, I'm not showing [testing] etc..

Nice to know: In case you're wondering how I came up with all the names, have a look into Zend_Oauth_Config::setOptions(). I particulary enjoyed looking for that one via an undocumented Zend_Oauth_Client::__call(). If you were wondering what @method is for in phpdoc, this is exactly it. ;-)

Make it all work!

So first off, we read the configuration file and push it all into an array $oauthParams.

We also initialize another array called $query (which is later appended to the URL).

Last but not least, we define $url, which is the address of the Search Boss endpoint.

$oauthCfg = new Zend_Config('oauth.ini', 'production');

$oauthParams = array(
    'oauth_version'          => $oauthCfg->version,
    'oauth_nonce'            => sha1(time() + rand(0,10)),
    'oauth_timestamp'        => time(),
    'oauth_consumer_key'     => $oauthCfg->consumerKey,
    'oauth_signature_method' => $oauthCfg->signatureMethod,
);

$query = array(
    'q'      => 'search query for yahoo boss',
    'format' => 'json',
);

$url = 'http://yboss.yahooapis.com/ysearch/limitedweb';

Both arrays are combined into $params for signature creation:

$params = array_merge($oauthParams, $query)

$utility   = new Zend_Oauth_Http_Utility;
$signature = $utility->sign(
    $params,
    $oauthCfg->signatureMethod,
    $oauthCfg->consumerSecret,
    null,
    $oauthCfg->requestMethod,
);

Then the signature is added to $oauthParams and we'll convert the array into an Authorization header:

$oauthParams['oauth_signature'] = $signature;

$oauthHeader = '';
foreach ($oauthParams as $key => $value) {
    $oauthHeader .= rawurlencode($key) . '="' . rawurlencode($value) . '",';
}
$oauthHeader = 'Authorization: OAuth ' . substr($oauthHeader, 0, -1);

Last but not least, we do the request!

First we assemble the complete URL, then we assign all the variables to the $client object and finally do a request!

$url = $url . '?' . http_build_query($query);

$client   = new Zend_Http_Client;
$response = $client->setUri($url)
    ->setMethod($oauthCfg->requestMethod)
    ->setHeaders($oauthHeader)
    ->request();
var_dump($response->getBody()); // json here!

If all went well $response contains an instance of Zend_Http_Response.

I hope I didn't forget anything, but it should be relatively straight-forward to wrap this into a lightweight object oriented wrapper.

Other pointers:

• always evaluate the status code (hint: REST-API)
• use try/catch

Fin

That's all. Sure hope this saves someone else some time.

Last week the usual round of PEAR-bashing on Twitter took place, then this morning Marco Tabini asked if PHP (core) was running out of scratches to itch. He also suggests he got this idea from Cal Evan's blog post about Drupal forking PHP.

• http://blog.tabini.ca/2011/04/is-php-running-out-of-itches-to-scratch/
• http://blog.calevans.com/2011/04/07/four-reasons-why-drupal-should-fork-php/

[Not submitting to your linkbait.]

Pecl and PHP

So first off — moving libraries from the core to an external repository was done for various reasons. One of them is to not have to maintain more and more in the core — keep it small and lean. Though small is pretty relative in this respect.

Of course doing so, means that people who do not have root on a server cannot install the module in most cases. But I'm inclined to suggest that when a pecl extension is (really) required, there should be nothing holding you back.

And if there is no way, thanks to PHP's extentability there's almost always a PHP-equivalent to any c-extension available.

Drupal and PHP

I know a couple Drupal people myself and most of them consider themselves to be Drupal developers before PHP. Why is that? It's because Drupal found a great way to abstract whatever people annoys about PHP from its developers, thus enabling them to build websites.

Is this a good or bad thing?

Of course it's a good thing because it makes people productive.

It's also a bad thing, because it seems that some (Drupal) people are rather disconnected from upstream [PHP].

Enabling people

Whatever people think about Drupal or any other framework, keep in mind that apparently it's PHP (and not Ruby, Python or pure C) which is more than a good enough enabler because PHP allows people to build a sophisticated content-management-framework like Drupal on top of it.

Drupal is of course no exception here. Despite e.g. the standstill in Ruby-land, in PHP other tools developer over the years who are a defacto standard: take a look at Wordpress or phpBB.

If you'd like to take it down to the framework-level there are projects like Symfony, CodeIgniter, Zend Framework, ezComponents/Zeta and also PEAR.

Fork vs. wat?

I think that forking PHP is a joke and I believe that Cal doesn't know the difference between a fork and a custom package (or a distribution).

A fork usually adds or removes features from the actual code base, but reading Cal's blog post he suggests a custom package. [Woo! Technical details! They get lost along the way!]

The thing is that a lot of people do this already. The people maintain a PHP package for a certain Linux or Unix distribution — Debian/Ubuntu, Gentoo or FreeBSD — there are doing it already. Using these as an example, whatever OS is used, it already runs a customized version PHP; some distributions customize more than others.

No one objects to the Drupal community suggesting ./configure flags or maintaining packages for the various flavours of Linux and Unix, or even Windows.

I would even go as far and say that in order to optimize the stack completely, it wouldn't hurt Drupal if its community recommended flags and extensions for people who run Drupal sites.

I doubt though that anyone will maintain packages for a couple distributions in their spare time and that the majority will not benefit from this effort because they don't run Drupal on their own server. But generally this optimization is enterprisey enough and indeed what I call a business opportunity.

Moving Drupal to ...

So what's "..."? Moving it to Python or Ruby, or maybe Scala? Good luck with that.

While the majority of Drupal developers don't consider themselves to be PHP developers, they still live and benefit off the PHP ecosystem. Think libraries used in modules or used for other areas like testing. Good luck porting those.

Then add PHP's vast adoption among webhosts.

Last but not least

Which brings me to the in my opinion biggest selling point: Doing PHP has another slight advantage over Ruby, Python and the other languages — it's installed on over 90% of the shared webhosts out there.

I invite everyone to google php hosting. It's trivial to find a host for as low as a Dollar per month — you just can't beat that.

Dear Cal, if you call this a business opportunity, I wonder why there's no Dollar Ruby hosting yet. Or Java hosting for a Dollar. But maybe someone is just not seeing this great business opportunity? [Note, Sarcasm.]

PEAR and PHP

What really bothers me about flaming PEAR is that the most vocal people in these flamewars never contributed any code. Open source is different from the Monday morning meetings some people are used to and where they talk people against the wall.

In open source land actual code contributions take the lead.

And while a lot of people complain about PEAR in general, here's something else:

• thriving download stats of packages
• PEAR package usage in other projects
• adoption of the PEAR coding standards and conventions
• PEAR channels thriving
• overall installer adoption

Despite being called a mess, PEAR is an enabler for many.

Point taken

PEAR being so many things is confusing!

PEAR packages are not as easy to use as some code you copy-pasted off the Zend devzone or phpclasses. While I agree, that we should try to make it just as easy, it's just not one PEAR's goals right now.

Scratching my own itch

Scratching your own itch, is what code contributions to PEAR are currently all about. Maybe always have been.

Active package maintainers most often contribute to packages they use themselves and they contribute to PEAR's environment to move development forward in areas where its beneficial to them. Call that selfish, but the reality is that most of us contributors actually work in this web industry and we know what we want and therefor we do it.

At the same time PEAR has coding standards and convention which are in place to ensure code is written so its to the benefit of most people.

Maintainership burden

The offiside to this situation is of course that components none of the maintainers have a use for get neglected — but calling this a PEAR-only problem really one-side.

Not even company-driven frameworks like the Zend Framework are prone to this; Zend_GData is/was rather unmaintained for a long time. Or frameworks where the proposal process and architecture are valued above all; I could point how broken ezcFeed is for me. Or general issues I see in projects where decisions are primarily driven by a single person — catch my drift?

This is not meant a pissing contest between frameworks, but I just can't hear it anymore.

Developer-friendly

Is PHP generally developer-friendly?

At the expense of watering the term developer, I'd say yes.

It is extremely easy to get started — embed the following into a .php file:

<?php
echo "Hello World";
?>

There's your PHP. It doesn't require a custom webserver process, root server or anything else. It really doesn't get any easier.

Are PHP frameworks easier than frameworks in [your other favorite language]?

Probably not! Or, hell no!

But that's the barrier of entry to any if not all frameworks on the planet. Some frameworks allow you to write your own blog in 10 steps, but you will soon discover that writing your own blog is not a great indicator for a quality framework.

Indiciators are:

• maintained code
• coding standards
• tests

And if I'm allowed a snarky remark — these are areas Drupal is literally just getting around to.

PHP vs PHP

Destruction breeds creation — but I get the impression that all these fights inside the PHP community don't really make it foster more.

Fighting and trolling may be an art for some and I agree they are entertaining at times, but when it becomes the only way people communicate contribute then let me be clear: it doesn't help.

The PHP community seems to be unaware how thriving PHP is and also its ecosystem. People often mistake stabilization for decline. There's nothing wrong if we don't crank out five new major versions every year.

• People in the real world are conservative anyway and adoption is slow. [Not a PHP-only problem either, just ask the Ruby folks.]
• People in the real world don't mind a more stable PHP environment, at the expense of buzzwords and all that crap.

Fin

In hindsight everyone always knew.

I feel like the more vocal people sharing their opinion, are pretty disconnected from the reality. That is dispite them running a magazine or a podcast about the community.

When people resort to flaming others in order to make themselves look smarter or their own project better, then that's just poor judgement on their part. Projects often die off as fast as they came about.

After running away screaming from Zend_XmlRpc we migrated of our internal webservices are RESTful nowadays — which implies that we make heavy use of HTTP status codes and so on.

On the PHP side of things we implemented almost all of those webservices using the Zend Framework where some parts are replaced by in-house replacements (mostly stripped-down and optimized versions equivalents of Zend_Foo) and a couple nifty PEAR packages.

RESTful — how does it work?

Building a RESTful API means to adhere to the HTTP standard. URLs are resources and the appropriate methods (DELETE, GET, POST, PUT) are used on them. Add status codes in the mix and ready you are.

To keep it simple for this blog post the following status codes are more or less relevant for a read-only (GET) API:

• 200: it's A-OK
• 400: a bad request, e.g. a parameter missing
• 401: unauthorized
• 404: nothing was found

... and this is just the beginning — check out a complete list of HTTP status codes.

Setup

To serve PHP almost all of our application servers are setup like the following:

1. nginx in the front
2. php (fpm) processes in the back

Nginx and PHP are glued together using fastcgi (and unix-domain sockets).

For an indepth example of our setup check out the nginx-app and php-fpm recipes (along with our launchpad repository).

Problem?

The other day, I noticed that for some reason whenever our API returned an error — e.g. a 404, for an empty result — it would display a standard nginx error page and not the actual response.

Solution

Digging around in /etc/nginx/fastcgi_params, I discovered the following:

fastcgi_intercept_errors on;

So what this does is that it intercepts any errors from the PHP backends and attempts to display an nginx error page. All errors may include the various PHP parse errors but apparently also a PHP generated page with a 404 status code.

So for example, the following code served by a PHP backend triggers the nginx page:

header("HTTP/1.1 404 Not Found);

The obvious fix seems simple:

fastcgi_intercept_errors off;

Sidenote: I think a similar issue might be in nginx's proxy_intercept_errors.

For both directives the manual suggests that they will intercept any status code higher than 400 — or 4xx and 5xx. But that's not all.

Tell me why?!

Reviewing the manual, I noticed that nginx will only act on fastcgi_intercept_errors on; when an error_page is (previously) defined. Checking out the rest of my sites configuration, the following part is to blame:

location / {
    error_page 404 /index.php;

    include /etc/nginx/fastcgi_params;

    fastcgi_pass  phpfpm;
    fastcgi_index index.php;

    fastcgi_param SCRIPT_FILENAME /var/www/current/www/index.php;

    index  index.php index.html index.htm;
}

So indeed the error_page 404 /index.php is what set it all off to begin with. And that's what I ended up removing, though it doesn't hurt to understand the implications of fastcgi_intercept_errors.

I think historically we used the 404 error handler as a cheap excuse for a rewrite rule since we only serve dynamically generated pages (and super-SEO-friendly URLs) to begin with. But that doesn't seem to be necessary — testing will be required.

Fin

The moral of the story is: nginx is really never to blame. ;-)

This is precisly what happens when you copy/paste configurations from the Internetz and don't review each and every single line to understand the full scope. In the end this was more or less a picnic on my part but I wanted to share it anyway because it was one of those WTF-moments for me.

Ever since I started playing around with Unix ~13 years ago, I've been a fan of automating things. What started out as writing little (maybe pointless) shell scripts slowly but surely morphed into infrastructure automation today.

As for my, or maybe anyone's, motivation to do these things, I see three main factors:

• I'm easily bored — because repeating things is dull.
• I'm easily distracted (when I'm bored).
• I'm German: Of course we strive for perfection and excellence. ;-)

Being on Unix (or Linux) it's fairly simple to automate things — add some script-fu to bash or csh (or even better zsh) and off you go wrapping things into a small shell script! Then execute again and again!

Before we decided to moved to AWS (and RightScale) in late 2009 we had half a rack of servers (in a Peer1's POP in NYC) and never did any or much infrastructure automation. We had an image and a set of commands to get a server up and running, but it was far from two mouse-clicks today.

At the time, I had read about cfengine a couple of times, but datacenter-grade infrastructure management along with a rather steep learning wall (at that time anyway) seemed overkill. Add to that, that there is not a lot of time for research Fridays when you work in a small company.

Moving to AWS and RightScale required us to write lots of small shell scripts using bash and Ruby. When we moved from RightScale to Scalarium in late 2010, we went from shell scripts to Chef.

Using Chef meant that we created so-called recipes which are used to bootstrap our servers. Recipes are little Ruby scripts which live in a cookbook — open source projects are sometimes so creative. Before this move I had very little, or next to no, experience with Chef, and Chef being Ruby didn't exactly make me want to try it either.

So what exactly is Chef?

A Chef recipe is a tiny bit of Ruby code — essentially a high(er)-level wrapper around calls such as installing certain packages, creating and parsing configuration files and many other things.

Chef offers a robust abstraction about everything you can do with shell and with a little effort it's also possible to write recipes which run on multiple OS'. Supported are Ubuntu, CentOS, FreeBSD and others. For an intro to Chef see the slides of a talk I gave a couple weeks ago; I briefly blogged about it too.

Our Chef recipes currently include things like installing and configuring PHP (from source and through a launchpad repository), nginx, MySQL, CouchDB, haproxy and many other things. The list was literally growing every day for the first few weeks.

One of the more frequent questions I see on the mailing lists and IRC is, "How do I take over a package?".

Very often people start to use a PEAR package and then at some point encounter either a bug or they miss a certain feature. The package's state however is inactive or flat unmaintained.

Offer help

The first step is to offer your help. And the easiest way to help is to report bugs or open feature requests.

A great bug report or feature request includes:

1. Accurate problem description.
2. A proposed solution.
3. A patch (unified diff, aka diff -up or svn diff)
4. A PHPUnit test case so this bug stays fixed.

I realize that these are a lot of requirements, but I also said a great bug report and no one said it was easy.

Getting fixes in

Often people persued these first steps and went the whole nine yards. But the bugs are still not fixed, the feature isn't implemented either and no release was pushed out.

How do you move forward now?

1. Email the maintainer and cc [email protected].
2. Ping PEAR QA again after reasonable period of time passed (e.g. 1 or 2 weeks tops)

If the package is generally unmaintained or the maintainer did not respond, PEAR QA will be happy to apply patches for you and do a new release. If the maintainer replied, work with them to get the issues fixed and a new release shipped.

Do you want to maintain?

To become the maintainer of any package we need to see your contributions first.

Contributions include:

• patches for bugs and features
• tests
• bug reports
• (maybe) helping people on the IRC channel

The fastest way to take over an orphaned package is to provide patches to any open issues so the QA team can push out a new release.

Contributing code is not just the preferred way but a hard requirement. It's the only viable way for others to assess your skills.

Another way to maintainership is of course when the current maintainer asked you if you want to contribute more since you showed them an interest through your work.

Find a package

Here's a list of currently unmaintained PEAR packages: http://pear.php.net/qa/packages_orphan.php.

Fin

I hope I summarized the process a little to make it more transparent. In the end it's pretty simple and straight forward to contribute to PEAR — and lots of people will be tankful for your contributions too.