Update, 2010-03-04: I just rolled a 0.0.2 release. In case you had 0.0.1 installed, just use pear upgrade-all to get it automatically. This release is trying to fix a random hang while reading documents from the source server.
I also opened a repository on Github.
---
As some may have guessed from a previous blog post we are currently running a test setup with CouchDB lounge. My current objective is to migrate our 200 million documents to it, and this is where I am essentially stuck this week.
No replication, no bulk docs
The lounge currently does not support replication (to it) or saving documents via bulk requests, so in essence migrating a lot of data into it is slow and tedious.
I have yet to figure out if there is a faster way (Maybe parallelization?), but DB_CouchDB_Replicator is the result of my current efforts.
I think I gave up on parallelization for now because it looked like hammering the lounge with a single worker was already enough, but generally I didn’t have time to experiment much with it. It could have been my network connection too. Feedback in this area is very, very appreciated.
DB_CouchDB_Replicator
DB_CouchDB_Replicator is a small PHP script which takes two arguments, --source and --target. Both accept values in style of http://username:password@localhost:port/db and attempt to move all documents from source to target.
Since long running operations on the Internet are bound to fail, I also added a --resume switch, and while it’s running it outputs a progress bar, so it should be fairly easy to resume. And you also get an idea of where it’s currently at and how much more time it will eat up.
These switches may change, and I may add more — so keep an eye on --help. Also, keep in mind, that this is very alpha and I give no guarantees.
Installation
Installation is simple! :-)
apt-get install php-pear
pear config-set preferred_state alpha
pear channel-discover till.pearfarm.org
pear install till.pearfarm.org/DB_CouchDB_Replicator
Once installed, the replicator resides in /usr/local/bin or /usr/bin and is called couchdb-replicator.
Fin
The code is not yet on github, but will eventually end up there. All feedback is welcome!