c. scott andreas

Over the past few weekends, I’ve been working to create a Datamapper adapter for Tokyo Cabinet’s TDB (table-like) datastore for use with Ruby scripts and applications.

Many have suggested that “if you need/want an ORM, non-relational databases aren’t for you.” I’d like to suggest that this isn’t necessarily the case.

Get the code right now

Like MySQL, Tokyo Cabinet has several different storage engines designed according to different paradigms, such as a simple hashtable, a b+ tree, a fixed-length datastore, and a table-like database.

Tokyo’s table datastore is particularly interesting in that it is built upon the hash datastore and mimics much of the functionality found in standard relational databases, but is internally non-relational and has no fixed schema or data types. As such, TC/TDB offers much of the power of a traditional relational database like MySQL, Postgres, et al, but with tremendously more flexibility. (Note that my TDB implementation is distinct from Makoto Inoue’s BDB implementation, which is built upon Tokyo’s b+ tree datastore). I’d like to thank Makoto for providing the inspiration behind this project and for giving me confidence that writing an adapter for Cabinet/TDB would be both possible and fruitful.

I decided to build the Datamapper adapter as an experiment in making Tokyo Cabinet more accessible to the Ruby community. By placing it under Datamapper, it’s possible to work with content much like ActiveRecord using calls like “Post.first(:slug.like => params[:slug])” – just as you’d use any other database. It’s worth mentioning that my motivation had nothing to do with the dreaded “s-word” and entirely with wanting to try something new and fun.

DANGER AND STUFF
If you’re using this adapter in its present condition to do anything important, you’re probably crazy. That said, here’s a proof-of-concept blog I whipped up using this adapter and Sinatra called Stockness. It’s full of really terrible stock photos that I’ve found online and on CDs.

Here’s what works, what doesn’t, and what I haven’t tried:

What works:

• Basic data types: String, Integer, Text, Date/Time, Boolean
• Standard CRUD operations: Create, read, update, and delete all the data you like.
• Basic Datamapper queries: support for standard “equal, not, like, greater than, less than, greater than or equal to, and less than or equal to” queries.
• Basic associations: I’ve tried has_n and belongs_to (comments on the blog linked above are powered by such an association).

What doesn’t:

• Parameterized SQL queries: Anything like “conditions = [‘foo = ? and bar = ?’, foo, bar]” is not yet supported and may require restarting the application if attempted. Since the TDB datastore is not SQL-based (and has no concept of SQL in the first place), it would take some more work to translate this syntax into TDB’s “query.addcond()” approach.
• “Model.save” and “Model.update_attributes” always return false. I’m not sure what’s causing this, but if a Datamapper guru sees this and cares to offer some thoughts, that’d be much appreciated!
• When a new object is created, its ID is not immediately attached. Querying for the object after saving will return the full object, with ID included. This one’s super high on the list – sorry!
• I need to drop in support for ordering results (asc/desc). This really isn’t difficult at all, but it didn’t make the first preview release. Pardon that.
• Something ood is happening with DateTime columns, but it’s hard for me to pin down just now (and may just be an issue in the sample blog app I threw together). More info soon.

What I haven’t tried:

• Validations, callbacks, complex queries, and a million more things I haven’t thought of.

Using the TDB adapter
Implementing the TDB adapter is pretty simple. Just make sure that you’ve installed Tokyo Cabinet and its Ruby bindings, along with Datamapper. Then, begin your script with something like this, and you’re on the way:
require 'rubygems'

require ‘lib/tokyo_cabinet_adapter’

DataMapper.setup(:default, :adapter => ‘tokyo_cabinet’, :data_path => Pathname(FILE).dirname.expand_path + ‘data’)


Make a folder called “data” in the application’s root, then start building something just as you would with Datamapper using, say, MySQL or SQLite.

Getting Help
If you have a question about this adapter, how to use it, or have stumbled on one of what I’m sure are a mountain of bugs, leave a comment, hit me up on Twitter as @cscotta, shoot me a message on GitHub (cscotta), or drop me an e-mail at scott [at] phoreo [dawt] com.

Enjoy!