c. scott andreas

Over the past few weekends, I’ve been working to create a Datamapper adapter for Tokyo Cabinet’s TDB (table-like) datastore for use with Ruby scripts and applications.

Many have suggested that “if you need/want an ORM, non-relational databases aren’t for you.” I’d like to suggest that this isn’t necessarily the case.

Get the code right now

Like MySQL, Tokyo Cabinet has several different storage engines designed according to different paradigms, such as a simple hashtable, a b+ tree, a fixed-length datastore, and a table-like database.

Tokyo’s table datastore is particularly interesting in that it is built upon the hash datastore and mimics much of the functionality found in standard relational databases, but is internally non-relational and has no fixed schema or data types. As such, TC/TDB offers much of the power of a traditional relational database like MySQL, Postgres, et al, but with tremendously more flexibility. (Note that my TDB implementation is distinct from Makoto Inoue’s BDB implementation, which is built upon Tokyo’s b+ tree datastore). I’d like to thank Makoto for providing the inspiration behind this project and for giving me confidence that writing an adapter for Cabinet/TDB would be both possible and fruitful.

I decided to build the Datamapper adapter as an experiment in making Tokyo Cabinet more accessible to the Ruby community. By placing it under Datamapper, it’s possible to work with content much like ActiveRecord using calls like “Post.first(:slug.like => params[:slug])” – just as you’d use any other database. It’s worth mentioning that my motivation had nothing to do with the dreaded “s-word” and entirely with wanting to try something new and fun.

DANGER AND STUFF
If you’re using this adapter in its present condition to do anything important, you’re probably crazy. That said, here’s a proof-of-concept blog I whipped up using this adapter and Sinatra called Stockness. It’s full of really terrible stock photos that I’ve found online and on CDs.

Here’s what works, what doesn’t, and what I haven’t tried:

What works:

• Basic data types: String, Integer, Text, Date/Time, Boolean
• Standard CRUD operations: Create, read, update, and delete all the data you like.
• Basic Datamapper queries: support for standard “equal, not, like, greater than, less than, greater than or equal to, and less than or equal to” queries.
• Basic associations: I’ve tried has_n and belongs_to (comments on the blog linked above are powered by such an association).

What doesn’t:

• Parameterized SQL queries: Anything like “conditions = [‘foo = ? and bar = ?’, foo, bar]” is not yet supported and may require restarting the application if attempted. Since the TDB datastore is not SQL-based (and has no concept of SQL in the first place), it would take some more work to translate this syntax into TDB’s “query.addcond()” approach.
• “Model.save” and “Model.update_attributes” always return false. I’m not sure what’s causing this, but if a Datamapper guru sees this and cares to offer some thoughts, that’d be much appreciated!
• When a new object is created, its ID is not immediately attached. Querying for the object after saving will return the full object, with ID included. This one’s super high on the list – sorry!
• I need to drop in support for ordering results (asc/desc). This really isn’t difficult at all, but it didn’t make the first preview release. Pardon that.
• Something ood is happening with DateTime columns, but it’s hard for me to pin down just now (and may just be an issue in the sample blog app I threw together). More info soon.

What I haven’t tried:

• Validations, callbacks, complex queries, and a million more things I haven’t thought of.

Using the TDB adapter
Implementing the TDB adapter is pretty simple. Just make sure that you’ve installed Tokyo Cabinet and its Ruby bindings, along with Datamapper. Then, begin your script with something like this, and you’re on the way:
require 'rubygems'

require ‘lib/tokyo_cabinet_adapter’

DataMapper.setup(:default, :adapter => ‘tokyo_cabinet’, :data_path => Pathname(FILE).dirname.expand_path + ‘data’)


Make a folder called “data” in the application’s root, then start building something just as you would with Datamapper using, say, MySQL or SQLite.

Getting Help
If you have a question about this adapter, how to use it, or have stumbled on one of what I’m sure are a mountain of bugs, leave a comment, hit me up on Twitter as @cscotta, shoot me a message on GitHub (cscotta), or drop me an e-mail at scott [at] phoreo [dawt] com.

Enjoy!

Come and get it!

Haystack is my C++-based entry in Engine Yard’s cryptography contest.  Download it here and give it a try.  I’ve provided compiled Mac binaries for Intel’s Core 2 Duo and Dual Core Xeon processors (MacBook Pro/iMac/Mini and XServe, respectively).

[ DOWNLOAD NOW ]

Source is included, or see it at GitHub.  Screenshot below.

Good luck!

If you’re not making daily backups of your databases, you’re either not doing anything important or living dangerously.

Here’s a little Ruby script I wrote to remotely back up a MySQL database each night.

This script is designed to log into your server via SSH, back up a database of your choice using mysqldump, compress it using tar+gzip, and download the result before cleaning up after itself.

This method is preferable to one that might run on the server then push the backup to an off-site location.  In the event that your server became compromised, the attacker would be able to remotely erase the backups you’ve created.  By logging in to pull them rather than push, this vulnerability is mitigated.

Steps to install and use:

• Make sure you’re running Ruby 1.8.x or higher and that you have the net/ssh and net/sftp gems installed (If not, do a quick (sudo) gem install net-ssh net-sftp)
• Fill in the SSH and MySQL Database configuration sections of this script.
• Create a directory on your computer to store the backups, and create another on the server for the temporary files to be stored.  Then, enter these as absolute paths without trailing slashes in the “Backup File Locations” section.
• You should be good to go.  Type “ruby backup_db.rb” to give it a whirl.
• If you’d like to run this automatically, add it to your crontab.  A sample entry might look like this:  0 0 * * * ruby /home/user/scripts/backup_db.rb
• Protip: Replace the “/home/user/scripts/” section with the actual path to the script.

About a month ago, Jeff Atwood wrote a post describing the Internet as a “bathroom wall” for code.  It was an odd analogy, but it hit home - often when Googling for a quick solution to a common problem in a particular language, one will find a pile of half-assed and/or suboptimal answers that have a high PageRank, drop them in, and quickly discover that they’ve given their app the equivalent of some gross STD.

Welcome to my bathroom wall.

I make no guarantees of the code posted here, other than that it was at some point useful to me.  Most of it will lack tests, it may perform poorly under heavy load, and odds are decent it’s not a drop-in solution for your problem.

But it worked for me, and many of the problems I solve are pretty common.

Feel free to offer suggestions, corrections, or to berate me wildly.  That’s half the fun of it, anyway.

You can imagine how much fun I had writing this. Maybe you’ll find it useful.

Day 1: I will pretend never to have heard of Twitter, fail to understand.

Day 2: I will refuse to talk to anyone who does not call themselves a rockstar, ninja, superstar, sniper, or code poet.

Day 3: I will defend the IE 5.5 UI to my death and laud its rendering engine with praise due to outstanding ActiveX integration and support for Java.

Day 4: My first words in every conversation will be, “Is there an iPhone app?”

Day 5: In any conversation with a developer, startup, or business dude, my first words will be: “I have no idea what you’re talking about, but here’s what you should do,” then mumble something about mobile web 2.0 advertising.

Often when I’m feeling frustrated or burnt out when working on a project, I’ll take a break to experiment with a new technology, language, or framework.

After all, the cure for too work is…more?  Maybe I’m doing this wrong.

But playing with new ideas and platforms is a great way to break out of a funk.  After finding myself burnt out working on a side project, I started experimenting with building an iPhone app (for about the fourth time).  While my previous attempts left me frustrated and feeling helpless against the powers of CGRect and NSWhatever, things began to click this time.

I spent the entire weekend hammering this out, and ended up refactoring significant critical portions of Sunago.org in the process.  By Sunday night, Sunago got a RESTful API and an iPhone app with basic CRUD functionality powered by (and updating) a remote data source was working great.  I was still ignorant, but invigorated.

I wouldn’t describe myself as a proper polyglot, but I do believe that these experiments have left me a better programmer.

In another, I abandoned a planned roadmap for a weekend and found myself hell-bent on migrating Sunago to Ruby 1.9/YARV and got it running on JRuby to boot.  Though Monday came with another feature unimplemented, the application itself emerged stronger and more portable while I became more acquainted with the intricacies of various Ruby implementations.

During another weekend, I hatched a secret project with Matt King and hammered out a prototype.  I admit that I’ve been grossly delinquent on the follow-through (sorry, Matt) - but learned a ton experimenting with unfamiliar protocols, microframeworks and communication platforms (more on this soon).

I’m reviving this blog to keep track of these experiments and hopefully share something valuable in the process.

Enjoy.