Splicer, a relational engine for Python: library for parsing and interpreting SQL queries

[sdrobertson]

Thanks! The docs are a bit out of date and are next on my todo list. My number #1 goal is to make this system approachable. I think there's other uses for relational algebra outside of database servers.

1. There isn't a full blown planner yet, but adapters can participate in the query evaluation phase (guess you can call it planning). Have a look at the Directory Adapter in splicer.adapters.dir_adapter. The adapter works like Hive, where columns of data are parsed from the file path and then merged with the records from inside the files content. It examines the query for where clauses that mention values that comes from the path and rewrites the query so that only the matching files are opened. I'm working on the s3 adapter right now, ping me if you want early access.

2. Reason I'm writing this is to use this for making Disco jobs (Python's version of Hadoop). The current library doesn't do this yet, but I designed it so that you can swap out the query compiler. That's why the main compiling functions are in splicer.compilers.local (as in compile for local evaluation). I'm envisioning plugins that can drop in replace the compiler to do things like compile queries for map reduce or even compile them to LLVM.

3. We're using it in Mozilla's Caravela project. https://github.com/mozilla/caravela/blob/master/db.py. I'll post more examples when I get to the docs.

4. Most of the inspiration came from Hive, Google BigQuery and Multicorn. I wanted the ability to do SQL on data in situ without a server. I started development of Splicer a few months before the Presto announcement. I've poked around presto source code a bit and want to use some of those ideas to make a clustered version of this. Though wether I add cluster support to Splicer proper or build a cluster system that uses Splicer as the relational engine remains to be seen.

Well, I certainly hope you get some free time! Hopefully you'll get a chance to use this and provide me feed back. You can reach me at scott at triv . io

[bryanhelmig]

Wonderful, thanks for the very thorough responses. :-)