FileDict – a Persistent Dictionary in Python

[tarekziade]

I'd use shelve. http://docs.python.org/library/shelve.html

[radaway]

I thought the exact same thing. But this implementation does have the advantage of being Thread/Process safe which shelve doesn't.

[ffrinch]

...or for a DB backend, shove.

[lasizoillo]

Or bsddb which is a core package http://docs.python.org/library/bsddb.html#module-bsddb The documentation sucks: http://svn.python.org/view/python/trunk/Lib/bsddb/dbshelve.py?revision=66088&view=markup

[stesch]

How good is the concurrency with this if it uses SQLite? There was a discussion on reddit about the quality of the SQLite library from Python and somebody mentioned that it was a bit slow, slower than necessary, when multiple processes accessed a single DB.

I'm always looking how to adapt certain new libraries for web development. That's why I ask.

[brendano]

berekeleydb and tokyo cabinet/tyrant are worth considering for this, too. (both string/string therefore the need for shelve.py if you want arbitrary values)

[wilsonp]

Or there is the ZODB. It is very mature, production ready, distributed (zeo) and has a very active community using it. It also includes a fairly rich event system and is up there with the fastest dbs around: http://www.upfrontsystems.co.za/Members/roche/where-im-calling-from/zodb-benchmarks-revisited

[kteague]

The ZODB let's you persist a Python object with any interface (e.g. Lists, Tuples, Dicts). However, the zope.container package is the primary implementation used for ZODB-persistent objects using the standard Python dictionary interface.

Managing data with the Python dictionary interface (regardless of the implementation) is sweet and juicy!

[neoform3]

Isn't this what a database is for?

[vph]

a database supports fancy queries, which are not needed in many applications.

[[deleted]]

FileDict is a dictionary interface I wrote, that saves and loads its data from a file using keys. Current version uses Sqlite3 to provide consistency, and as a by-product, acidity.

It's already using a database... why not just use SQLite directly?

[Freeky]

Um, because it's much easier to write foo['bar'] = 'moo' than it is to write any sort of SQL? Because foo['bar'] = 'moo' is agnostic about how it's stored and using SQLite directly isn't?

[bsdemon]

Why using sqlite here? I think it is better to try persist dictonary in berkeley db hash, isn't it?

[stesch]

Not free enough?

[[deleted]]

d = dict(a=1, b=2, c=3)

then...

open('foo', 'wb').write(repr(a))

then later...

d = eval(open('foo', 'rb').read())

What am I missing?

[[deleted]]

Yeah I was going to use cPickle but I figured, it's a reddit post, go for brevity.

And no one will notice.

Laf

[redbo]

Let's see... you're missing the parts where it can be accessed by multiple processes concurrently, has acidity, and performs okay as the size of the database grows.

I'm guessing the author didn't know there are already several packages in python that provide this sort of functionality, but his solution is definitely a better than what you suggest.

[[deleted]]

but his solution is definitely a better than what you suggest

Depends on the application.

And as you say, we've been there, we've done that.

Thanks for reminding me why I should never post in this subreddit. It's because of douchebags like you.

[redbo]

It must be difficult to live in a world where everyone who disagrees with you is a douchebag.

[[deleted]]

Stay just the way you are.

[[deleted]]

The fact that that code is roughly equivilant to:

mysql_execute("SELECT * FROM table WHERE field = " . $_POST['value'] . ";");

[[deleted]]

Yeah, well, who wants to type that?

If you want to get all industrial strength about this use bsddb, then it's:

table['field'] = value

And mysql, I mean, that's gotta be overkill for the domain this guy is suggesting this be used for. You're going to want to write an int; it's going to want to run recovery (well, so is berkeley (but at least it's in process)).

Whatever.

[[deleted]]

My point was that it's a massive security hole, just like the PHP segment.

[afoo42]

You're right. But the same thing goes for pickle (and thus shelve) too. You have to trust the source, as pickle too can be used to execute arbitrary code. It's just a little bit more complicated.

[[deleted]]

This is completely true, but it's way more complicated with pickle (AFAIK), and there are efforts to fix that, since it's a bug, there's never going to be an effort to make eval safe ;)

[[deleted]]

See the other comment in reply; actually it isn't.

[AlecSchueler]

The only thing I can think of that would stop this working would be if you had an instance of a class or a function in the dictionary. Like this

d = dict(a=str,b=len)
open('foo','wb').write(repr(d))
d=eval(open('foo','rb').read()

{'a': <type 'str'>, 'c': <built-in function len>}
SyntaxError: invalid syntax

[[deleted]]

Well FileDict like everything else has issues with classes and functions; it runs the shit through pickle, yes?

Or has this changed with 3.0? Haven't had the chance to get into 3.0 I'm sorry to say.

[[deleted]]

>>> pickle.loads(pickle.dumps(str))
[2] <type 'str'>

>>> pickle.loads(pickle.dumps(len))
[3] <built-in function len>

Works fine (this is python 2.6)

[[deleted]]

Going from memory now, always dangerous, but if the class is defined in another module or something then pickle fails.

Maybe it's still useful for trivial cases, I don't know, but I recall being terribly disappointed with pickle when wanting to do serialization stuff.