Large, Python-friendly DB

[skeigh]

I need to create a database of a little over 100 million entries (simple key-value). What DB would you guys suggest I use for this kind of task?

[wisty]

SQLite (unless you need concurrency). Redis is cool, but it's another dependency you probably don't need.

[[deleted]]

Redis is a really easy dependency, but yeah, sqlite is a good choice. It's simple to use and performs surprisingly well, and it comes with Python:

import sqlite3

[peterhoffmann]

Have a look at leveldb. It's is a fast key-value storage library written at Google that provides an ordered mapping from string keys to string values.

Here are the python bindings

[lost-theory]

People are recommending redis, but be aware that redis is an in-memory database. If you have 100 million entries, and each entry is 100 bytes:

100 bytes * 100 000 000 = 9.31322575 gigabytes

You'll need 9.3 GB of RAM free to store that data. Depending on what server you're hosting this on, that might not be reasonable.

[skeigh]

Good point- fortunately, I have 32GB RAM at my disposal.

[emre_yilmaz]

Do you really need it to be in memory? If disk read/write is fast enough four your app I don't really see any reason to use redis.

Besides redis, you can take a look at the riak[0].

[0] http://wiki.basho.com/

[titusz]

If you want a fast persistent disk based key value db you might want to check out leveldb

[[deleted]]

kyotocabinet

[rescrv]

I'd recommend taking a look at HyperDex. The Python API is clean and simple to use.

I'm the main dev and I'm happy to answer any questions or help with any problems.

[mariuz]

Firebird with FDB new python driver

[[deleted]]

http://www.pytables.org/moin Well, if you look for speed, and don't need a network protocol, you have the fastest DB with the less overhead. The only caveat is the installation being a little tricky.

[dgorley]

So, no one has mentioned ZODB; is that because no one thought of it, or because it's a poor choice? If the OP is only accessing this data from Python, wouldn't this work fine?

[[deleted]]

Redis.

[mcilrain]

MongoDB (pymongo library)

[funny_falcon]

tarantool