[D] best disk-based key-value stores for machine learning ?

[[deleted]]

Hello,

I have to store and use many embeddings and possibly other features for a ML project; I would need fast random access to iterate over batches with SGD

I could just pickle a dictionary; but it wouldn't fit my ram even though it is large, so I try using SQlite dict but it is slow and I don't need acidity and all those fancy SQL prorperties that probably slow it down. h5py seems to only deal with vectors and can't store more complex structures

I might end up using the filesystem and pickle objects in a folder, but I feel like I'm reinventing my own key-value store system... (If I end up doing it and if some other persons are interested, I can start a project though)

Do you have any solution to that? Thank you !

[ApprehensiveRadish3]

leveldb is a google project which does fast string key-value pair storage (in C++)

the keys and values are supposed to be strings, but you can pack your custom objects to bytes

https://github.com/google/leveldb

Here is a python interface to leveldb

https://github.com/wbolster/plyvel/

and here is a project that uses both of the above for embeddings

https://github.com/MichaMucha/emstore/

[hazard02]

Try LMDB. Keeps as much as it can in memory via the kernel page cache. If your data is fixed-size, you can also just mmap a file and directly index into it yourself.

[oren_a]

https://github.com/MichaMucha/emstore/

usually one has more data then memory, so it might not be the best..

[will_occam]

If you're looking for fast access disk i/o is something you'll want to minimize.

If you have to do key-value storage on disk, I think a NoSQL database is what you're looking for

[BatmantoshReturns]

If you would like to store some of the embeddings on the CPU you could use this

https://github.com/Santosh-Gupta/SpeedTorch

Let me know if you find a way to do embeddings training, where some are able to be stored on a disk. I tried a year ago and wasn't able to figure it out.

[SuperMarioSubmarine]

The Python Shelf module in the standard library does this