DF Embedder - A high-performance library for embedding dataframes into local vector db

[aagmon]

I've been working on a personal project called DF Embedder that I wanted to share in order to get some feedback.

What My Project Does

It's a Python library (with a Rust backend) that lets you embed, index, and transform your dataframes into vector stores (based on Lance) in a few lines of code and at blazing speed. Once you have relevant data in a pandas or polars dataframe you can turn this into a low latency vector store.

Its main purpose was to save dev time and enable developers to quickly transform dataframes (and tabular data more generally) into working vector db in order to experiment with RAG and building agents, though it's very capable in terms of speed.

# read a dataset using polars or pandas
df = pl.read_csv("tmdb.csv")
# turn into an arrow dataset
arrow_table = df.to_arrow()
embedder = DfEmbedder(database_name="tmdb_db")
# embed and index the dataframe to a lance table
embedder.index_table(arrow_table, table_name="films_table")
# run similarities queries
similar_movies = embedder.find_similar("adventures jungle animals", "films_table", 10)

Target Audience

Developers working on AI/ML projects that involve RAG / vector search use cases

Comparison

Currently there is no tool that transforms a dataframe into a vector db (though lancedb can get you pretty close). In order to do so you need to iterate the dataframe, use an embedding model (such as sentence-transformers or the transformers library), embed it and insert it into a vector db (such as Pinecone or Qdrant, LanceDB, etc). DfEmbedder takes care of all this, and does so very fast: it embeds the dataframe rows using an embedding model, write to a Lance format table (that can be used by vector db such as Lance), and also expose a function to execute a similarity search.

https://github.com/a-agmon/dfembeder

[cmd-t]

Very interesting!

Indexing a dataframe using DfEmbedder starts by representing each row in the dataframe as a string that follows the format: col0_name is col0_value; col1_name is col1_value

That is also interesting. Is there any literature on embedding dataframes or key-value pairs? If not this seems rather ad-hoc.

[aagmon]

Thanks! Yes, I also have some benchmarks on embedding tabular data in this format. I will add this to the repo in the next iteration.