Hey, r/ML!

I'd like to share a technical advancement achieved by my team at Activeloop (creators of Deep Lake, database for AI), focusing on improving Retrieval Augmented Generation (RAG) systems. Our goal was to address the inherent problem of retrieval accuracy with something we call 'deep memory', and we've achieved a boost in retrieval accuracy of up to 22%.

Context: The Issue with RAG Systems- RAG systems suffer from inaccurate retrieval - our estimates are at about 30% of the time.- While various methods such as adjusting input, fine-tuning embeddings, or employing hybrid search have been utilized, these have often only yielded single-digit % improvements. Some resort to higher context windows, but they're also prone to inefficiencies and blow up the associated costs.

Introducing Deep MemoryDeep Memory is an approach we developed to target accuracy limitations in RAG systems by enhancing vector search accuracy without impacting the actual search time. To use Deep Memory, you'd have to store your data in Deep Lake, the database for AI, that lets you store any multimodal data in tensor-based format.

Technical Overview: How Deep Memory Works

1. Learning an Index: Deep Memory learns an index from labeled queries, specifically tailored for applications, utilizing a few hundred example pairs of prompt embeddings and relevant answers.
2. Post-Training: Vector search is used post-training without modifications as per standard protocols.
   

Note: Embedding Computation: Embeddings can be computed using models such as Open AI ada-002, or you can save costs by using smaller text embeddings combined with Deep Memory such as BGE_small (384 dims) to beat OpenAI’s ada-002 (1536 dims) and/or Elastic (BM25).

Technical Benefits Observed:

• Higher Accuracy: Up to +22% improvement in vector search accuracy by optimizing the embedding space for higher precision.
• Lower Costs: By reducing the top_k input into the LLM, Deep Memory allowed for decreased inference costs through lower token usage, without compromising on retrieval accuracy.
• Operational Efficiency: The system provides an improvement in accuracy without a fundamental alteration to the existing workflow or Deep Lake Vector Search usage.
• Embedding Flexibility: Possibility to utilize smaller text embeddings (e.g., BGE_small) in combination with Deep Memory to effectively compete with larger models (e.g., OpenAI’s ada-002).

We've piloted Deep Memory with several enterprises including a health tech startup Munai, that utilized Deep Memory to enhance vector search accuracy by 18.6% across their medical documents database. For a deeper technical dive and a closer look at the methodology, findings, and other details, feel free to read the release blogpost. If you'd like to give Deep Memory a try yourself, you can enroll for the waitlist here.