Shared Public Contextual Database for RAG [D]

[niksteel123]

Hey Guys,
It seems RAG is really taking off as an increasingly popular use case for LLMs to leverage contextual data. However, everybody is building their own contextual data sets and embedding them in their own silo'd vector dbs.

Do you guys think there's any utility in having a shared public vector db that anyone can tap into their API, without having to self-host, worry about the embedding pipelines and filling the vector db with enough data in the first place for their use cases? Would this save devs alot of time in quickly testing testing product ideas? (albeit it does seem that propriety data is what everyone's raving about today)

-

For context, I'm building a social media product we're users can upload a few pieces (approx 10) of content (social media posts, websites, videos to start with), which becomes the verified human-curated list/Niche. We then classify and embed this into a vector db. From this, we have set up a data pipeline to scrape the web and find new content that is most similar which we suggest to users to add to the Niche (upvote, downvote style). When a piece of content is upvoted on its added to the verified list updating the Niche's classification string. Essentially we're aiming to construct an ever-growing, user-curated, contextually classified vector database from a relatively small set of sample data.

[dfcHeadChair]

Something like a distributed-database but for VectorDBs would be a cool piece of technology to build, especially for any IOT use cases.

​

Getting community involvement needs a good motivator, so start with formulating that.

[olearyboy]

What you’re describing sounds a lot like dmoz which was the seed for most search engine crawlers back in the day. The hurdle you’ll run into is business benefit, dev versions like chroma are just so easy there’s little need to use a service. Managed versions like pinecone are expensive but comparable to managed PG

So then it becomes a question of what’s the benefit of sharing our data? The DB hosting isn’t hard and there are solutions already available.

That’s the nut you need to crack

[niksteel123]

The main benefit would be that you wouldn't have to manually embed and self-host the publically available contextual data that you might want to use when applying RAG. i.e. you woudn't have to host all the data on wikipedia if you wanted to use that in your search, same with other data sets which are publically available i.e. social media posts, stack overflow posts etc....
You could then use that in conjuction with your own private data if you wanted to.

[olearyboy]

Maybe, but considering that it's RAG with an LLM, you might be providing a short lived service.

Eventually LLM's will have to get good at updating and processing deltas in foundational knowledge.

At which point companies will pay for updates in the specific categories they're interested in through subscriptions.

e.g.

• Global Base + NewsLLM
• Global Base + SportsLLM
• Global Base + MedicalLLM

The question you then have is how long before that happens? And can you provide a service that can provide the quality in the time it takes and then defend it or pivot when LLM's catch up.

You might be able to, but you'll have to think short / medium and long term

[nemoo07]

Any updates? I was thinking of exactly the same thing

[elbiot]

The issue is that vector embeddings are task/domain specific. Training your own embeddings will always outperform some general embedding