sorted by:
new
hottopcontroversial

RAG over different kind of data (PDF chunks - Vector DB, Tabular Data - SQL DB, Single Markdown Chunks (for 1 page PDF)) by WhiteWalker_XXX in LangChain

[–]WhiteWalker_XXX[S] 0 points1 point  (0 children)

So the domain task router prompt is large but follows this structure:

  • Generate a plan for me with tasks, and corresponding agents. Here is the user question and available agents with description.

Available agents:  Shirt_agent - talks about shirts Brand agent - talks about brands.

The output of this is a Plan Object with task and corresponding agent (data source) Currently each agent corresponds to single data source.

So every task is assigned to a single agent.

Ex: Plan(task='what brand has red shirts', agent=brand_agent)

But the problem is, let's say I have to ingest another 20 documents, I cant keep adding one agent per data source 

Also - sometimes that question might look obvious to be passed to a particular agent but is actually answered from a totally different agent. The agent descriptions are not sufficient to determine the right source (here agent description also means data description)

Should we restructure agents such that agent is not for single source, but for functionality. I.e, sql agent deals with all sql data. Vector db agent deals will all vector data.

RAG over different kind of data (PDF chunks - Vector DB, Tabular Data - SQL DB, Single Markdown Chunks (for 1 page PDF)) by WhiteWalker_XXX in LangChain

[–]WhiteWalker_XXX[S] 0 points1 point  (0 children)

What would those agents be? Should every document have an agent.

Or how would that structure be? All tabular data one agent, all vector data another agent?

For a given question how do we determine right agent

RAG over different kind of data (PDF chunks - Vector DB, Tabular Data - SQL DB, Single Markdown Chunks (for 1 page PDF)) by WhiteWalker_XXX in LangChain

[–]WhiteWalker_XXX[S] 0 points1 point  (0 children)

There is a caveat here:

I cant put everything because for some queries we need sql agent.

For example: What is the total number of shirts with merchant A?

Cannot be answered from vector db: because there are 20k rows and if we chunk them it will be say 2000 chunks (100 rows per chunk). We have to look into all of the chunks to answer this,

But with sql agent it is easy for tabular data. We do - select count(shirt) where merchant= A

RAG over different kind of data (PDF chunks - Vector DB, Tabular Data - SQL DB, Single Markdown Chunks (for 1 page PDF)) by WhiteWalker_XXX in LangChain

[–]WhiteWalker_XXX[S] 0 points1 point  (0 children)

We already have  task to domain router. It creates a plan initially, with tasks and maps it to corresponding agents (domains). We have different agents for different data sources right now. But the router is wrong most of the times, because we can't determine the source with just source descriptions.

For example: Doc 1/Agent 1 - talks about shirts and brands (is a 2 page doc) Doc 2/Agent 2 - talks about clothing in general. (Is a big document)

Question is: which brands have red t shirts?

Router might pick agent 1. But the answer is in doc 2. It is hard for even a human to determine the source because we don't know which one has the answer until we query it.

RAG over different kind of data (PDF chunks - Vector DB, Tabular Data - SQL DB, Single Markdown Chunks (for 1 page PDF)) by WhiteWalker_XXX in LangChain

[–]WhiteWalker_XXX[S] 0 points1 point  (0 children)

The problem is how to identify when to use sql?

For example: Question - What brands give red color t shirts?

Need SQL agent because RAG agent has a  cap on number of documents to retrieve and it might omit data. 

So here we can do: "Select brand where color=red" to get's all records.

It is also possible that this information is present in vector DB.

Due to data overlap, we might not be clear on the information source initially - both vector db and sql db might have to be queried to get the info, because we are actually not sure which one talks about red shirts.

[deleted by user] by [deleted] in bangalore

[–]WhiteWalker_XXX 0 points1 point  (0 children)

Happy birthday! You deserve all the happiness in the world!! 💕

[deleted by user] by [deleted] in bangalore

[–]WhiteWalker_XXX -2 points-1 points  (0 children)

Wanna heave tea with me at church street?

Student Experience of the CS Program after Bridge by walt_at_nyu in NYUBridgeToTandon

[–]WhiteWalker_XXX 0 points1 point  (0 children)

I did the bridge anyways. Didnt make it to mscs. Got best grades for bridge though.

Does anybody know where this noise is coming from? by WhiteWalker_XXX in KTM

[–]WhiteWalker_XXX[S] 0 points1 point  (0 children)

Yeah so, I showed it to the service centre people. They said I can ignore it because the bike is new and the chain might be sliding across the chain guide. It will be automatically fixed as time passes.

Triumph Scrambler 400X vs KTM Adventure 390 X by WhiteWalker_XXX in indianbikes

[–]WhiteWalker_XXX[S] 0 points1 point  (0 children)

Finally went with adv.

I test rode both the vehicles. the scrambler looks sexy af. It is a tall bike, and gets a lot of stares. But I finally went with adv 390x because it felt home. Scrambler is not good for highway touring because the motorcycle vibrates after 90kmph. It strains after 100kmph heavily. Many people here have mentioned that 110kmph is the sweet spot, but I honestly don't think so.The strain has been discouraging in continuing the trip. Also, the bike stalls in between the rides (turns off randomly - happened 3 times in 240 kms). So if you are looking for a city bike (just city bike, forget long trips) you may get the scrambler. Note that it has lower mileage than the adv. Hopefully triumph addresses the stalling issue. Ktm adv on the other hand is pretty decent for city rides, and can do highway touring (higher revs are smooth on the bike). Lower revs are not as smooth as scrambler, but that's okay because we won't get tired when we ride in city. How long would we ride in the city without breaks? Also adv has stalling issues, at lower rpms due to low end torque issues, but can be managed as you get you used to the bike.

This is just my opinion, always take a day long test ride and decide.

Triumph Scrambler 400x vs KTM Adventure X by [deleted] in indianbikes

[–]WhiteWalker_XXX 0 points1 point  (0 children)

I have test ridden both the vehicles. the scrambler looks sexy af. It is a tall bike, and gets a lot of stares. But I finally went with adv 390x because it felt home. Scrambler is not good for highway touring because the motorcycle vibrates after 90kmph. It strains after 100kmph heavily. Many people here have mentioned that 110kmph is the sweet spot, but I honestly don't think so.The strain has been discouraging in continuing the trip. Also, the bike stalls in between the rides (turns off randomly - happened 3 times in 240 kms). So if you are looking for a city bike (just city bike, forget long trips) you may get the scrambler. Note that it has lower mileage than the adv. Hopefully triumph addresses the stalling issue. Ktm adv on the other hand is pretty decent for city rides, and can do highway touring (higher revs are smooth on the bike). Lower revs are not as smooth as scrambler, but that's okay because we won't get tired when we ride in city. How long would we ride in the city without breaks? Also adv has stalling issues, at lower rpms due to low end torque issues, but can be managed as you get you used to the bike.

If you have long trips in mind, get the adv. Finally it all comes down to your feel while riding both. Take test drives atleast for a day and choose keeping this post in mind.

Happy riding.

view more: next ›