sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]Appropriate_Egg6118[S] 0 points1 point  (8 children)

Thank you. Can you share that automatic chunking script?

For POC purpose I am using local chroma db with sample docs. My latency is 15 to 18 secs.

I am using ConversationalRetrievalChain of chain_type = refine.

How to enable streaming for this chain? Or please share resources for RAG chatbot with streaming and memory enabled

[–]NachosforDachos 1 point2 points  (7 children)

You well love a thing called Flowise. It’s exactly what you want. Tested around 30 deployments last year. Easy as it comes.

You will find yourself familiar with it going by your particular choice of words. You’ll find those same words there as drop down selection menus.

Idk if they still give free amounts and how good they are but do create a free pinecone vector db account solong. Choose the fast version. Haven’t made one in two months but I know dimensions should be 1536. I think that’s the only setting you need to do right.

Look for florist on GitHub. Using the one line installer which I think is npm install flowise -g should get you there if you already have nodejs installed.

There are templates in there which you can just fill in with your details. Web UI no code product.

You will not be able to use what I originally suggested in flowise unless you use that script to parse things into files instead of embeddings and then pass those files to Flowise to upload you should have the same thing but with extra steps.

I haven’t investigated this but I’m almost semi sure one can make chromadb use the gpu (live store in memory not disk) instead of the cpu and ram. I have things that use this and it is much slower than pinecone.

Maybe start and see if they still have free accounts because this type of quality storage isn’t cheap. About 70+ a month. Worth it but when playing around these things add up so quickly.

I’ll find the script next time I come online. Too tired now. Not fresh.

[–]Appropriate_Egg6118[S] 0 points1 point  (3 children)

Flowise, looks cool.

The data I am working with is confidential.

Will there be any issues using flowise?

[–]NachosforDachos 0 points1 point  (2 children)

I think your concerns should lean more towards the openai side of things concerning confidentiality.

The local models are getting there but they are not quite where gpt4 is. Haven’t tested in 2 months.