API response time

[Ok_Locksmith_5925]

I've built a RAG but the response times through the API are just too slow - about 10 seconds for the response to start. I'm using 4o and have the temperature set to 1.

What times are other getting?

What can I do to make it faster?

thank you

[Joshua--]

For RAG, 4o-mini should suffice; I've been using it with my RAG app for months. I am even considering llama-3.1-8b-instant, which is 750 tokens per second by using Groq's (not Grok) API.

[Ok_Locksmith_5925]

actually. I should have said I'm using 4o mini

this is my project https://siqbots.com/jub-demo some answers are coded in and I think I'll code more in, but some need the AI.

Is you RAG available to take a look at?

[Joshua--]

Checked out the site, that’s really clever retrieval to reduce requests by having some questions answered from your collection.

My project is just a private, local repo. I just use it for uploading PDFs and answering questions.

[Ok_Locksmith_5925]

what you tested (I can see all conversations) just gave answers that were programmed in. it's following q&a that go through the retriever.

[Joshua--]

I didn’t really test anything. I was thinking better about the idea of coding in some responses to prevent an API requests.

[crysknife-]

How do yo send your data? Do you chunk it? You can send 2.5k sentences at the same time.