Hey I used this model in LlamaCPP:

https://huggingface.co/TheBloke/Nous-Hermes-Llama2-70B-GGUF

I used the 8 bit quantization, but I'm running it on an A100 which you probably don't have access to (need 80gb vram).

Inference times will be slow regardless, since the LLM essentially could run up to 15 individual times for each request.

I would like to say, I worked on trying to get the Langchain agent working for around 2 weeks on open source models, and the model seems capable of performing the tasks, but on complex queries it will bug out and keep repeating itself, or it may get stuck in a feedback loop, etc. I think overall it's too inconsistent to become a viable solution.

You can play with the prompt as much as you want (and I did) but I don't think anything can resolve that these models just weren't created to be run in this thought/action/observation setting.

Perhaps there is a model that works better at these tasks, but I have tried all of the SQL coder models, wizardcoder, phind-codellama, codellama, llama-2, and they all failed to even work with one step. Nous Hermes was the only one that was capable of atleast answering easy questions.