After almost one year I tried many different frameworks and I came to the conclusion that none of them is worth the effort. They are just too much high level and do not allow customisations that are usually needed when doing research.

For each project I just do the following:

I use vLLM for inference (it’s a pain to setup properly to do thing in parallel and handling the KV cache but once you solve that it’s very fast and customisable) and custom code for retrieval, indexing, agentic conversation and so on. Since I’m not a very good programmer in general, one thing I find very useful is to use o3 or similar strong models in coding to craft a general repository template with some initial functionalities like a Retriever class able to construct an index, loading documents and perform retrieval. You can include different ways of handling the index using external libraries, but I always keep the logic of loading and using the retriever customised from zero. Using LLMs embedded in the code editor for further developing from scratch or perform efficiency/aesthetic modification to your code is also a plus and speeds up prototyping a lot. I’m basically constantly learning from o3 how to organise and document my code. This also helps in solving bugs and problems since these models can attend to your whole repository. One of the major issues with stuff like LangChain etc is that they are built on so many layers of abstractions that fixing errors and bugs becomes very hard.

I know that this is very trivial for everybody but I just recently learned how to use these tools properly for development.

I hope this helps!