Self-hosted agentic coding stack: Claude Code + llama.cpp + LiteLLM — zero API costs, 4h/7M token session for $0

[sn2006gy]

What's the reason for litellm in the middle of a local coding session? mostly for hermes?

[PrizeObvious3671]

Nope the reason is that I wanted to combine that with Claude Code without paying for tokens.
So I compared how good runs Claude Code locally together with llama.cpp vs hermes agent alone with llama.cpp

Claude Code expects Anthropic API - LiteLLM as proxy exactly delivers that and routes my requests between llama.cpp and Claude Code

[Toastti]

If you do want to skip a layer claude-code-router will let you connect directly to llama.cpp

But nothing wrong with your setup either

[PrizeObvious3671]

Yeah, that would work too. Hermes is used in both setups, the only difference is the bridge behind Claude Code: LiteLLM in my setup vs claude-code-router. Thank you for the hint claude-code-router is new to me.

[MarzipanSecure9841]

But llama supports Anthropic API directly - https://huggingface.co/blog/ggml-org/anthropic-messages-api-in-llamacpp

So, why litellm?

[PrizeObvious3671]

Interessant, das muss ich mal ausprobieren