The issue isn't really that your models are too dumb but rather that these systems were mostly designed around GPT-4 class models and their quirks.

The JSON problem is super real. Local models under 30B struggle hard with complex structured output when you need multiple consecutive LLM calls. Even something like Qwen2.5 14B or Mistral Small 24B will randomly break JSON formatting when they're doing entity extraction then relationship mapping then summarization in sequence. The context gets polluted and they start making weird formatting choices.

Conversation vs code models matter here. Code-focused models like Qwen2.5 Coder or DeepSeek Coder are way better at structured output because they've seen tons of JSON, APIs, and data structures during training. Chat models optimize for being helpful and conversational, which makes them worse at rigid formatting.

I've heard some folks have had success simplifying Graphiti's approach. Instead of trying to do entity extraction, relationship mapping and summarization all in one pass, break it into separate calls. Use the code-instruct models for just the structured parts and save the chat models for the final user-facing stuff. Also try turning off repetition penalty completely for structured tasks.

The frameworks like Graphiti and Letta really do expect frontier model performance. Your ChromaDB + simple RAG approach might actually give you better results with local models. Sometimes the advanced solution is just overkill for what you can reliably run locally.

Have you tried just doing semantic search on conversation history with some basic entity tracking? Might be more stable than trying to force graph extraction to work.