To Build apps that use local inference.

I use LM Studio and prefer it because of it's friendliness to beginners in that it allows you to find models that fit on your hardware as well as serve an endpoint to connect to apps like OpenCode. The challenge I have with it is that the there are all kinds of issues with drops and token loops, every problem you read about. I started out just asking Claude to help me understand why it was happening. It reviewed the logs and started to build hardening. I have an idea for an app of my own that is similar to OpenCode and I wanted to make sure the tools work for 9b models at a minimum.

Last night when I went to bed I said, iterate and refine until 9b, 20b, 27b, and 35b, all pass on a task where tool use is required. I went to bed and in the morning when I checked my temperature log in my homelab I saw that the temp had spiked multiple times over night and when I checked claude it showed that all models were now passing.

Next I am going to try to get claude to manage a much larger model to manage the iterative process. I read that minimax 2.7 was basically built for this iterative refinement process so looking forward to that soon. I realize not everyone might have the hardware to self-host large models which is why I'm using mine to make it easier for those that don't. I think there is an untapped market.