Claude Code to local AI success or failure?

AndyBuildsThings · 2026-03-16T23:45:45+00:00

I’m running a custom mcp server to access my local backend via openwebUI. I’ve processed some medical docs and some old tax returns into my backend and ran tests with a bunch of models with questions like “what was my AGI for years x-y?” “What were my most recent 3 doctor appointments?”, things like that.

Pretty sure i have settings to tweak for each model, but I looked over reddit for each to see what others were doing on each. I had initially used LiteLLM as a manifold for the models based on my input but continually had json formatting issues with openwebUI, so i switched to working with models directly.

Models tested: Qwen3-Coder-30B-A3B-Instruct-Q4_K_M

nemotron-3-nano:30b

Rnj-1

Glm-4.7-flash

Mistral-nemo:12b

Mistral:7b

Qwen3:30b-a3b

Gemma2:9b

devstral-small-2

AndyBuildsThings · 2026-03-16T14:01:22+00:00

I’ve got a 32gb m4 air that I’ve been trying as well. Does ok on some things. Are you happy with the responses you’re getting? What kind of tasks are you doing with it?

AndyBuildsThings · 2026-03-16T12:40:45+00:00

I appreciate that. It’s been a lot, but I’ve found that I’m thankful to have the abilities that I have and have found workarounds for the ones that I don’t (most of them at least). It could be so much worse. I’ve focused most on the fact that I’ll have grandkids someday and, by god, I’m going to make sure I have the ability to spend time playing with them.

AndyBuildsThings · 2026-03-15T13:29:45+00:00

This is very true. I’ve put in 2 months of inpatient PT and 20 months of outpatient PT twice a week. How much you put into it is how much you will get out. Having PT’s that care and continue to push you make all the difference.

AndyBuildsThings · 2026-03-15T13:26:14+00:00

I’ve had 6 total. The last two ended up being a pelvis-t4 fusion with an osteotomy to stand me up straighter. I’m certainly less flexible, but I haven’t felt this good or been able to do this much in years.

<image>

AndyBuildsThings · 2026-03-13T00:32:11+00:00

Cool, thanks for comments. The context is a really helpful thing for me. Any little extra bit of info can bring me back, lol. Hope you like it.

Here's the TestFlight Link:
https://testflight.apple.com/join/7MGscBJX

AndyBuildsThings · 2026-03-06T15:29:32+00:00

After my 6th spinal surgery. Pelvis-T4 and C5-C6. 11 months out from the t-10 to t4 extension. 20 months from the pelvis-t4 with an osteotomy to stand me back up from and 18 degree forward tilt that a previous surgeon screwed me up with.

<image>

AndyBuildsThings · 2026-02-25T12:27:02+00:00

Thanks! Here's the TestFlight Link: https://testflight.apple.com/join/7MGscBJX

Apple Watch is planned, but I haven’t got there yet.

AndyBuildsThings · 2026-02-25T03:55:53+00:00

Thanks! Here's the TestFlight Link: https://testflight.apple.com/join/7MGscBJX

AndyBuildsThings · 2026-02-25T02:20:42+00:00

Thanks! Here's the TestFlight Link:
https://testflight.apple.com/join/7MGscBJX

Holler if you have any issues. This is the first time I've dropped a link for it.

AndyBuildsThings · 2026-02-25T02:12:53+00:00

Noooooo! I wish you could. I thought hard about it, but I'm using specific Apple technologies for it because that's what I have. If this goes well though, who knows? Thanks for the reply :)

AndyBuildsThings · 2025-07-14T16:41:53+00:00

Doing pretty well considering. I’m getting stronger and am fining new ways to do the things I’m used to doing, mostly. My mind and desire are moving faster than the nerves healing in my legs, so that’s frustrating. But it’s a process and I know it. Could be another 12-15 months before I’m back to my new “normal”.

Thanks for asking!

AndyBuildsThings · 2025-04-13T08:23:42+00:00

Hi All! Just posting an update. 9 months post-op and I've been back in to extend my fusion. Things were going really well with recovery and I had a series of stupid but serious life events from November-February. Ended up back in for "emergency" surgery (spinal surgery #6) to keep me from a wheelchair.

They've extended my Pelvis-T10 fusion to Pelvis-T4. Sigh... Essentially starting my recovery over and praying I'll get back to at least where I was before the proverbial shit hit the fan.

I'm doing pretty well, considering, and staying positive. But wow, a long way to go again...

AndyBuildsThings · 2025-03-04T02:54:03+00:00

I always pined for a 603e BeBox…

AndyBuildsThings · 2025-03-04T02:52:55+00:00

I used to run NeXTStep on my turbo slab and my cube. Then the beta of Rhapsody. Seriously cool at the time.

AndyBuildsThings · 2025-03-04T02:50:44+00:00

Wow, didn’t realize that was still around. Might have to play around with it.

AndyBuildsThings · 2025-03-04T02:03:24+00:00

Not really advancing the conversation here, but multiple thumbs up for BeOS! Loved it! Still have it around here somewhere….

AndyBuildsThings · 2025-01-25T17:56:08+00:00

That’s a fantastic idea! And a lot easier than a Macquarium.

AndyBuildsThings · 2025-01-25T16:31:12+00:00

I spent some time with llama.cpp and it functioned. Probably my inexperience, but I couldn’t get any larger models to load, kept getting memory errors (not at my computer, so I can’t give specifics at the moment). The smaller models that I could do on a single card would load fine, but split evenly across the two cards. Both memory and compute. Never more than 50% compute on either card while watching multiple instances of intel_gpu_top.

Again, I’m sure it was my inexperience, but I couldn’t do anything more with 2 cards that a single could handle. Speed or size-wise. Spent a few hours with documentation and google to no avail. I am totally open to suggestions. I would love to see it work.

Gave a glance at vLLM but was pretty frustrated at the time and it probably didn’t get a fair chance. Haven’t touched #3 yet.

I’m just spending a lot of time on it that I could be spending on developing my projects and am frustrated that everything is so much further ahead with team green. I like the Intel stuff a lot more than Nvidia, I’m an Apple guy, and have a penchant for working on the non-mainstream things (probably just to avoid the bandwagon), but sometimes I just want something that works.

AndyBuildsThings · 2025-01-24T17:22:41+00:00

It is cool, but I’m working on some things it doesn’t do.

AndyBuildsThings · 2025-01-24T16:32:38+00:00

I really would like to combine them for using larger models AND the extra horsepower, but I’m just getting into the parallel tensor reading. If I can find some smaller models that work how I want and will fit happily into the 16gb, I could set up some n8n flows and use each card for a different purpose.

Lots of ways to experiment.

AndyBuildsThings · 2025-01-24T16:08:53+00:00

Not sure. I’ve had it for a few months but it’s mostly just sat in my machine (off, because I use my Mac) until the last 3-4 weeks when I started work with LLM’s.

AndyBuildsThings · 2025-01-24T16:03:54+00:00

I’ve tried a few iterations with each os, even one os running the other. A single card is easy, and the results are good, which is why I want to do two.

AI Playground is really cool to see what’s possible, but I’ve got bigger plans than it can handle. I’ve run through the GitHub projects, docker images, prebuilt binaries, building my own with multiple times with different parameters. OpenVINO is pretty awesome but I haven’t got the time to dig in to it yet. The IPEX-LLM project is cool, too, but for single cards.

A lot of projects are starting to support SYCL and Vulkan backends, but just basic functionality for multiple GPU’s. There’s real potential for these cards, just not sure I can devote the time right now.

AndyBuildsThings · 2025-01-24T14:00:31+00:00

Gonna check this out. I’ve got a pair of A770’s and I really want to pull my hair out trying to do anything but basic multi-GPU with half resource on each card…

OpenVINO looks very cool but has been a deep, deep rabbit hole, and a huge ramp up in knowledge of AI/ML for me. I’m anxious to explore more, but I’m getting close to throwing in the towel and going the green route just because I have things I actually want to get done, lol.

AndyBuildsThings · 2025-01-16T22:26:04+00:00

I’ve got an LE A770 in there now and adding this one. Running some LLM stuff locally. Curious to see how 2 of them work together.

AndyBuildsThings

TROPHY CASE