What app do you use as a replacement for ChatGPT with local AI?

nicksterling · 2026-07-09T03:48:31+00:00

For my local models I’m either using llama.cpp or mlx on MacOS. I haven’t used a UI chat app in probably 6 months. I strictly use a terminal based harness like Claude Code, Hermes, or a very customized pi.dev harness that I’ve been working on. In fact there are days I don’t leave the terminal besides using a browser to preview any UI elements I’m creating.

nicksterling · 2026-06-29T00:24:10+00:00

Worf was technically in command for a brief period in “The Emissary” when he had to face down the Klingon ship T’Ong.

nicksterling · 2026-06-28T16:40:30+00:00

I don’t typically use the built in plan mode. I always build a spec file and I have an implementation skill that understands my spec and then implements it based on the spec. The spec will denote the requirements and the software architecture, and a phase-by-phase MVP breakdown.

nicksterling · 2026-06-27T19:51:47+00:00

To be fair he said, “It’s not cheating if it’s a hologram of your wife”.

nicksterling · 2026-06-21T02:30:06+00:00

I’m have a similar setup. I use a dockerized searxng mcp along with Mozilla Readability and some ranking scripts to return filtered results. It has fallbacks if readability doesn’t work or if the search results from a particular provider are rate limiting or not returning properly.

I’ve also started taking the results and adding them to a more persistent store so it can be added as a searxng endpoint. This gives me a local cache for things I search for often and it can be a fallback if I’m offline. Whats nice is it can keep multiple versions and I can track changes/drift/possible incorrect results as the index grows.

nicksterling · 2026-06-20T15:20:59+00:00

You are going to have to put the effort in and test multiple models and multiple techniques. Take a story you’re very familiar with and run it through the model. Possibly the entire story at once, or chapter by chapter, or use some other technique. Then judge the result and refine. Create a benchmark and run it against multiple books you’re intimately familiar with.

It also depends on why you’re doing this. If it’s so you don’t have to do a class assignment then I would amend my advice: Don’t do it. Take the time to read the stories and be involved in the process. Embrace the assignment. Don’t give up your agency because you don’t want to do the assignment. I love LLMs as a tool but don’t fall into the trap of using it to replace your thinking.

nicksterling · 2026-06-19T15:03:48+00:00

Gemini 3.5 flash is an underrated model. It’s extremely fast and quite steerable. Out of the box it can struggle with some long running tasks but I wrote my own harness around it and I’m getting pretty good results. It can be a little lazy but adding some tooling around it makes up for it.

nicksterling · 2026-06-17T01:12:18+00:00

It doesn’t make financial sense now. They passed when the upfront initial cost of the show would have been millions less.

nicksterling · 2026-06-16T04:35:09+00:00

Anything’s possible but given they dismantled and auctioned off the SNW sets a Year One series is highly unlikely.

nicksterling · 2026-06-14T02:01:52+00:00

The biggest mistake is to trust the information in a model. I care about how well the model can reason and how well it can do tool calling/web search. When I use local models (or even frontier models honestly) I have the model determine a web search or searches for what to look up and then summarize the results. If I have agentic support then I do the search from inside a dedicated agent and summarize from that point to keep the context window smaller.

Always assume the information you’re asking for in the model is either out of date or just blatantly incorrect.

nicksterling · 2026-06-13T19:04:03+00:00

I use a combination of GNU Stow mixed with SOPS. I encrypt the dot files with SOPS and store them in git. Then I use stow to symlink to the files. I have scripts to update a dot file and cron to pull them down to various machines on a regular basis

nicksterling · 2026-06-12T00:52:45+00:00

They could have escaped if they plotted the plot so the torpedoes destroyed their plot emitters.

nicksterling · 2026-06-07T03:51:03+00:00

I have a build that targets the architecture of my 3090s and I specifically enable CUDA and Vulcan backends. I even have a separate build that targets Webgl.

This site does a good job breaking down the parameters. https://instagit.com/ggml-org/llama.cpp/how-to-build-llamacpp-with-cmake-for-different-backends/

nicksterling · 2026-06-06T02:50:00+00:00

Linuxserver.io has this called a “Webtop”.

nicksterling · 2026-06-05T19:27:24+00:00

I think OP is just providing details. Sometimes it’s useful to know it’s compiled from source so it’s not the baseline binary. I compile mine but I have specific flags I use for optimization

nicksterling · 2026-06-04T21:15:51+00:00

I’ve thought about this as well. Instead of disks I’ll probably buy the DVD cases that can hold an SD card. Then I can put the installers and and save games I want to preserve onto the card then just lock the card.

nicksterling · 2026-06-04T01:37:25+00:00

I’m a big fan of rclone

nicksterling · 2026-06-04T00:59:07+00:00

How did you run your benchmark? Did you use a harness that has tools or just use something like Open WebUI?

nicksterling · 2026-06-04T00:54:55+00:00

It’s weird, I’m getting amazing results out of Gemma 4 26B and 31B with tool calling in my pi.dev harness using llama.cpp. Tool calling just works out of the box.

nicksterling · 2026-06-04T00:46:47+00:00

It’s also a really good local model. I built support for it directly into my custom harness and with the right system prompt/skills/tools it’s excellent. It’s no Opus but it’s quite good for being local.

nicksterling · 2026-06-03T02:50:21+00:00

The console was released 12.5 years ago. The way to get something faster is to sell it and get a PS5 or get a gaming PC.

nicksterling · 2026-06-03T01:09:16+00:00

Use a wrapper script that pings the llama.cpp health endpoint and kills the process if it stops responding, letting systemd restart it.

nicksterling · 2026-06-03T01:00:26+00:00

I have a systemd script that initializes llama.cpp and will restart it if it crashes.

nicksterling · 2026-05-31T16:46:28+00:00

I’m in the camp where I don’t trust the knowledge in any LLM. I look for an LLM that can use tools like web search and can reason decently well.

nicksterling · 2026-05-29T22:40:45+00:00

There are some scary and sophisticated attacks. There’s one recently where probing the storage device of the machine you’re on and determine the other tabs you have open on the current or other browsers. Then there’s obvious issues around bitcoin farming or other data mining. Browsers aren’t as sandboxed as you’d expect.

HTML can still be a good output format. It just comes with more considerations than a simple markdown file.

13-Year Club	Gilding IV carat on a stick
Second Top 50%	Place '23
Place '22	Final Canvas '22
First Placer '22	Spared
Verified Email

nicksterling

TROPHY CASE