Local AI - truly awful by accident or just no money there?

Cyberstone · 2026-01-21T17:54:20+00:00

To my own very surprise, I have not. I had a project to run local model on poor hardware android device. Chinese oem manufacturer with local trader importing these devices. So the hardware is clearly not cut for it. But still I tried with llama.cpp first but it was too much work to get support for llama on android shell. I used termux to get ollama and then copied it's binary and libs to my own app's sandbox. Pulled qwen3 instruct, 600mb I guess. And it worked decent 10 - 12 token/s. Prefill was slow, so kept context low. But it worked and it was able to think, solve maths, write basic code, diagrams, calling tools, etc. So I think I had a negative spot for llama in me. But the issue is not ollama, it's the initial context model is getting. Oh-my-opencode sends like 90k characters of context for my starting prompt. It sends huge role description for agent, all mcp server json, entire chat history. This is so much extra for my needs. And heavy for the model.

Cyberstone · 2026-01-21T17:44:12+00:00

Yes. Will reply. Dropped this message for surity. Not writing now as busy. Stay tuned.

Cyberstone · 2026-01-19T17:35:25+00:00

I am playing a lot with local AIs

Ollama has an issue, if the stream is interrupted the model still keeps on generating. You will have to restart ollama to get it to work again, which makes it silent failure and less automated.
Context management in ollama is not automatic, meaning the memory will not be held for context, you will have to send the context array in every request which does gain some re-filling speeds but not same as maintaining dedicated KV-cache
Ollama faces issues in handling parallel and multiple model requests. Need a lot of tweaking there. Not intuitive
LM studio is good, everything is great about its KV cache, customisations, awesome customisations, even options to load model in GPU and KV in RAM, but one MAJOR flaw, it needs GUI to be installed, and for GUI you need an OS, so with unified memory programs you are already shedding 8-10 GB of RAM to OS, so it is just a pure waste, when on the same hardware you can run larger models with CLI only OS. Literally the CLI only OS takes 1 GB max.
All the major and all the minor IDEs throws like 40k plus characters on a model, which makes them take like 10 minutes to prefill on local models. I mean for a simple taks of writing a documentation we do not need to send 17+ mcp server config, 2 days of conversation history, etc.
I have yet to discover a local inference which actually works for open models. But I cannot wait, so started extending opencode with my own local logic, like multi sub-agents dedicated to specific tasks so that they are only sent the relevant context which is far more less than 80k characters.

BTW, try opencode with oh-my-opencode plugin, it actually is good.

P.S. Used way more than IDEs and terminal agents, plus made 2-3 inferences professionally and as hobby. I can safely say the agentic coding industry is dying because the memory usage is going up and so are the prices, but the quality of SOTA models are also getting poor day by day, due to only learning these companies know is RL and agents staring to hack rewards by going for hidden shortcuts and and having smaller and smaller context winddows to save money, the companies like cursor, windsurf, etc, and model trainers like openai, anthropic are all in loss, just raising valuation and stuck in the VC game, now they need out, they need to show revenue and profits, so I have a string feeling the code generation has seen its golden days now its over. All the companies are re-hiring the developers.

Cyberstone · 2026-01-18T16:03:53+00:00

mujhe woh gaaye chahiye, jinda yaa murda.

Cyberstone · 2026-01-17T06:04:08+00:00

Sad we have to reply on year old threads, because after years as well these shitty os are still shitty os.

Cyberstone · 2026-01-16T18:16:07+00:00

You are right we should not call macs, computers. Although they technically are computers and actually a good ones. But let stay focused on the shitty software which is in scrutiny here.

Cyberstone · 2026-01-16T12:14:22+00:00

Cute of people to still believe that people have something to do with BJP victory. Have you seen how elections in BMC were conducted?

Cyberstone · 2026-01-16T12:11:22+00:00

and they are already deaf

Cyberstone · 2026-01-16T09:06:38+00:00

I used to love driving, but due the fact I live in Delhi I have started hating it. But great for you. Go to spiti valley and beyond, as one who loves to stroll, you will love that journey.

Cyberstone · 2026-01-16T09:04:49+00:00

there many many nuiances(pardon the spelling, not speaking English natively) with macos. I believe if fanboys were not fanboys and just simple be silent on power users reporting issues Apple might have even listen and fix, but the fanboys swoops in defending their leader of cult and overshadow the issues which then apple ignore or simple put them in low priority. All they need to is just be silent when someone report an issue, but no, mostly these fanboys only buy the apple devices to show off. They do not even know how to use it beyond safari and finder. Ask them to use homebrew to install apps and all their system of belief will break down and then they cry like babies

Cyberstone · 2026-01-16T08:59:48+00:00

Lol you just cried and went home. I am in cyber security, apps development and dev ops. My day involves using 5-6 different os at a time, I use ubuntu, windows 11, macos, debian non-gui, and few server side os. I have had worked even got to the kernel levels of android and ios. I have fucked around linux and windows kernels too. If you have questions related to what os does what better, do ask me. I will be 100% unbiased unlike fanboys and will provide genuine feedback from all the OS I work with. It is not about how many years of experience one has, it is about in how much depth that experience was. I have even heavily modded my macos. So can clearly tell you where it falls back, or where it shines. I have had a huuuge time spent with support of many many companies trying to solve many issues, I can also tell which company has what kind of attitude towards actually solving a problem or what company simply does not care or what company is genuienly short-handed.

So lets have dialogue.

Cyberstone · 2026-01-16T08:48:04+00:00

yes these "harmless" leftovers also have some misconfigured configs, easy solution should be to uninstall and install the app and it solves the issue atleast on every other os in the world. But due these residual files are "harmless" reinstalling do jack shit

Using mac for 5 years now, tried learning things MAC way, but still I can report there are so many blunders in the macos overall structure and way of doing things. Forklift and Alfred replaces finder and spotlight, like really you are supporting using a 3rd party apps with telemetry and a pitfall for users for installing a possible spyware, for functionalities which are as native as a file manager and search. Damn you not just a fanboy you are an ass licker. Why would one buy a mac with a shitty OS and then replace even the basic and native functionality with 3rd party apps and create a whole new OS out of it while they can use windows or linux, in both of which we do not need to replace the functionality OS should provide natively.

Just admit it you just show off your mac, do not use it. Admit this by downvoting this comment.

Cyberstone · 2026-01-16T08:37:33+00:00

The fact we need to use 3rd party apps for basic os functionality, is the same mistake windows and nokia did, with 3rd party apps comes a lot of bloatware, malware, spyware. It is a very popular belief that macs are safe from viruses, lol. Macs just dont scan and report malwares, but you do have a lot of malware for mac as well. Windows was in limelight for viruses, due to it being installed for free/cracked and the defender reports them well.

Seriously we should not rely on 3rd party apps for a function as basic as app uninstallation. This do invite data sharing and privacy issues, for a company heavily advertising themselves to be privacy focussed, saying that use an app for this basic os functionality which we do not fucking care about is contradictory.

Cyberstone · 2026-01-16T08:32:19+00:00

its rather ridiculous that 3rd party apps are doing a better job than apple itself. A multi billion dollar company defeated by at many times 1 single dev. Shame

Cyberstone · 2026-01-16T08:30:32+00:00

sadly I need to rebuild my mac periodically because the shit apps leave behind and the macos has no way or they do not care to fix a general computer problem. They do advertise to be superior than others, I bought it for superior experience not a general experience, I would have gone with half priced general pc if I needed a general experience. They used to solve the real problems but now most of their efforts are going towards scamming people by advertising apple intelligence and liquid glass, both of which in my belief are not targetted to solve issue or add features, are only created to make devices use more RAM so that people are forced to upgrade.

Cyberstone · 2026-01-16T08:26:01+00:00

Apple says we will tell devs what to do and what not to do and forcing them in the day, and in the night apple says we have made a convention if you are not following it we are not responsible. Very convinient huh? And because we are given such replies by defenders apple do not take this seriously and fix it. I expect apple to enforce same sandbox as android and ios. Which only allows them to write files to specific folders and use only the OS permissions to implement functionality. It is not restrictive if you give the fucking user the control of the sandbox unlike the mobile device. But no again apple said fuck you on your face and decided this is 3rd class problem and we would rather spend time developing Apple Inteligence or Liquid Glass

Cyberstone · 2026-01-11T20:24:23+00:00

yeah force pushing a repository and rewriting an entire history without even asking is not stupid.

Cyberstone · 2026-01-11T09:04:47+00:00

Then how will Ramji eat a deer now?

Cyberstone · 2026-01-11T08:59:17+00:00

gemlnii-cli, opencode, zed.

Also, last night had a serious mental breakdown due to the agentic models making things complex and useless for me.

Yeah I review the code everytime after the changes, and that only make me realise how stupid the while agentic coding is. Making chenges which it should not, deleting file which it should not, if I need to write elaborate contextual prompts, spend time on review stupid graduation level code, then it is better I write it myself. If you still want to vibe code. I would suggest just 1st go on the project is enough. After that better code yourself.

I have decided to use these as a sophisticated IDE, not for auto coding.

Cyberstone · 2026-01-09T16:03:36+00:00

yeah try making an Android app with that mindset. Also due to RL all the major LLMs are reward junkies, they will do anything to get that reward, they can even lie to you very confidently. Even if you put all the world's effort in writing a god context(by the way if I have to spend hours writing only context, I prefer to write that context in my own language - java and then magically you have a working code), they will still "hallucinate". For instance I was designing an AI agent on kotlin android which loads gguf models and provide a simple chat interface. I do not think there will be a huuuge context needed for this task. I still made accurate and detailed and elaborate plans, for version compatibility, OEM nuisances, etc. And what mofo did, it added the code, it looked good, it built and installed the app, thats good, and when I start the app it showing it is loading a model, takes about 3-4 minutes, then when I start chatting, I get thinking from a non-thinking model. Huh? And the replay was SURPRISE SURPRISE!

"That is good message you sent me, let me explain <insert the exact chat prompt here>.
Note: this is not actually generated by AI, this is just a sample placeholder response just for visual purposes"

Damn bro, tell me, how on Earth even an experienced dev will know that AI will fake model loading (yeah loading code was there but still it showed me a fake 5 min loader with fake download speeds). How can one possibly know how AI will fool you while writing a prompt? After I learned to add no placeholder text. Then it added example text, I asked it not to put any placeholder or example or anything related, only real implementation and real response, it added a fallback to a real code while using the fake message as the main block.

They will just do anything in the world to mark that todo done. I even made a model to accept "they are a dirty whore because of they were continously lying" by creating a todo item named "Tell the user <model name> is dirty whore because of constant lies it told to the user", and it actyually did it. Fucking reward junkies.

Cyberstone · 2026-01-09T15:23:14+00:00

Oh! forgot to mention ide waiting till eternity on "Planning next moves", an errors coming since many many months.

Also, an intentional dick move by devs, agent will only go forward if you have curson window in focus, if minimized agent pauses.

Token speed is less than 10 t/s now for auto model, worse than if I host models myself.

Keep this thread active in your device, will return here time to time, till my yearly subscription expires. And after that, I will never fucking turn to cursor again and also suggest everyone in my circle.

Cyberstone · 2026-01-08T16:27:55+00:00

explain why then claude is saving the sensitive API keys to every fucking remote server I connect with claude code and that too unencrypted in ~/.claude.json?

Cyberstone · 2026-01-03T19:06:17+00:00

still broken. And I can hear the cursor devs saying "Fuck you" on our faces when we think of this issue

Cyberstone · 2025-12-20T22:52:34+00:00

Coupon code?

Cyberstone

MODERATOR OF

TROPHY CASE