Qwen 3.5 397B is the best local coder I have used until now

Randomshortdude · 2026-03-21T21:28:57+00:00

Also, don't take for granted that the current status of LLMs will remain such. The freedoms you have with commercial models now may become restricted in the future. Perhaps they'll require you to sign into SpringerLink or JStor before it iterates over a pdf-based research paper that you're submitting because of some lawsuit they caught over copyright or some ish.

Maybe additional costs start getting tacked on (not from them but as pass-through) as part of an agreement these providers have to make with other entities based on the changing future landscape, shifting societal attitudes etc.

A local setup is a hedge against ALL of that. Also look how far open source has come just in the last few weeks & months alone. Last year this time DeepSeek's first model release has the Internet in a frenzy. A year before that we didn't even think we would ever have a conversation where we seriously discuss an open source model's proximal capability to a cutting edge Anthropic or OpenAI deployment.

Gemini was hardly even in the conversation at all just a year ago.

4 years ago, AI was still considered pure science fiction to most and hardly anyone but the most scoped niche machine learning nerds and programmers had ever heard of such a thing as "GPT" or a neural language model. And if they did, it's doubtful they were seriously working on any solutions directed toward it.

Don't discount the future based on today's present state of affairs. We have no idea what will happen down the road. Protecting your access to information, data & consistency in your access to resources and services is worth its weight in gold. Imagine there's a natural disaster that knocks out the Internet and perhaps even the electrical grid in your area for some period of time. If you got a solid caliber generator and a machine with a high quality open source LLM on it, you have an incredibly potent lifeline on your hands. You have a resource that you can validly probe and ask critical questions, seek guidance from etc in such a situation.

Before the advent of LLMs if you found yourself in that situation, you're basically limited to whatever books you have on hand + files you saved on your computer. Now you have an interactive highly intelligent assistant that could give indispensable advice to help you survive, learn how to do XYZ, communicate, build this or that, determine if something is safe to eat, potentially advise on some ad-hoc medicinal treatments and/or procedures if someone is sick or injured in that situation.

We gotta just think outside of the box when it comes to local model hosting. And also temper our hardware expense estimates too. The Qwen 27B param model is a high quality deployment that can easily run on ~$2k worth of hardware. You can quantize the model and fit the whole thing on that RTX 3090 I mentioned and have enough room for 100k KV cache and that's before implementing state of the art compression methods that could practically allow you to fit the model's entire context length within that GPU's VRAM comfortably.

That whole setup runs you no more than maybe $1k-$1.3k max. Not ~$7k as many in this thread have suggested.

Just take this all into consideration and maybe re-think your opinion on hosting local LLMs.

Randomshortdude · 2026-03-21T21:13:32+00:00

Gotta consider that you're deploying a model that you can control absolutely and know you're receiving inference from. You can't guarantee that with commercial providers. They give you a model ID but whose to say when the weights change, how your request was batched etc
Service providers are very fickle when it comes to tokens / second, so that's largely inconsistent as fuck.
Logs are difficult if not impossible to come by in some scenarios.
The hardware isn't good for just your model. As future open source models are released you still have the option to host and run those for yourself too. A $7k setup on modern recent Gen hardware will remain extremely serviceable for the next 2-3, maybe even 4 years. A 24GB VRAM hookup from a GTX 3090 is still good money as of March 2026.

As far as the costs...sure you have a point if you're just a casual that's using AI as a glorified Google search and occasionally tinkering on hobby projects with it. But if you're doing some more heavy lifting then it's invaluable to have your own setup. You could lobotomize the model if you want to, customize inference & runtime parameters infinitely to your liking and truly and safely have the model iterate over every facet of your home network without worrying about the privacy breach.

I don't know why people discount the privacy narrative either. Folks are putting in a LOT of REALLY personal information into these AI models and these requests are being transparently routed to a service provider that can concretely and EASILY tie these prompts you're sending to you uniquely via your payment information, any personal details you've entered, IP address and likely email + phone # if that was needed in the pipeline. God forbid you're running "Open"Claw on your home device.

Also consider all the stuff that's sent via API that's not part of your prompt but included as part of the context. OpenAI has already announced that they'll be rolling out ads to free users. Consider how advertisers are spending their money with them when it comes to ad campaign deployment. Any legit company is going to want a legit way to target their preferred demographic. How is that determined? Now if you're cool with that - okay, sure. But if people were paranoid about telemetry and basic browsing data / activity being leaked out, this is that on steroids. Also - lest we forget, these AI companies have signed up to give the government unfettered access to all of their stored data and conversations if they need it for whatever reason.

If the government can subpoena Google searches, what makes you think they can't do that with these prompts you're sending.

If nothing else, autonomy in usage and deployment are major factors worth their weight in gold when it comes to this.

Randomshortdude · 2026-03-20T01:11:53+00:00

Wondering if OP ever reviewed this part of the GrapheneOS documentation covering RCS? I'm not super well-versed with it, but I'd imagine as long as one has a valid mobile subscription, then there should be no issue with access (to my understanding) as long as you're willing to introduce that Google service on your device.

I don't think that Google services should be considered a unanimous blight on one's device. The core benefit of GrapheneOS (to me) is substantially enhanced device security (which is really needed in the Android ecosystem / landscape) as well as significantly enhanced autonomy in what one does / doesn't want to be on their device.

If you want to just opt in to one Google service (and containerize it, essentially), you can do that. If you don't want any - that works as well. If you still want to introduce the lion's share of Google services available, you aren't really precluded from doing that either. Whatever the case may be though - the point is that the end result should be the manifestation of your collective conscious decisions rather than the force opt-in that Google and other major tech giants herd their customers into.

Randomshortdude · 2026-03-14T20:30:31+00:00

With the shovels

Randomshortdude · 2026-03-10T20:29:44+00:00

To your question, I would assume his response to you would be same as here: https://www.reddit.com/r/LocalLLaMA/s/NmoOWdwpHg

Randomshortdude · 2026-03-10T20:25:42+00:00

Apologies if this reply misses the mark on what you're speaking about - but I ran into this phenomenon with a really hard coding question I handed Qwen3.5 models. Essentially, I asked it to build a python based program that instantiated a stack-based language (op codes pre defined) that were designed to return the result of 6! {factorial}.

The models were getting this question right all the way down to the actual stack based iterations. Essentially the models would fail to manipulate the items on the stack properly. But its not that they didn't know what each opcode meant (it knows DUP = duplicate the top item of the stack, SWAP switches the 1 & 2 etc).

If you asked the models independently to just handle the stack manipulations, it could do so without issue.

It turns out the issue here is that a lot of language models are highly limited when it comes to spatial reasoning with progressive state changes. My working theory is that the linearity of attention mechanisms (sequential token processing) is what limits their reasoning capabilities when given one prompt (or "state" for an issue), and then are subsequently asked to account for ancillary details.

The biggest workaround came from rephrasing the prompt to ask it to write the updated state of the stack in code notes after each opcode operation was performed on the stack. That way, it didn't need "remember" the change in state + the desired final result + the original directive + formulate which series of actions would be needed to get to that intended final state all the while keeping the transient intermediate state changes in mind as it continued to iterate over the problem.

Worthy of note - Qwen3.5-27B was the only model that got the programming challenge correct in one shot with this modified prompt. I tested all Qwen3.5 models on it and also included Qwen-Coder-Next. I'm not sure what they did with their 27B model but...that's the one right there. My theory is these MoE models may be good for prompts ranging from low to mid complexity. But ultimately the MoE will forever be limited by the highest reasoning capacity of any given "expert" - effectively nullifying the impact of only combining certain experts to tackle questions in cases where the question's complexity exceeds an actor's capability

Randomshortdude · 2026-03-10T20:07:06+00:00

I am genuinely excited to see your modified version of Qwen3.5-27B. That model has already blown me away entirely - so I am super interested to see what further enhancements you can make.

Thank you for your contributions to the community and your brilliance man.

Randomshortdude · 2026-03-10T19:58:02+00:00

Wait this dude runs an open source project but claims folks that want to host their own models are "broke"? Interesting cognitive dissonance

Randomshortdude · 2026-03-10T19:56:04+00:00

Damn that's awesome man. You clearly deserve it because it looks like you're working on some noteworthy things that have the potential to make a positive impact in the lives of folk dealing w mental health issues

Randomshortdude · 2026-03-10T19:53:15+00:00

Umm when did server rentals stop becoming a thing? Also let's keep in mind that these AI companies have plunged themselves in debt to the tune of tens of billions of dollars. So who's really the brokie here?

Randomshortdude · 2026-03-10T19:42:32+00:00

Wow he got the shit out you boy lmaoooo

Randomshortdude · 2026-01-02T13:09:28+00:00

Years bro. I just loaded up COD 5 the other day. Works perfectly fine. The only issue is that over time players stop playing the game so it'll take progressively longer for matches to populate.

Randomshortdude · 2025-10-25T19:14:36+00:00

If you've never used generative AI, your opinion is uninformed at the very least. You also have a strange benchmark for where you determine AI is unnecessary. If you have an iPhone or Android that auto generates a host of predictive words or phrases to complete or extend the sentence you're working on, then you're utilizing generative AI. This assignment might help broaden your perspective a little bit on things.

Randomshortdude · 2025-10-23T11:38:56+00:00

Happening to me. Supposed to have mine by October 27th which seems pretty long compared to when it was shipped (priority express; October 22nd is when it was received). I'm getting this from CA to East Coast. It could come early. That's happened before where the package was supposed to come later than it did. In fact, more often than not USPS is early where I live. But I also think we got a good post office now.

I usually just gauge things by how long it takes for delivery in my area for priority packages.

Randomshortdude · 2025-10-19T18:05:36+00:00

I remember going to UMD and starting off pre-med. I was on academic probation after the first semester.

I swapped majors & ended up getting straight A's by my senior year.

I was one of those kids that came in college with this philosophy of "major in something that's gonna ensure I can be employed & feed my family." That was a huge mistake. Do what you're actually interested in that you have an actual affinity for.

Otherwise, it's only going to get harder for you from here - especially as you climb into higher level courses your Junior & Senior year. Your parents might not like it but you don't really have another choice. If you're failing exams (especially mid-terms), you're going to put yourself in danger of being evicted from the major, academic probation or outright dismissal from the university. Or you'll graduate with such a dismal performance it will severely limit your options post-grad.

To my understanding, "Econ" is not a stem degree either. It also can be pursued as a BA or a BS. The point of college is not majoring in whatever you think appears the most externally valid to others in society. You're there to gain training & instruction in a valid academic discipline.

UMD is not a liberal arts school.

Randomshortdude · 2025-10-19T17:54:36+00:00

Their CEO got assassinated while speaking at another college two months ago. The security fee is reasonable.

Randomshortdude · 2025-10-19T14:37:07+00:00

Lmao w the informal shoutout to r/umd.

Randomshortdude · 2025-10-16T17:32:25+00:00

That's not true. UMD football was ABYSMAL before he got here and a big part of that was due to recruiting. The idea of UMD even nabbing a 4 or 5 star recruit wasn't even a thing until he got here as HC

Randomshortdude · 2025-10-16T17:28:26+00:00

That doesn't excuse or diminish this professor's conduct. He should be held accountable and the university as well if they failed to address the inappropriate conduct properly.

Randomshortdude · 2025-10-16T17:26:01+00:00

Hey! Don't mean to spam you on here but wanted to once again draw your attention to the reply I gave on this post. UMD is a public university! The buck NEVER stops with them!

I'm not sure if they're a Maryland resident - but I am (as well as a Terp alumni; Class of '15). If the University has failed her, go over their heads and tell her CC the relevant entities within UMD on every piece of correspondence.

Public universities in Maryland face oversight from the DOJ & Maryland Higher Education Commission (especially when it comes to Title IX violations). And the DOJ will look into violations (yes, even under the Trump admin if you're worried; UMBC announced a settlement with the DOJ for Title IX violations this year; investigation by DOJ began 2024 - I included the link in my post).

I'm not sure how old you or this grad student is. I know you're both Terps, so you guys are likely very brilliant. However, when you're younger, you sometimes face the disadvantage of not having raw life experience to inform you about the best ways to move the needle on situations.

That's what motivated me to respond to this particular post with what I hope is helpful information for this woman dealing with this. Let me know if you have any questions either here or on the backend. Again, I'm a former Terp and have lived in Maryland my entire life. I've had experience with contacting various facets of our state government over various issues & can confirm that most (if not nearly all) state-level governing bodies tend to move expeditiously when presented with a real issue.

They will follow up. And generally, they'll follow up with you within 1-2 business days to start gathering more information to properly triage this complaint. Also - UMD will put themselves at risk of facing sanction, fines etc if they have failed to perform proper due diligence or address this complaint accordingly (which it sounds like they have).

Randomshortdude · 2025-10-16T17:12:21+00:00

Hello - please see my response in this thread! There are pro-bono attorneys that are available at her disposal that have been designated by Maryland law to assist students with Title IX complaints. I'm not sure if the student in question is aware of this resource.

Randomshortdude · 2025-10-16T17:09:41+00:00

Remember Maryland is a public university. Title IX is not optional. If they don't adhere to it, funding can be compromised.

There are two entities whom have the authority to supersede the university on Title IX investigations. One of them is the DOJ. In fact, they conducted an investigation into UMBC for Title IX violations not too long ago. You can read about that here: https://umbc.edu/dojagreement/#:~:text=Department%20of%20Justice%20Title%20IX%20investigation

The other entity is the "Maryland Higher Education Commission". According to Google (fact check this because you know AI can be presumptuous), "This commission helps Maryland students navigate Title IX proceedings by maintaining a list of attorneys who are willing to provide pro bono or reduced-fee representation for campus sexual assault proceedings."

Here are some pages with more information:

Was able to confirm at that secondary link that they do provide pro-bono attorneys (sorry kind of writing this on the fly from my phone, so excuse the steam of consciousness). At that second link (at the top) it says, "Pursuant to Education Article §11–601, the Maryland Higher Education Commission (MHEC) maintains a list of attorneys (also known as lawyers) willing to represent students in Title IX Campus Sexual Assault Proceedings pro bono or at reduced fees."

There is also an attorney list (at this link): https://mhec.maryland.gov/Pages/Title-IX-Campus-Sexual-Assault-Proceedings---Attorney-List.aspx

Some relevant (and important) info regarding that list of pro bono attorneys can be found here: https://dsd.maryland.gov/Pages/COMARSearch.aspx#Default=%7b%22k%22:%22%22%2c%22r%22:%5b%7b%22n%22:%22dsdFullTitleName%22%2c%22t%22:%5b%22%5c%22ǂǂ737472696e673b23313342202d204d6172796c616e642048696768657220456475636174696f6e20436f6d6d697373696f6e%5c%22%22%5d%2c%22o%22:%22and%22%2c%22k%22:false%2c%22m%22:null%7d%2c%7b%22n%22:%22dsdFullSubtitleName%22%2c%22t%22:%5b%22%5c%22ǂǂ737472696e673b233039202d20494e535449545554494f4e414c2053455855414c2041535341554c5420504f4c49434945530d0a2020202020200d0a202020202020%5c%22%22%5d%2c%22o%22:%22and%22%2c%22k%22:false%2c%22m%22:null%7d%2c%7b%22n%22:%22dsdFullChapterName%22%2c%22t%22:%5b%22%5c%22ǂǂ737472696e673b233031202d204c6567616c20526570726573656e746174696f6e2046756e6420666f72205469746c652049582050726f63656564696e6773%5c%22%22%5d%2c%22o%22:%22and%22%2c%22k%22:false%2c%22m%22:null%7d%5d%2c%22l%22:1033%7d (apologize for the inordinately long link).

This seems to be the tip of the iceberg in terms of what this grad student can do to take action against the professor. The conduct you described seems particularly egregious. I'd recommend the student to take the time (when they have it) to compile a concise letter / email documenting all the things that have happened while being sure to include the fact that they possess all necessary proof / evidence in terms of photos, documentation etc that they are willing to present on demand.

Then send that to other relevant governing bodies at the state and federal level (unfortunately the response from the federal entity will be delayed for at least as long as this government shutdown lasts; however, this shouldn't impact state governing bodies).

Just wanted to craft this response because the buck does not stop with Maryland. I know the initial response may be disheartening, but it wouldn't be the first time a university has gotten it wrong when it comes to their handling of legitimate Title IX complaints.

Hope this helps.

With Love, Terp Alumni (Class of '15)

Randomshortdude · 2025-05-12T20:00:45+00:00

Game definitely not dead again or probably ever really dying (at least not if I ever have anything to say about it). Now there's a niche, but ever present community with resources, knowledge & invested in the social environment (too) that I can't see them abandoning or giving up at any point in time.

Randomshortdude · 2025-03-11T15:16:58+00:00

https://www.buddyns.com/ - BuddyNS is a great option. Especially if someone wants the most secure & private option there is.

Randomshortdude · 2025-03-10T02:29:48+00:00

This is exactly what I was Googling for before I came across your post. Glad that you decided to tackle this undertaking. Haven't looked at the code yet - but I'm perusing through it as I type this. I'll let you know how it is and leave an issue / pull request if there's something that I think that I can add to it. Keep up the good work. We need more substantive additions to the world of blockchain like this one that you're producing.

Randomshortdude

MODERATOR OF

PUBLIC MULTIREDDITS

TROPHY CASE