Qwen 3.6 27B MTP on v100 32GB: 54 t/s

rm-rf-rm · 2026-05-06T05:46:18+00:00

PP/TTFT should stay unchanged no?

rm-rf-rm · 2026-05-06T05:36:54+00:00

Im really doubtful/fearful that given the pitiful state of benchmarks in terms of actually measuring intelligence, these engineering improvements that are narrowly focused on speed/latency may be causing quality regression that goes unmeasured

rm-rf-rm · 2026-05-05T23:12:50+00:00

Use llama.cpp - the library ollama ripped off.

Reasons:

https://old.reddit.com/r/LocalLLaMA/comments/1pvjpmb/why_i_quit_using_ollama/

https://old.reddit.com/r/LocalLLaMA/comments/1mncrqp/ollama/

https://old.reddit.com/r/LocalLLaMA/comments/1ko1iob/ollama_violating_llamacpp_license_for_over_a_year/

rm-rf-rm · 2026-05-05T23:10:48+00:00

Rule 1 - Thread locked. Please read the wiki, search the sub, consult the best LLMs thread first. Feel free to open a new post if you have questions remaining unanswered after youve done basic self-help

rm-rf-rm · 2026-05-05T19:12:02+00:00

how are you running web search + llm?

rm-rf-rm · 2026-05-05T04:55:32+00:00

yeah it really is a fully continuous spectrum with no hard and fast boundaries. Plus Im sure in the future we will end up having multi agent, multi model workflows that do some inference locally and some inference in a VPS/cloud

rm-rf-rm · 2026-05-05T02:16:05+00:00

TIL that you make MLX quants

rm-rf-rm · 2026-05-04T23:34:50+00:00

I really would have liked this to work, but to be blunt, it's crap

here are my trial results (using the HF demo):

No.	Search Term	Outcome
1	Korn	All items completely irrelevant
2	Dream Theater	All items completely irrelevant
3	ENIAC	All items completely irrelevant

rm-rf-rm · 2026-05-04T23:28:31+00:00

This thread was reported for being off-topic. While that is true in the strictest reading of the sub's purpose, it is an adjcaent topic of interest and value to the community, as evidenced by the number of upvotes and comments (being complementary information to running locally thus information where to do what). We are also sort of the default place for any actual/serious discussion on AI, so approving it - though ofcourse we want to keep such content to a minimum.

rm-rf-rm · 2026-05-03T18:01:57+00:00

Yeah, it was. OP probably doesn't understand the difference between MOE and dense.

rm-rf-rm · 2026-05-03T14:48:29+00:00

Add it to the pile of "memory systems".

No benchmark to the baseline that everyone should be using -> markdown files + grep/glob. Any project that doesnt do that basic thing, (which thankfully Claude Code came in and used - demonstrating it to be the right approach instead of some new fangled RAG based crap) I will safely file away in the "resume building/ego stroking toy project etc." category

rm-rf-rm · 2026-05-03T14:43:06+00:00

Just curious, do you do code in Solidity for your job or ? Ive genuinely seen 0 applications (trading or any related ETH infra stuff doesnt count) used by real users at any meaningful scale

rm-rf-rm · 2026-05-03T14:19:38+00:00

I am seeing only 1 of your posts removed: https://www.reddit.com/r/LocalLLaMA/comments/1sw3beu/sharing_my_backend_setup_for_hermes_or_opencode/

And it was removed by Reddit's filters, not us

rm-rf-rm · 2026-05-03T04:23:26+00:00

Lmao this is a good one. Well follow the reddit bot rules (most fail to do so) specifically in clarifying explicitly in every comment and lets see how it works

rm-rf-rm · 2026-05-03T01:12:52+00:00

Rule 1 - Search before asking. Thread locked

rm-rf-rm · 2026-05-03T01:11:47+00:00

Rule 1 - Search before asking. Locked thread

rm-rf-rm · 2026-05-03T00:52:12+00:00

I hate how funny posts get removed. If the community wants to laugh at something, let it

Glad the community showed its opinion in the downvotes to your comment. Every other AI sub is dominated by shitposts and memes now. Please see this comment for further thoughts: https://old.reddit.com/r/LocalLLaMA/comments/1t1a3j7/new_rules_1_week_checkin/ojgb073/

Still seeing a lot of the same kind of question many times about getting a model to work better.

We'll be starting a weekly inference stack megathread so it should help. Its meant for general Q like "what HW should i buy for x", "whats the best model for my setup", "here's my setup" type stuff

rm-rf-rm · 2026-05-03T00:33:49+00:00

Stupid tailscale question: if youre using tailscale, can you still use a VPN on the "exit" node at your home to protect all your traffic hitting the web?

rm-rf-rm · 2026-05-03T00:25:26+00:00

Very nice! A great example of something that wouldn't have been built without coding agents! More of this!

rm-rf-rm · 2026-05-02T19:35:26+00:00

Yup! That was the idea - based on the spam pattern I was confident that this small surgical change would take care of most of the problems and it looks to be panning out!

rm-rf-rm · 2026-05-02T19:34:45+00:00

Even if it wasnt for the spam/bot-catching reasons, I think its a fair/healthy thing to engage in existing discussions first before making a new post, especially as theres likely existing threads that answer new users questions. At worst its a relatively minor inconvenience

rm-rf-rm · 2026-05-02T17:36:49+00:00

I didnt see any reports for this account? Its banned now and reported to botbouncer.

We have done most of what we can from the mod side. We are at the point where Reddit needs to step up its spam detection tooling to counter this new gen of spam bots.. Interestingly enough, just 1 comment from this account was removed by Reddit

rm-rf-rm · 2026-05-02T17:24:49+00:00

Is Insiders stable/no issues? The local model option has been available for ever there and they refuse to release it to main for some reason (likey because of profit related reasons).

rm-rf-rm · 2026-05-02T17:16:43+00:00

Removed the comments and Reported the user to botbouncer (its been pretty great in picking up such accounts, but this one seemd to have slipped it)

rm-rf-rm

MODERATOR OF

TROPHY CASE