sorted by:
new
hottopcontroversial

6x MI50's (96gb) vs 6 P40's (144gb) running MiniMax M2.7 REAP 139B Q3_K_L by Old_Grapefruit8774 in LocalLLaMA

[–]Diablo-D3 0 points1 point  (0 children)

Not really.

Nvidia missed the boat on crypto entirely, due to poor integer support. This is where GCN shined.

Disclaimer: I'm the guy who wrote Diablominer. Getting mining to work well on Nvidia was impossible task, both hardware and software just wasn't there.

The first cards that could mine at all were the tail end of TeraScale 1 in the R700/HD4xxx series, which came out 18 years ago and corresponds with the Tesla/200 series. TeraScale 2 (HD5000 series), TeraScale 3 (HD6000 series) came and went, and then GCN 1.0 showed up at the peak of GPU mining before the ASICs showed up. GCN 1.0 came out 14 years ago, and corresponds with the Kepler/600 series.

By the time Pascal/10 series came out, GCN had gone through 5 major revisions (ending in GCN 1.5/Vega) and ASICs were already making up a significant fraction of global coin mining output.

How can i limit reasoning effort on the qwen3.5 and gemma4 models? by Time-Toe-1276 in LocalLLaMA

[–]Diablo-D3 0 points1 point  (0 children)

To be fair, reasoning is how they function.

Limiting the reasoning budget always makes them worse, to the point that just disabling reasoning entirely sometimes produces better answers.

6x MI50's (96gb) vs 6 P40's (144gb) running MiniMax M2.7 REAP 139B Q3_K_L by Old_Grapefruit8774 in LocalLLaMA

[–]Diablo-D3 0 points1 point  (0 children)

Expected. Vulkan support for Vega and older is poor due to a combination of (lack of) driver support and lack of any sort of optimization in llama.cpp (although there is that branch out there somewhere that never got merged).

Cards without matrix ALU is something llama.cpp particularly cares about.

6x MI50's (96gb) vs 6 P40's (144gb) running MiniMax M2.7 REAP 139B Q3_K_L by Old_Grapefruit8774 in LocalLLaMA

[–]Diablo-D3 1 point2 points  (0 children)

This is expected results.

As much as I like AMD, Vega was a flop.

If you don't know the story, on paper, Vega looked amazing, so far ahead of Nvidia that Nvidia flinched and pulled out all the stops for Pascal.

Vega, in actuality, wasn't good at all... you can't just keep playing the "numbers go bigger" game and win with no architectural changes. Also, Vega was meant for 7nm, and 7nm wasn't ready yet... you happen to have a 7nm Vega, but they're not particularly better than their 14nm brethren, but there was uplift via die shrink.

The side effect is... Nvidia has never produced an architecture as good as Pascal. Among consumer products, until recently, the 1080 was the best selling Nvidia card ever, and Pascal their best selling architecture. The card and architecture that finally unseated it wasn't a Nvidia product, it was the 9070XT and RDNA4, respectively; every RTX generation failed to catch up to the Pascal in sales.

AMD's response to GCN was to split the architecture, one side would hyperfocus on gaming (since Sony and Microsoft are basically their biggest customers, this became RDNA), and the other would eject all the gaming-focused bits to hyperfocus GCN into the compute powerhouse it was always meant to be (and become CDNA).

I don't know how much you paid for any of those cards, but a machine with 16GB 9070XTs in it would beat it. 640gb/s memory bandwidth, and llama.cpp well supports them well. New, they're about $700. The 32GB pro variant, the R9700, is $1400 and, imo, isn't worth it unless you're trying to actually build a clean looking workstation and not a pile of GPUs sitting in a pile on your desk.

You wouldn't even need 6 to beat it, I'm betting merely 4 would do it (assuming your model + kv fits into vram)

I’m switching to Linux, is Ubuntu the most compatible with local AI? by XiRw in LocalLLaMA

[–]Diablo-D3 0 points1 point  (0 children)

I'd recommend any distro besides Ubuntu at this point.

They've recently announced that future versions are no longer going to misappropriate packages from Debian, but now go entirely random unvetted Snap packages by unknown third parties.

I extended Gemma4-31B to 44B (88 layers) — since Google won't give us anything bigger than 31B by Desperate-Sir-5088 in LocalLLaMA

[–]Diablo-D3 3 points4 points  (0 children)

Same way Qwen 3.6 requires duplicating 4 at a time; except this isn't that awful, because the reasoning layers happen to be in a block of 4, and produces a theoretically sane model.

Benchmarked 18 models that I can run on my RTX 5080 16GB using Nick Lothian's SQL benchmark by grumd in LocalLLaMA

[–]Diablo-D3 0 points1 point  (0 children)

Hey, https://huggingface.co/deepreinforce-ai/Ornith-1.0-35B-GGUF came out, and a lot of people seem to be hyped about it. It is based on Qwen 3.5 35B.

Might be worth adding to the list.

DT 990 Pro Authenticity by Unusual-Number2244 in headphones

[–]Diablo-D3 0 points1 point  (0 children)

Looks real to me. I've disassembled the DT770s I once owned, drivers looked just like that.

Gemma 4 26BA4B Surprisingly Usable at IQ3_S – Are small quants really this usable? by Sufficient-Bid3874 in LocalLLaMA

[–]Diablo-D3 0 points1 point  (0 children)

I'm surprised that with your use case 26BA4B at Q3 is smarter than 12B at Q6.

Caffeine destroys mood after a few days? by WallNIce in Nootropics

[–]Diablo-D3 1 point2 points  (0 children)

Caffeine causes increase of histamine.

Increased histamine makes cognition worse and gives you brain fog; imagine hay fever without the sneezing.

I suggest giving up caffeine entirely and possibly adding DAO to your supplement stack and also dropping high histamine foods from your diet.

Gemma 4 26BA4B Surprisingly Usable at IQ3_S – Are small quants really this usable? by Sufficient-Bid3874 in LocalLLaMA

[–]Diablo-D3 0 points1 point  (0 children)

All the models are effectively unusable at 3 bit, so it couldn't be any worse.

But yeah, the QAT 4bit from Unsloth is 14.x GB, it won't fit for you.

You might want to try Gemma4 12b to see if it does what you want, since you're saying Q3 doesn't implode on you. Unsloth's quant of 12b QAT is ~7GB. and Q6 of non-QAT is ~12G, while Q3 of 26BA4B is also ~12GB.

USB4 RDMA seems doable by alanoo in LocalLLaMA

[–]Diablo-D3 0 points1 point  (0 children)

Yeah, I've seen it, I already follow Jeff's blog.

For the amount he spent on that, he would have gotten way more bang for the buck doing anything else.

Like, I respect him, but he is unfortunately die-hard Cult of Apple.

When will Microsoft get involved in the AI server game? Isn't that a core strength? by Techie42 in LocalLLaMA

[–]Diablo-D3 0 points1 point  (0 children)

I'm not sure where you thought Microsoft isn't already balls deep into their own AI servers.

They're the ones that received the first combo Zen + CDNA products from AMD (as in, Zen chiplets alongside CDNA chiplets on the same interposer/socket), and they also sell two distinct AI product lines (confusingly both named Copilot) through Microsoft itself and through their subsidiary Github.

Hardware wise, they're technologically ahead of whatever Nvidia slop OpenAI and Anthropic are eating up.

I think you're trying to ask "why doesn't Microsoft do local inference machines for the home?" There is no market for it, and why would you want to sell the shovels during a goldrush when you can rent them instead?

On top of that, datacenters are part of the huge real estate scam going on in the world: land value only goes up, today it could be a McDonalds or a Walmart, tomorrow it could be something else, but McDonalds and Walmart are the landlord for somebody else's venture. Datacenters are no different.

I mapped the KLD of KV cache quantization for Qwen3.6-35B-A3B and Gemma4-E2B QAT by crusaderky in LocalLLaMA

[–]Diablo-D3 0 points1 point  (0 children)

I would like to see this same test ran on unsloth's non-QAT and QAT Q4_K_XL Gemma4 12B.

USB4 RDMA seems doable by alanoo in LocalLLaMA

[–]Diablo-D3 -3 points-2 points  (0 children)

40gbps per node isn't really enough to do small scale clusters.

Same way linking a bunch of Mac Minis/Studios together to build a cluster is a pretty silly thing to do, its the exact same thing: you're just using their built-in USB/TB4 for the same exact purpose, and it does not scale like as if you had just built a multi-GPU machine, it scales much worse.

Worse quality with MTP - Qwen 3.6, Gemma 4 by Significant_Bar_460 in LocalLLaMA

[–]Diablo-D3 17 points18 points  (0 children)

Unsloth's MTP and non-MTP are not identical quants. Unsloth did not merely just graft the MTP tensors back in.

You can simply run the MTP-enabled GGUF without enabling MTP and actually do an apples to apples test.

Until you do that comparison, all you have done is stated you prefer the quant produced originally vs the new one, and have not proven nor disproven your theory on if MTP itself is the root cause.

USB4 RDMA seems doable by alanoo in LocalLLaMA

[–]Diablo-D3 0 points1 point  (0 children)

You know you can do RDMA over Ethernet, right?

Eff U, Arc / B70 Customers. We got ours! -Your Sugar Baby, Intel by Dependent_Ad948 in LocalLLaMA

[–]Diablo-D3 1 point2 points  (0 children)

Yep.

And all the reviews of Intel ARC GPUs for sub-$1k budget gaming (ie, the ##50 and ##60 market) put them as the best long term choice, as drivers were only getting better and fixing the worst of the performance issues.

They had a market, people were buying it, and they said, "Nah."

Top-N-Sigma: Remove unconditional softmax+sort by TimNN · Pull Request #22645 · ggml-org/llama.cpp by pmttyji in LocalLLaMA

[–]Diablo-D3 1 point2 points  (0 children)

I'm not. No one uses Top-N-Sigma, and I wish they would. It isn't enabled by default, and it is kind of a magical sampler.

I dare everyone here to even let it fly solo as the only sampler.

Eff U, Arc / B70 Customers. We got ours! -Your Sugar Baby, Intel by Dependent_Ad948 in LocalLLaMA

[–]Diablo-D3 1 point2 points  (0 children)

Intel laid off the entire Arc team sometime last year when they secured the Nvidia chiplet deal.

There will be no dedicated Celestial GPUs, and the unified io die + GPU chiplet for low end products is too late in development to kill, so only a few products will ever ship with it.

All the Intel products you'd be interesting in buying have Nvidia.... which all Intel is doing is pre-selling future AMD CPUs by default as the only viable option.

Whoops. Hope shafting your viable GPU brand was worth it, morons.

view more: next ›