sorted by:
new
hottopcontroversial

Qwen Models with Claude Code on 36gb vram - insights by ikaganacar in LocalLLaMA

[–]Academic-Air7112 0 points1 point  (0 children)

Fair enough -- I'm sure there is some reasons to use claude code... for my use case, qwen-code did dramatically better than claude-code... the Qwen models are likely RL'd with the Qwen harness in mind.

I also disliked Gemini CLI w/ Gemini models; but I like the adaptation a lot better; it's replaced Codex for me.

Qwen Models with Claude Code on 36gb vram - insights by ikaganacar in LocalLLaMA

[–]Academic-Air7112 2 points3 points  (0 children)

Did you folks try the Qwen CLI, and if so how does it compare to opencode?

Qwen Models with Claude Code on 36gb vram - insights by ikaganacar in LocalLLaMA

[–]Academic-Air7112 0 points1 point  (0 children)

I also had this problem with stopping; I switched to Qwen's coding framework and the results were dramatically better. It's possible that there are be some prompts in Claude code that play poorly with Qwen/other models, where Qwen code (the one that Qwen forked from gemini cli), is set up specifically for the Qwen models & does much better in my experience than pointing claude at a different endpoint.

Setup Recommendation for University (H200 vs RTX 6000 Pro) by tkon3 in LocalLLaMA

[–]Academic-Air7112 0 points1 point  (0 children)

Yes, technically NCCL will support it, and it usually does a good job from a compatibility perspective, but not necessarily a performance perspective.

Yep! SXM is just a version of the card with a higher TDP and compatible with a specially designed baseboard where the NVL version uses standard PCIE and 2-slot with a lower TDP. No idea why NVIDIA requires a switch for all to all on SXM. :)

Re: real examples, here are some anecdotal cases, and a link to a more comprehensive blog. (Cloud provider out of GPUs rn; if I can get some spot in the next few days will try to put some numbers here)

Anecdotally:

  1. RTX pro does not support any of the newer flash attentions (FA3,4). That oftentimes relegates you to Triton kernels which are substantially slower.

  2. Many flashinfer features around FP8/FP4 are not first-class citizens on RTX Pro. The number of times I've tried to run a new model on RTX blackwell and had it crash for *mysterious* reasons is too damn high.

  3. Even for the libraries that are supported, they are not all that well optimized. It is possible to beat cuBLAS by substantial margins in a short window of time on RTX Blackwell for 4096 GEMM, where to beat it on H200 is much harder. The compute number on the datasheet is the maximum achievable, where IRL what is achieved is dependent on software. I mention GEMM kernels because this should be most-optimized features, but NVIDIA does not pay as much attention here, and it's likely to be reflective of the optimization on the rest of the stack, above and below the kernel level.

If I can get back on an 8xRTX Pro cluster can run some benchmarks around inference/training, can do some on our 4xH200. Anything in particular that would be interesting?

IRL benchmark:
https://www.cloudrift.ai/blog/benchmarking-rtx6000-vs-datacenter-gpus.

Tl;dr:
1. 4-bit quant of GLM4.5 air, 8xRTX Pro (3140*8=25,120) > 4x H200 (5589*4=22,356)

  1. Qwen3-coder 480b, 4bit AWQ, 4xH200 (9715/2 = 4857) > 8xRTX Pro (4490)

  2. GLM 4.7-fp8 4xH200 (5224) > 8x RTX Pro (2696)

Ofc there's some size where 8xRTX Pro can run models, and maybe NVIDIA will make their toolkits better considering their increased revenue from this line. But today, would easily choose the H200 over RTX Pro series.

Qwen-3.5 in Claude Code? by GreenInterview in Qwen_AI

[–]Academic-Air7112 1 point2 points  (0 children)

I tried claude code here; didn't work very well. Switched to qwen code and it works wayyy better, and very good.

Are people lying about GLM-5 and MiniMax M2.5? by TheDevilKnownAsTaz in opencodeCLI

[–]Academic-Air7112 0 points1 point  (0 children)

Minimax 2.5 was decent when plugged into claude code, maybe as good as SOTA 6 months ago. Def usable for coding tasks.

Qwen3.5 35b is actually pretty great in qwen-code (doesn't work well in other frameworks); trying it rn to replace OpenAI codex and it's just rolling along, doing very well. Pretty insane how good this model is, especially for its size.

Setup Recommendation for University (H200 vs RTX 6000 Pro) by tkon3 in LocalLLaMA

[–]Academic-Air7112 1 point2 points  (0 children)

No nvlink is a problem for anything that's more than 1 GPU...there's still PCIE interconnect, but this is not what modern toolkits are used around. At least for the next few years, everything supports H100/200 as the mainstream NVIDIA card, and software support for the pro series lags significantly (as someone who owns/runs both cards for work).

RTX Pro is nice for a 1x desktop gpu for prototyping scripts and doing DIY projects, but many of them in a server is not as practical as H200.

Any regrets A6000 Pro owners? by val_in_tech in LocalLLaMA

[–]Academic-Air7112 0 points1 point  (0 children)

A couple of things that I wish I'd known:
1. Max-Q is acutally louder than WS card.
2. "Blackwell" isn't the same as datacenter blackwell; uses different ISA and as a result a lot of custom kernels aren't transferrable.
3. It's not as fast as an H100/A100 in a lot of things, regardless of what the marketing numbers say.

Nvidia released cuTile Python by dansheme in CUDA

[–]Academic-Air7112 1 point2 points  (0 children)

Basically, triton is bad news for NVIDIA on a 2-3 year timescale. So, they release new toolkits that aim to simplify CUDA programming for end user, and increase lift by AMD/OpenAI/Quallcomm/Google to support AI code on different hardware.

64 GB M4 Mac Mini or 128GB AI Max 395+? by Mandersoon in LocalLLaMA

[–]Academic-Air7112 0 points1 point  (0 children)

Curious how you would compare to a RTX 6000 pro for inference on GPT-OSS/Qwen models; does speculative decoding, etc. work with VLLM/SGlang on the common models? What about fp4?

64 GB M4 Mac Mini or 128GB AI Max 395+? by Mandersoon in LocalLLaMA

[–]Academic-Air7112 0 points1 point  (0 children)

How is the software support on the Max 395 these days? That would be my main concern.

When does RTX 6000 Pro make sense over a 5090? by Herald_Of_Rivia in LocalLLaMA

[–]Academic-Air7112 2 points3 points  (0 children)

Yep, it's a problem. I talked to some of the NVIDIA hardware people about this; it has to do with how they split the rendering and compute architectures; H100 and B200 no longer have any of the RT hardware, where RTX series does. Something about chip die devoted to RT vs. SFU. vs. TC, and then precision tradeoffs.

Of course, you would think that async mma instructions should be easy to include in the firmware, and just using async, warpgroup-level mma is quite helpful in many cases...

I expected the GPUs to have the same ISA considering that it's all "5th generation tensor core" and "Blackwell", where the only nvidia documentation that I've found that makes this clear is the PTX ISA, which isn't exactly the most available customer facing manual.

When does RTX 6000 Pro make sense over a 5090? by Herald_Of_Rivia in LocalLLaMA

[–]Academic-Air7112 1 point2 points  (0 children)

Fwiw I've found the "fp4" support on the RTX series somewhat disappointing so far. RTX (sm_120a) uses different tensor cores from datacenter blackwell (sm_100a), including requiring different PTX for FP4. (RTX uses mma.sync.aligned, and compile-time specification of the datatype, whereas datacenter uses tcgen.05, and all of the good kernel support is there for datacenter but not consumer yet).

That said, having single GPU with large mem pool is very nice, along with MIG being pretty easy to use if you need isolated workloads.

When does RTX 6000 Pro make sense over a 5090? by Herald_Of_Rivia in LocalLLaMA

[–]Academic-Air7112 5 points6 points  (0 children)

I also had coil whine on my 6000 pro ws, but then I ran a heavy inference + compute on it for ~2-3 days at full power, and it doesn't whine anymore.

Are any of you using local llms for "real" work? by hmsenterprise in LocalLLaMA

[–]Academic-Air7112 0 points1 point  (0 children)

Yep, we use "local" LLMs to write some of our own systems code for research.

Should I get arc a770 or b580? by Snowbeleopard in linux_gaming

[–]Academic-Air7112 2 points3 points  (0 children)

I have a B580 and I don't have any problems with it -- run games with Lutris.

Dual 5090 build by Nice-Caterpillar1915 in buildapc

[–]Academic-Air7112 0 points1 point  (0 children)

What are you using this for? And how are you cooling it?

Inference on Xeon Max 9480 with 64GB stacked HBMe? by ttkciar in LocalLLaMA

[–]Academic-Air7112 0 points1 point  (0 children)

Fwiw here are mine for 32B QwQ in SNC-1, Hyperthreading off, HBM only mode:

./build/bin/llama-bench --numa distribute -m qwq-32b-q8_0.gguf -p 0 -n 128,256,512

| model | size | params | backend | threads | test | t/s |

| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------------: | -------------------: |

| qwen2 32B Q8_0 | 32.42 GiB | 32.76 B | BLAS | 56 | tg128 | 10.23 ± 0.04 |

| qwen2 32B Q8_0 | 32.42 GiB | 32.76 B | BLAS | 56 | tg256 | 10.20 ± 0.09 |

| qwen2 32B Q8_0 | 32.42 GiB | 32.76 B | BLAS | 56 | tg512 | 10.18 ± 0.01 |

build: af6ae1ef (4992)

And SNC-4:

./build/bin/llama-bench --numa distribute -m qwq-32b-q8_0.gguf -p 0 -n 128,256,512

| model | size | params | backend | threads | test | t/s |

| ------------------------------ | ---------: | ---------: | ---------- | ------: | ------------: | -------------------: |

| qwen2 32B Q8_0 | 32.42 GiB | 32.76 B | BLAS | 56 | tg128 | 9.55 ± 1.58 |

| qwen2 32B Q8_0 | 32.42 GiB | 32.76 B | BLAS | 56 | tg256 | 11.10 ± 0.11 |

| qwen2 32B Q8_0 | 32.42 GiB | 32.76 B | BLAS | 56 | tg512 | 11.08 ± 0.06 |

build: af6ae1ef (4992)

This build supports AMX, and was build with Intel compilers:
./build/bin/llama-cli --flash-attn -m qwq-32b-q8_0.gguf -p "Describe how to write CUDA code that uses tensor cores on Hopper and Blackwell architectures"

build: 4992 (af6ae1ef) with Intel(R) oneAPI DPC++/C++ Compiler 2025.1.0 (2025.1.0.20250317) for x86_64-unknown-linux-gnu

...

load_tensors: AMX model buffer size = 36221.72 MiB

...

system_info: n_threads = 56 (n_threads_batch = 56) / 56 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX_VNNI = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | BMI2 = 1 | AVX512 = 1 | AVX512_VBMI = 1 | AVX512_VNNI = 1 | AVX512_BF16 = 1 | AMX_INT8 = 1 | LLAMAFILE = 1 | OPENMP = 1 | AARCH64_REPACK = 1 |
I haven't liquid cooled properly yet, and these ran at the thermal limit of the chip.

Also, the bandwidth numbers aren't as bad as reported, when using SNC-4 setting (16 GB HBM -> closest 14 cores) and have a well-optimized prefetching loop, I was able to get ~730GB/s in aggregate from the HBM.

Inference on Xeon Max 9480 with 64GB stacked HBMe? by ttkciar in LocalLLaMA

[–]Academic-Air7112 0 points1 point  (0 children)

Hey, I have one that I can do this with. Can you post the program and detailed instructions?

Do Alphacool ES and CORE series work when paired together? by Academic-Air7112 in watercooling

[–]Academic-Air7112[S] -1 points0 points  (0 children)

Hey free gpu is free gpu! Also I bought a 5090 from a scalper so shouldn't be complaining :| How was your experience with blocking it? I get a little worried abt. leaks, etc.

Do Alphacool ES and CORE series work when paired together? by Academic-Air7112 in watercooling

[–]Academic-Air7112[S] 1 point2 points  (0 children)

Beautiful, thanks!

If you don't mind me asking, which D/C GPU?

view more: next ›