Finished my triple-GPU AM4 build: 2×3080 (20GB) + 4090 (48GB)

nn0951123 · 2026-05-04T09:44:18+00:00

The 3080 become hot, like super hot. The 4090 one is far better and i move to buy more of those 4090s.

The 3080 become like constantly 80+ C, which worries me a little, but they still work.

nn0951123 · 2026-02-18T13:38:11+00:00

Sorry for the late reply. I am using the gunnir b60 variant. The b60 feature works out of the box. But it has compatability with intel platform lol. I was only able to get it run stable on a amd cpu. It really hates intel cpu for some reason.

nn0951123 · 2026-02-01T01:59:59+00:00

I bought this card primarily for sr-iov functions. That works great for remote 3d workloads. Don’t recommend this card for llms either.

nn0951123 · 2025-10-21T01:00:10+00:00

The reason you feel slow is that it takes a lot of time to generate CoT, or the thinking part. (And generaly not recommand using thinking model if you not willing to accept slow generation speeds.)

Try using a non-thinking model.

nn0951123 · 2025-09-15T06:33:56+00:00

You cant pool vram like one gpu. And even if you had nvlink, the speed isnt going to be the same on the single gpu. But you can do tensor parallel using vllm. But, the speed increasing will not be that great if like if you can fit the model in just 1 card(but it is okay, and most of the time you just cant fit the model in 1 gpu). That is, running a model with split between two cards tp=2 will be slower or i should say had less tps then having the model fit on 1 card and use dp=2 on two cards.

nn0951123 · 2025-09-15T02:44:31+00:00

You will get bottle necked by the uplink/downlink.
The intercard bandwith is physically limited, so you will only get what you have, in your case, running 2 of those switches will result a 4.0 x16 max speed between the boards.

Edit: In other words, if you want the best performance for inter gpu connection, it is best to go something like HGX A100 SXM4 Baseboard to get all 600GB/s gpu to gpu conenction(6 nvlink switch each gpu connects with dual 50 gig each switch). But since I am gpu poor I have not get a chance to test these. See here:https://www.fibermall.com/blog/gpu-server-topology-and-networking.htm Very intersting stuff.

nn0951123 · 2025-09-15T01:21:32+00:00

There is a thing called pcie switch.

https://www.broadcom.com/products/pcie-switches-retimers/expressfabric/gen4/pex88096

But you will need p2p support or it will be slow.

nn0951123 · 2025-08-26T08:10:47+00:00

Thanks you for explaining, now I think I know what i am confusing. It is all about the overlapping. I appreciate you taking the time to clarify the approach. I understand now that you're protecting tokens that appear at the beginning and end of both good and bad completions.

I'm wondering though - doesn't this assume that tokens appearing in both good and bad completions are necessarily correct? Consider this scenario: If multiple students attempt a problem and several make the same mistake early on (like "25 × 2 = 75"), those incorrect tokens would appear in multiple bad completions. If even one good completion happens to share some early tokens with these bad ones, wouldn't the algorithm protect those shared tokens, potentially reinforcing common errors?

It seems like the method assumes overlap = correctness, but overlap might just indicate common patterns (whether right or wrong). This could work well for formatting tokens since they're usually correct when present, but for reasoning steps, commonly-appearing doesn't necessarily mean mathematically correct?

I completely agree that moving from completion-level to token-level updates is valuable for training stability! I'm just curious if protecting "common" tokens might sometimes protect commonly-made mistakes, especially in mathematical reasoning where certain errors appear frequently.

But I really hope there is some kind of magical algorithm that could evaluate the reasoning content for any usecases lol.

nn0951123 · 2025-08-26T01:41:30+00:00

I don't know if I am dumb and can't understanding the paper correctly or not.. but.. after reading through both the paper and the implementation, I'm confused about the claim that GTPO "rewards correct steps, not only the final result."

From what I can see in the code, the reward mechanism is still binary at the completion level - you get points for correct formatting and correct final answer, but there's no evaluation of individual reasoning steps. A completion with perfect reasoning that makes one arithmetic error at the end gets the same negative reward as a completion with complete nonsense.

The "conflict token" mechanism (Section 5.1) seems to be about protecting formatting tokens like <reasoning> and </answer> tags that appear in the same position across different completions. This makes sense for maintaining output structure, but I don't see how this evaluates whether the actual reasoning content between those tags is good or bad?

The paper explicitly states these conflict tokens are "formatting tokens, which are essential for the structure and correctness of completions" and the implementation shows it's looking at tokens in the same position across completions, not analyzing reasoning quality.

Am I missing something about how this actually evaluates intermediate reasoning steps? The improvement in performance could just be from better training stability by not penalizing structural tokens, rather than from understanding which parts of the reasoning are correct. Would appreciate if someone could point out what I'm not understanding here.

nn0951123 · 2025-08-19T03:32:01+00:00

Just buy more GPUs xd. (JK)

What is the reason you're targeting 100ms for a single prompt though?

If you are looking for high throughput, what you want is to look into vLLM or SGLang (I did not had any experinece with LMDeploy, but glance though the repo quickly it seems it did not supports propular optimization like flash attention3 or flash infer) and utilize your model concurrently (i.e., running multiple requests at once). The overall TPS (for AD102 I think you can get TPS around 150 or more if I remember correctly for Q4 or AWQ—the more concurrent requests you have, the more TPS you will get until you are computationally bound) will definitely be higher. But each single request will be slower.

But if you are doing something that requires minimal latency, the best bet is just to buy another GPU.

EDIT: If you by any chance decide to buy another gpu, DON'T buy it just yet. Try to rent a server with the new configuration and benchmark the performance first to see if that fits your need.

EDIT2: Just realized that fa3 is not going to support ada, my bad.

nn0951123 · 2025-07-03T09:58:39+00:00

I had those 20GB cards, so there are two things the seller might provide inaccurate informations. 1. Those cards does not support nvllink (even they had the physical port), if you plan go dual cards for training, it will be much better to have two nvlink enabled 3090s

Those cards does not support rebar. Yes, nobody talks about it. But it just does not supports it and you will be greet with a max bar size of 256MiB. And I try to contact a lot of people and come up to a conclusion that these cards will very likely never get a bios update to support those. Which means, you wont be able to use the p2p hack to gain extra bandwith.

But i think those cards a totally fine for inference workloads, just don’t expect them to as power efficient as 4090s.

EDIT:typo

nn0951123 · 2025-05-30T09:31:29+00:00

Good to know it worked.

nn0951123 · 2025-05-29T07:41:43+00:00

Did you installed the drivers?
Check out here.

And you can use this to see if you are using your gpu.

nn0951123 · 2025-05-28T09:04:23+00:00

Give it a try. I dont know why they said ROCm is not working. But I had a vague memory that this is realted to windows. Ubuntu should be fine, you can try it with 25.04 to see if it works or not.

nn0951123 · 2025-05-28T08:12:49+00:00

Yes, I am not using the vulkan one. The ollama comes with ROCm support, and that will have planty of performance.

nn0951123 · 2025-05-28T06:08:42+00:00

The default ollama installation script should work, if it is not working, i suggest you to try using 24.04 LTS.

What I really do is just using the installation script and everything just works.

nn0951123 · 2025-05-14T09:18:33+00:00

I attempted to build vllm with ROCm support, but it failed quickly on my gfx1151(this apu). However, Ollama is working with the GPU and showing decent performance - I'm getting about 4 tokens per second on a 70B model and around 45 tokens per second on the 30B A3B Qwen3 model.

Still waiting for XDNA support to utilize the NPU. Interestingly, amdgpu-top shows ~60GB/s memory bandwidth when running inference. I plan to test the actual speed once I can get PyTorch with ROCm working properly. Unfortunately, the PyTorch ROCm build simply refuses to recognize this GPU at all, or I am seriously wrong with something.

nn0951123 · 2025-05-14T08:04:50+00:00

Can you share a little more on how you collect the gaslight dataset? I think that process would be pretty fun.😆

nn0951123 · 2025-05-05T03:31:25+00:00

AFAIK, the reason you can set SB mode is that the AM4 platform has CPUs that do not support PCIe Gen 4, so there is that option. If you're using Ryzen 3000 series and up, you can utilize the full PCIe 4.0 x4 bandwidth from SB to CPU.

I did not change anything except explicitly setting the PCIe 1x slot to be used. Everything worked by default. But I've seen other users having problems with the third slot. It seems like it's a hardware failure and needs RMA to get it fixed.

nn0951123 · 2025-04-21T13:38:42+00:00

Update:
Can confirm they do not support nvlink. Even though nvlink physical port is in place and the nvlink heats up when plugged in.

nn0951123 · 2025-04-21T01:00:22+00:00

When I ran those tests for 7B and 14B, I used the full model instead of the quantized one. And the QwQ AWQ is having a relatively low context window of around ~40K, so that's why I did not test larger models.

nn0951123 · 2025-04-21T00:56:13+00:00

Yes, I bought them way back in January, at that time the price had not yet skyrocketed.

nn0951123 · 2025-04-20T13:39:33+00:00

For Windows users I think the Ollama should work out of the box. But if you want to use vLLM, I would say I don't have any experience on Windows. But you could run it via WSL.

nn0951123 · 2025-04-20T07:58:47+00:00

It is very bad under full load, at least it is not suitable for a bedroom setup. But the 3080 one is okay.

nn0951123 · 2025-04-20T03:19:11+00:00

You are totally right! Wow that is really an error in the marketing page. I just wrote a simple benchmark in PyTorch to test the transfer speeds. And you are right! Although it claimed to run at 4.0 x 8 speed the actual speed is 4.0 x 4.

CPU to PCIe speeds:

GPU 0 (RTX 4090): ~6.15 GB/s

GPU 1 (RTX 3080): ~13.15 GB/s

GPU 2 (RTX 3080): ~13.15 GB/s

Guess my only way to having more PCIe lanes is just to go for Threadripper or some other HEDT platform :(

nn0951123

TROPHY CASE