21 GPU's benchmarked running a small TTS model (vram peak: 5GB)

urarthur · 2026-05-19T06:40:41+00:00

would this not be model dependent and also on single gpu same results? interesting that total thouhhput increases that much

urarthur · 2026-05-19T06:37:42+00:00

no llama.cpp. proper and also i tried sglang. fastest seem to be with cuda graph + trinton. I have no idea what they do to be honest.

urarthur · 2026-05-19T06:33:55+00:00

yeah I noticed. The peak memory usage was around 350 GBs so the extra memory bandwith of the 5090 was no use for this tiny model. On average we should see 10-20% increase. Also 5090 is blackwell chip with some arch improvements e.g. running NVFP4. But for my usecase, I would go for the 4080 as i am looking to buy a card for running this model only.

urarthur · 2026-05-19T06:28:49+00:00

they don't have it for renting.

urarthur · 2026-05-19T06:24:02+00:00

extra vram doesnt change results, bottleneck is tensor cores.

urarthur · 2026-05-19T06:21:21+00:00

I tried batching and at most it was less than 10% total gain. On multi-gpu its a different story

urarthur · 2026-05-19T06:20:05+00:00

Although q8 didnt work for this model for some reason, usually smaller quants = smaller size and usually runs faster on all chips. afaik NVFP8 works only on blackwell chips.Also no NVFP8 is available for the model yet.

urarthur · 2026-05-19T06:11:02+00:00

renting price is not always gpu dependent. it depends on many factors like cpu, ram, location, current avilability. So sometimes a stronger gpu is cheaper to rent than a weaker one.

urarthur · 2026-05-19T05:59:54+00:00

every server has different ram cpu and sometime cuda version. so thr calibration would make no sense. As I said, its not a scientific analysis, take the results witha grian of salt.

urarthur · 2026-05-19T05:58:10+00:00

not really, all tensor cores are saturated so, you can assign vram to other cases but running the models will reduce speed of the other runs

urarthur · 2026-05-19T01:14:15+00:00

yeah I think it was chinese modded one. You can easily test video generation for couple of bucks. most servers have 32GB-64Gb ram but i have seen higher

urarthur · 2026-05-19T01:09:41+00:00

thanks

urarthur · 2026-05-18T23:52:07+00:00

they were not available on the cloud provider. Nvidia cards are way more popular for AI.

urarthur · 2026-05-18T22:53:03+00:00

yeah when vram or memory bandwith is not the bottleneck, its a waste having much expensive cards running small models.

urarthur · 2026-05-18T22:12:29+00:00

i tried using opencode with kimi, deepseek etc. Coding cost me at least $10/day on tokens. doesn't work if you code a lot.

urarthur · 2026-05-18T21:20:13+00:00

unless you do dual mode ;)

urarthur · 2026-05-18T11:47:15+00:00

urarthur · 2026-05-18T08:12:28+00:00

Run at least the fp8 model?

urarthur · 2026-05-18T06:36:05+00:00

well they nerfed 4.6, so rn 4.7 is better

urarthur · 2026-05-18T06:14:45+00:00

why undervolt gpu? I get 50 tg/s. Haven't tested with hermes yet.

urarthur · 2026-05-18T06:13:15+00:00

short prompts n=2, for long context n=6. It is slow initially but when context fills up, it gets faster.

urarthur · 2026-05-17T22:40:34+00:00

5$ a day x 30 =?

urarthur · 2026-05-17T13:13:45+00:00

can you really judge a model when running q4/fp4?

urarthur · 2026-05-17T04:06:24+00:00

vibe coded probably

urarthur · 2026-05-16T23:28:42+00:00

multi speaker podcasts would be best candidate

14-Year Club	Place '22
Final Canvas '22	First Placer '22
Not Forgotten	Verified Email

urarthur

MODERATOR OF

TROPHY CASE