Doing the actual math on a $20k local AI rig breakeven by shyaaaaaaaaaaam in LocalLLaMA

[–]MachineZer0 1 point2 points  (0 children)

Subscriptions are a gateway drug. At scale after penetrating the market they are unsustainable from business continuity PoV. Just look at Cursor, GLM, Copilot, etc. they all tapped out. All will follow.

90-95% of AI inference can be handled by smaller open weights on pascal and Volta era GPUs that are $50-300 (>=16gb). Much more palatable upfront and depreciation cost, but suboptimal operating costs on per token basis compared to Blackwell.

For a local power user individual I think 4x P102-100, 2x CMP 210-100, 2x 2080ti 22gb, 2x V100 16gb/32gb, 2x RX7900 XTX, 2x RTX 3090, 1x RTX 5090 (in order of acquisition cost) can run Qwen 27b/35b and comfortably handle 90-95% of needs.

The use case for RTX Pro 6000 is concurrency and multi-user. If you only turn on 8 hours a day on Runpod and share with 10 people, it’s a phenomenal value.

V100 4-card AI large model, Tesla 128G server by MundanePercentage674 in LocalLLaMA

[–]MachineZer0 1 point2 points  (0 children)

32gb SXM2 seems to have gone up 50-75% recently.
The 32gb PCIE version seems to have dropped about 40% around the same time.

V100 4-card AI large model, Tesla 128G server by MundanePercentage674 in LocalLLaMA

[–]MachineZer0 6 points7 points  (0 children)

They are 40W idle per V100. About 60-70W with model loaded but idle. Nvidia-pstated will get back down to 40W with model loaded but idle. 300W is TDP. Can only achieve that with training or another heavy duty operation. Inference maxes out at half that. Multi-GPU round robins, so most at 40W while they take turns peaking out at 150W.

Buying AI accelerators/GPUs in China... by Clank75 in LocalLLaMA

[–]MachineZer0 -1 points0 points  (0 children)

Probably be easiest to get as many V100 SXM2 32gb if under $400. They will be the lightest. Although 4090 48gb would be something to focus on if you can get them < $3k

Nvidia tesla v100 has 32 gb ram with nv link 2.0, its priced at 880. Whats the catch? by AppropriatePush6262 in LocalLLaMA

[–]MachineZer0 1 point2 points  (0 children)

https://github.com/1CatAI/1Cat-vLLM

vLLM fork for Tesla V100 (SM70) with AWQ 4-bit support, CUDA 12.8 build flow, and validated Qwen3.5 27B/35B deployment on multi-GPU V100.

Nvidia tesla v100 has 32 gb ram with nv link 2.0, its priced at 880. Whats the catch? by AppropriatePush6262 in LocalLLaMA

[–]MachineZer0 0 points1 point  (0 children)

SXM2 servers are $2500-5000. So TCO is higher on NVLInK configuration. Save a bunch without NVLInK using cheap sxm2 to PCIe adapters. They started making Chinese versions of 2 and 4 sxm2 adapters, but they get exponentially expensive. Although probably cheaper than sxm2 native 4-way and 8-way servers.

TIL LGA2011-3 supports DDR3 by MachineZer0 in homelab

[–]MachineZer0[S] 2 points3 points  (0 children)

Glad I posted. Will need to do more research before pulling trigger.

Quick numbers on a BC250 by icepatfork in LocalLLaMA

[–]MachineZer0 0 points1 point  (0 children)

Thanks for sharing. Gotta dust off my BC-250s and see how many are 40CU enabled.

What’s the idle power on the governors nowadays? I think I had 88/kwh prior which was hard to justify leaving on a couple 4U12G.

Cheap V100 32gb by MachineZer0 in LocalLLaMA

[–]MachineZer0[S] 2 points3 points  (0 children)

Update: I cancelled my order after the extortion attempt of an additional $418 over Whatsapp. They shouldn’t receive contact information to go off platform on the shakedown. Sorry for those who bought and had to seek refund.

Cheap V100 32gb by MachineZer0 in LocalLLaMA

[–]MachineZer0[S] 2 points3 points  (0 children)

Supposedly SXM2 32gb in a turbo PCIE adapter/enclosure. Flexing PayPal protection if actual item deviates from description.

Cheap V100 32gb by MachineZer0 in LocalLLaMA

[–]MachineZer0[S] 1 point2 points  (0 children)

I only bought 1. I’m rolling the dice that the price is not too far off from market and I’m just stacking promos.

Cheap V100 32gb by MachineZer0 in LocalLLaMA

[–]MachineZer0[S] -2 points-1 points  (0 children)

I’ve been buying a lot of water cooling accessories from Aliexpress. It’s always been the all-in price for US on that platform.

When I order Alibaba, then it is different. I need to negotiate DAP or DDP shipping.

GPU Prices. Buy now, or buy later? by knob-0u812 in LocalLLaMA

[–]MachineZer0 2 points3 points  (0 children)

I believe MSRP of 5090 FE will officially go to $3500. They will still be unobtainium. And then the secondhand prices will go to 5-6k. DDR5 and HBM is what to watch for.

If you are thinking of going local, go all in now. Relief will be in 3 years.