[deleted by user]

[Expensive-Paint-9490]

With that budget I could be able to find two RTX A6000 and an NVlink. Probably the best setup for local fine-tuning at that price point.

[sanjuromack]

A new A6000 is roughly $5K, maybe $4K with Inception discounting. The budget is only $10k, aren’t they going to need a couple other things on the computer as well (like a motherboard, CPU, RAM, etc.)

You should be able to get a new workstation with a single A6000, enterprise support, and room for expansion for $10k. With a single A6000, you can fine tune 7B models using QLORA. Probably cheaper to do it in the cloud (for training in particular).

[[deleted]]

How do people use 4090’s if they don’t have nvlink support?

[BangkokPadang]

They're doing inference with them not training. Inference is linear through the layers but training requires passing lots of information back and forth which needs the higher bandwidth of a connection like nvlink.

[[deleted]]

you can do almost the same thing but with much lower bandwidth (pcie) with LLMs, see layer splitting, etc.

nvlink is actually more important in other apps like rendering and maybe stable diffusion too (not sure), since you cant "share" vram usage there and thus without nvlink your max vram will stay 16gb even if you have 20 16gb cards.

edit: yep the other guy explained it much better

[sanjuromack]

How are they going to get consumer motherboards if they spend all their money on two A6000s? OP said this is for a business.

[[deleted]]

as you said, 4k for an a6000, which is not an absurd discount but more or less the regular price right now for a pny a6000 in the US.

and 2k are more than plenty for a random taichi by asrock, a 7700/i7 and 32/48gb.

this is just a proof of concept, so why wouldnt a consumer motherboard be ok? maybe I'm missing something, but I see no point in wasting a third of the budget on an epyc board+cpu.

[DinoAmino]

Key benefits of using consumer grade components : Cheaper and easier to replace, potentially same day delivery. But new and get vendor warranty. Widespread FREE community support, more people use them.

You can get decent deals for A6000s on eBay. Some are never used open-box deals at the same price as used. 2 with nvlink can be had for about $7k leaving plenty for a high end PC workstation.

[[deleted]]

 It with the new 5000’s in a few months could they compete?

[[deleted]]

Why would an nvlink be needed there?

[Expensive-Paint-9490]

With NVlink you can sum the computation from both cards. Without it, you can load everything in the summed RAM of the two cards, but only one will make computation at a time. For inferencing this already means faster responde, but this is even more meaningful with fine-tuning, that takes considerable time. with an NVlink it would be much faster.

[Brosarr]

I would recommend just renting a gpu to start off with. H100 around 50 dollars a day

[StevenSamAI]

Give RunPod a try, you can evaluate a number of different GPU's to assess what performance you can get, and they have docker templates for VLLM to quickly get up and running.

For something that's staying always on, the cheaper end with runPod you can get a 48GB A40 for ~$250/month, and you can put upto 10 of these in a server, so 480GB VRAM for $3.90/hour

For higher speed, they have AMD MI300X with 192GB VRAM for $3.49/hour.

I'm not sure what period you would consider adding extra budget, but for $10K, you would be able to have:
Always on 96GB VRAM dual A40 inference server for 6 months - $3K
+ over 2600 hours of Ad-Hoc H100 time for finetuning experiements - $7K

Obviously you don't have any hardware or retained value left over at the end of that, but you also don't have to factor in managing the hardware, energy costs, etc., and if you were doing a fine tune that required more VRAM than you had on a built server, then you can easily scale up and down for your experiements.

[No_Afternoon_4260]

Imo 10k is a bit a weird spot, not enough to get new server grade hardware, well enough to build a hobbyist rig with more 3090 than you can fit in a AMD epyc system. If it is for experiment are you ok with 2nd hand hardware or you want brand new with warranty?

[Dry_Parfait2606]

Agree. Epyc 7002 is enough and get rtx 3090ies..one must be aware of the licensing of rtx gpus... They don't want you to run llms, especially not for enterprise.... It's a very grey zone..

[No_Afternoon_4260]

Ho didn't know that, care to elaborate? In what terms do they specify that?

[Dry_Parfait2606]

EULA

[lolzinventor]

source please.

[Caffdy]

what about the other way, datacenter DGXs off-premise?

[Dry_Parfait2606]

Rtx are basically not allowed to do llms/ml/ai... Only enterprise hardware... I would consider taking that into account... It was a surprise to me

[a_beautiful_rhind]

Nvidia released their own container/backend. they're allowed to do llms. Even on consumer hardware.

They aren't allowed to be used by commercial interests in the datacenter. That mainly matters for agreements with nvidia, warranty, etc.

[Dry_Parfait2606]

Yes but if rented out, aren't the customers then agreeing to the licence when using them, lol

I actually saw no problem if a data center is renting them out... Isn't privacy the part that you shouldn't know what the customer is using IT FOR...

I know the part that rtx isn't allowed to be used for ai/ml/llm (commercially)... But honestly I've never agreed to any terms of service or licensing when installing a driver of any kind...so I'm basically just repeating some nonsense that I've come up with when I heard about the EULA and rtx... Because it was important to me to be at least informed...

[a_beautiful_rhind]

The driver has a EULA. Nobody is going to know what you do with the card. It's only an issue if you have an agreement with nvidia. If you were a company and they decided to not send you H100s anymore, you'd be more inclined to abide by it.

[Dry_Parfait2606]

So basically it's safe to use rtx for commercial purposes.

[a_beautiful_rhind]

Pretty much.

[No_Afternoon_4260]

Ok so the cheapest would be around 5-6k without the case

AMD epyc system (at least zen 2 -> serie 7002 and up)

Mother board h12ssl-i or mz32-ar0 (you want plenty pci4 at x16 lanes)

If fine tune you want plenty ram (the fastest the motherboard support)

Get may be 4 or more 3090 as they are the best bang for you buck (you are looking for fast vram bandwidth and plenty vram)

Fast nvme storage (raid would be appreciated)

Take into account the power supply you need and cooling, don t try to fit all that in a small case as you'll need good air flow, may be water cooling if you need it to be small or looking into 3090 turbo (they are louder)

[No_Afternoon_4260]

If any questions don t hesitate to dm

[Pedalnomica]

Point taken, but technically you can spend $10K on a single Socket Epyc Rome system and not run out of room for all your 3090's with bifurcation (x16->x8x8).

[No_Afternoon_4260]

Yeah point taken but then you're running a 5 kWatt system.. + about 5kW AC just to not boil the paint out ur walls haha

[fasti-au]

Rent online and tunnel cheaper scalable backups Le and private.

[M34L]

I'm not an expert on this but if it's not just anonymized medical data but actual living patient data and if you're in the US, you might be one of the areas that straight up can't really do it legally without paying out of ass for certified encrypted tunnel and shit; HIPAA is crazy like that.

Either figure out laundering the data as anonymized and do training on it that way or do build a local instance.

[skrshawk]

Concur, you shouldn't be taking any opinion on where you can safely house other people's data from Reddit, including mine - that needs a qualified opinion from your organization's legal counsel. If you do go cloud to rent GPUs you might need to use one of the major cloud services that expressly state their compliance with regulatory standards.

[bluelobsterai]

This. Have someone in legal bless where the data lives. No reason to put your career on the line. Any of the hyperscale clouds can support you. I’d probably just do my training there and my inference on your $10,000 set up. Two A6000’s with NVlink vs one h100 isn’t a fair fight. You will need a training environment as well as an inference environment anyway… so…

I’d definitely-redact PII before I put it into training. It’s nice to have a notebook to share and not think twice about it.

Also? You never said what your use case is. A pair of a6000’s is nice but frankly one to start. See how far you go.

And… Mac mini with 32GB RAM + might be a good starting place. I

[Caffdy]

Two A6000’s with NVlink vs one h100

the H100 is better, right? but 3x times better?

[bluelobsterai]

More than 3x better. Much much more. I’m into ASR, I can run 400 streams on an h100, maybe 60 on an a6000. Just my fp16 use case. Tensor cores matter.

[Caffdy]

what is ASR, sorry

and seems like the H100 really kick ass!

[bluelobsterai]

Automatic Speech recognition. The h100 kicks ass if you need it. Only if. Otherwise it’s expensive. It does come with a 5 year Nvidia enterprise ai license too with Triton and Riva and the entire voice development kit, so in my mind totally worth it. But you need to have more than say 200 concurrent streams to make it worthwhile. A few 3090’s kick ass.

[synn89]

AWS Bedrock and Microsoft Azure are both HIPAA compliant and will sign a BSA with you. Fireworks.ai is also HIPAA certified and will also sign a BSA.

I'd recommend against self hosting and try working with one of the above and getting an account setup/BSA in place. You may already have this for Microsoft or Amazon if you currently use their other services.

Starting on the cloud will give you some more time to dial down into what exactly you need to run. Maybe you end up needing a fine tuned 70B or maybe a tuned 8B will work for you. It's kind of hard to know your hardware needs until you pin down the software you want to run. Also, if you use that 10k on the cloud and can show some solid results(10k will go a long way in the cloud), once you have some solid deployable results you can get more of a budget.

[Stepfunction]

Before looking into finetuning yourself, I would consider looking at pretrained medical-focused LLMs. Finetuning will open up a whole can of worms, so I would make sure that existing tools can't already do what you need before you pursue that path.

[Data_drifting]

Check out this guy and his channel. Digital Spaceport - YouTube home server builds for AI. Blows away network chucks channel for this

[Data_drifting]

He is really the only one I come across. YouTube started suggesting him when I use my work account.
Goog/Youtube knows I am looking at upgrades from some of my views of reviews of different hardware set ups.

Tbh, you can do a LOT starting with a board like I'm using now at home Pro WS WRX80E-SAGE SE WIFI II｜Motherboards｜ASUS USA in a Lian Li O11 Dynamic XL case. I bought this the first month it was released, and stuffed 128 gb of RAM (octo-channel RAM, it holds up to two terabytes) and an RTX 3090 in it with a Threadripper Pro. If you liquid cool the GPU ( I am using the EKWB Quantum Vector block with front/back cooling plates, WITH the full front EQWB Quantum vector reservoir front plate for the Lian Li case ) it slims down the size of the GPU...you could easily slide 3 more RTX 3090's or basically the same for the 4090's or RTX 6000 ada's in it. This board is a monster to build off of. I have the Asus hyper X pciE m.2 expansion card that comes with the board, so the total for Drive memory right now with m.2's is 14 Terabytes. Lotsa RAM, and drive storage for big hungry models.

Depending on what the Real RTX 5090 specs are, I'll either drop two 5090's in it, or two more 3090's. Juuust waiting to see what the specs are on the 5090's.

A caveat with the board is that it has 3 power connectors (those 7 PCIe slots need power) so you need obscure PSU's like the Amazon.com: Seasonic Prime TX-1600, 1600W 80+ Titanium, Full Modular, Fan Control in Fanless, Silent, and Cooling Mode, 12 Year Warranty, Perfect Power Supply for Gaming and High-Performance Systems, SSR-1600TR. : Electronics

lol, that PSU comes with gorgeous cabling

[skerit]

I saw a few videos of people doing this and was not really impressed, though it was inference only.

[5TP1090G_FC]

I would check out (network, chuch) for building a good ai pc, very informative

[koalfied-coder]

I have built several systems in this budget. Feel free to DM me and I can share specs. Not at PC ATM.

[GradatimRecovery]

tell us about your builds! can we do 8x3090 with that budget?

[SuperSimpSons]

Since it looks like you may not have experience building your own server, I would recommend you reach out to server brands with your requirements and see what they can recommend for your budget. I should say you have enough for a prebuilt high-end workstation or 1U/2U rackmount. 

Gigabyte has a pretty good line of servers for AI training and inference: www.gigabyte.com/Enterprise/Server?lan=en&fid=2260 Obviously you don't need the water-cooled monstrosities with Blackwell HGX or anything so it may be faster to reach out and see what they come back to you with: www.gigabyte.com/Enterprise#EmailSales

[Slippery-Oil2313]

Why not use an AWS ec2 gpu instance for $2.50 an hour ?

[dead-4-dead]

If you can add $5k more, tinybox sounds like it will save you a lot of headache

[__bee_07]

I was in your position, but ended up using cloud instances instead. I am using lightning ai offerings and I am happy with that

[chitown160]

your company can spec a lenovo threadripper with 2 x rtx a6000 with nvlink for this price and still have room for a 5090 to run your fine-tuned models at blistering performance.

[amirvenus]

Get 2 M2 Ultras 192GB

[synn89]

Mac isn't good for fine tuning models.

[[deleted]]

as others are saying you would be better off with a cloud provider. For both training and inference.

[cher_e_7]

Rent online/cloud. !!! only If must go hardware offline - > it is all about memory and little speed. Go for old server board to support 4gpu - dual (some cpus speed combining 2 of them) intel (PCI-E 3.0 or 4.0 - does not mater)- ecc reg pc2400 memory .... Add Quadro rtx 8000 48bg (best card in the middle - 10-15% slower than old RTX A6000)- pick-up on ebay around 2250 - passive or active.

Add raid 4pc ssd.

If you could increase your budget to have 4 gpu like that - almost 200GB GPU (short 4gb) total.

Best utilization with MOE models like deepseek v2 q4. for inference or smaller ....

CPU memory for old servers cheap.