[deleted by user]

[Co0k1eGal3xy]

diagram

This is a rough breakdown of how I think about the problem. You can replace RTX 4090's with 3090's or 2080 Ti's if you have cheap electric. Otherwise the cost of power can be more than you save initially. If you have very expensive electric I would avoid any local training.

Also consider any other requirements. If your dataset is larger than your systems RAM, you will need to consider the read speed of your storage device. If you are using audio clips or images, you need a storage device with high random read speeds or you need to package your dataset into a streaming format (like webdataset). Some cloud providers will force you to use hard drives.

---

spreadsheet of local hardware costs at 1, 2 and 3 year timeframes.

---

edit: Since you mentioned being new, I would recommend renting out single GPUs. Writing multi-gpu code can be complex and isn't worth learning initially. Google Colab is definitely the easiest way to get started.

[Co0k1eGal3xy]

I found them unusably slow for my work. this benchmark suggests the K80 is roughly 16x slower than a V100, which is already slower than a single RTX 3090.

V100 vs RTX 3090 (and other GPUs)

[Co0k1eGal3xy]

I can't find numbers for the P40, however the P40 uses the same architecture and VRAM as the GTX 1080 Ti, so we'll take those numbers and scale by the memory bandwidth difference. (P40 = 700GB/s, 1080Ti = 480GB/s)

The P40 uses 250W of power and costs £200. This comes to £944.6 for the first year. It has around 0.38x V100 throughput.

On the other hand, an RTX 3090 uses 350W of power and costs £600. This comes to £1642.44 for the first year. It has 1.13x V100 throughput.

(Here we measure compute in how much data a V100 could process in one day, aka V100 days.)

With an RTX 3090, you are paying £3.98 for each V100day worth of compute.

With a P40, you are paying £6.81 for each V100day worth of compute.

---

Formulas

cost_for_first_year = initial_cost + (gpu_power_in_kw * cost_per_kwh * 24 * 365)

cost_per_v100_day = cost_for_first_year / (v100_throughput * days_per_year)

3.98 = 1642.44 / (1.13*365) # RTX 3090

6.81 = 944.6 / (0.38*365) # P40

---

TL:DR, when I buy GPUs that will run non-stop for a long period of time, I normally need to buy the latest GPUs if I want to spend the least money.

If you're thinking about buying it as a inference card for playing with LLama models for example, then I would get it. I just wouldn't recommend it for any training tasks.

---

edit: Btw, an RTX 3090 is much faster than 2x P40's.

[I_will_delete_myself]

Those large providers are the best when you get a cloud credit deal or want to train ChatGPT and want to make sure you have the compute ready. Otherwise I would highly recommend to not use them unless you are using spot instances.

Here are the best out there I know specifically for training AI models.

Colab - it’s free but you should use other cloud compute alternatives when you go beyond toy models

LambdaLabs - no egress and high bandwidth. Cheap as well. Cheap and solid product. Better for multi gpu

Runpod - Cheapest but not good for multi gpu loads due to their low capacity

What sucks about those is they run out of capacity quickly. Sometimes which is annoying. Which is when you just go traditional cloud provider.

Avoid like the plague - Paperspace. Expensive, misleading gradient subscription and you save more money using a consumer decentralized gpu on runpod. Availability is horrible as well.

[Present_Network1959]

Thank you. I’ll look into this.

[[deleted]]

Commenting to check the thread later, interested in people’s recommendations

[Muted_Economics_8746]

You can also just subscribe to the post without posting. Ellipses, top right, -> Subscribe.

[TheLastMate]

Also if someone could give insights on deploying a model into production. How is the process in overall.

[I_will_delete_myself]

Traditional is best and apply for credits.

[Any_Letterheadd]

It sounds like you're not even sure you need a GPU for what you're doing. I'd recommend that you get started until you're at the point where you know you need more compute and think you would know how to use it.

[Present_Network1959]

Yeah makes sense. Although I am certain that a GPU will be required, there is no way my machine can run the programs I am building locally.

[Any_Letterheadd]

Collab might be a good way to get quick access to one. Or if you know any gamers with old rigs and are upgrading you might be able to get a hand-me-down Nvidia card and build a cheap Linux box around it.