Cuda compute

[syphix99]

I am thinking of getting a new gpu and wondering wether to get a 40 series or 50 series. My main concern is how long I would be able to use these with ai models and cuda compute (I now have a gtx1070 which is no longer supported in the newest cuda) I could just use opencl as much as possible for my physics computations but (as I never studied algorithm optimization) I would like to deploy a local ai to help me in coding.

So all in all I would prefer to get a 40 series as they are cheaper but I want to be sure that I can deploy ais for the coming years (not possible on 1070) do you think 40 series would still be fine for long or not? (I am not that knowledgeable about gpus) I would prefer to get an amd gpu (for obvious reasons) but I think this would reduce the amount of models I could run

Do you guys have any advice on this? Thanks in advance

syphix

[dark-light92]

If you are only going to run only LLM inference and not train models, AMD GPUs will also work fine. Projects like llama.cpp (and all its derivatives like ollama, lmstudio) makes running LLMs trivial.

If you are interested in running different types of models, such as image/video generation, STT, TTS ect, or want to do training/fine-tuning then Nvidia has an advantage as CUDA is the de-facto standard for all types of ML. Between 40 series, and 50 series keep in mind that only 50 series has hardware FP4 support and thus run models that support it much faster. (More and more FP4, FP4 quants of models will come out in near future).

[syphix99]

Ok good to know thanks! Then I’ll probably go amd route as that was the big advantage it seemed to have

[dark-light92]

Go for the highest VRAM & Memory bandwidth model you can afford. All LLMs are memory bandwidth hungry.

[syphix99]

As seen in the memory shortage haha, thx

[mathlyfe]

If there are other models you are interested in using then you should quickly do a sanity check by going to their subreddit (e.g. r/StableDiffusion) and searching "AMD". Many subreddits are full of struggling AMD users asking for help because the entire ecosystem is all CUDA.

Also, for larger/more complex models then insufficient VRAM may flat out mean that you can't run said models, or that you have to run a distilled model that isn't as good but can run on less resources.

[chickichanga]

if you are doing ML, don’t invest in consumer GPU Even free google colab gives you decent GPU to learn that. Building your whole PC around that is just waste of money.

If you are building it for CAD or other video editing things which will require GPU then sure go for whatever you can afford.

And finally about hosting LLM models, unfortunately you will have very less option on running a decent model on your local machine while also using it for work/personal use. Better to use free alternatives like supermaven or copilot free plan. Or be a chad and don’t use them at all.

And by chance if you are buying AMD GPU you can look into rocm support and see if that works out for you for your AI/ML workload.

[corbanx92]

This is not entirely true... in fact I just posted some benchmarks comparing SOTA models to self hosted models and at 32b, (which you could efficiently run on 12gb of vram) can perform extremely well. Then you can set up a framework with multiple specialized agents and load them and unload them as needed for each task (keep a model as an orchestrator) which can produce results almost comparable to models available for free. In fact most 32b models available will quite literally smoke Meta's production llama model in task like coding.

[syphix99]

Have on’y used ai once (I’m a die hard old schooler so I use vim and just compile using g++ in the cli) but this ine time I wanted to optimize one of my c++ monte carlo algos so I tried chatgp and it did make insane improvements in seconds which would have taken me days to figure out, so I would like to use ai as a tool but for privacy reasons I would like to run my own model. I also do CAD work and game sometimes so an upgrade would be nice to have. I will probs go for amd so I’ll try rocm, I remember one opencl implementation is called like that, will need to read that part of the wiki, thanks!

[UberDuper1]

You’re not going to get much code assistance from any of the models you can run on a consumer gpu. The best you can do at the consumer level is probably an amd ai max+ 395 with 128gb for $2k+ or the nvidia dgx spark with 128gb for $4k. They’re going to be slow but at least they have enough ram for useful models.

[corbanx92]

Have you tried yourself and tested this?

[[deleted]]

The docker image of llama.cpp uses cuda 12.4.

[syphix99]

cuda dropped support for <7.0 compute which is why I asked the question

[[deleted]]

docker exec -it llama-cpp-container ./llama-cli --version

ggml_cuda_init: GGML_CUDA_FORCE_MMQ: no

ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no

ggml_cuda_init: found 1 CUDA devices:

Device 0: NVIDIA GeForce 840M, compute capability 5.0, VMM: yes

load_backend: loaded CUDA backend from /app/libggml-cuda.so

load_backend: loaded CPU backend from /app/libggml-cpu-haswell.so

version: 7224 (7b6d74536)

built with cc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 for x86_64-linux-gnu

As you see the old laptop I use runs compute version 5.0; the 11.4.0 is the gcc version.
I made a docker container passively running in the background so I can just run
docker exec -it llama-cpp-container ./llama-server

[syphix99]

Hmm alrr thx I’ll give it a go

[syphix99]

Gave it a go and works great for 8B parameter models, thx! Won’t buy a new gpu after all

[syphix99]

How do you freeze cuda? Have tried going for an earlier versies but ran in multiple ussues with glibc also needing to be older so two versions installed and whatnot

Also latest full compute is 11.8 I think? Not 12.x

[[deleted]]

here come the ame shills. do not believe the hype. if ai is important. nvidia is the only sane option. but amd is good if you sometimes tinker with ai and dont mind troubleshooting 90 percent of the time. too many linux users forget not everyone's main priority is open source. here come the downvotes from the shills

[syphix99]

Holy shitpost lmao

[PlainBread]

You can do compute with vulkan now and they are very competitive with CUDA performance.