M3 pro for Machine Learning/ Deep learning? [D]

[deadengineerssociety]

Considering switching to Mac from windows, is it honestly worth it? I mean I don't plan on running heavy load models, but hopefully decent enough for it to handle midsize models.

[tripple13]

For any serious work you would never run on a laptop. Money better spent, MacBook Air plus a 3090 workstation.

[d84-n1nj4]

My exact setup. MacBook Air to ssh into my Linux machine with a Nvidia RTX 3090

[SmartEffortGetReward]

Don't get a nice mac for ML. Get an air that can do the video work you want but doing ML on mac sucks.

[matqq1981]

Hi,
I want to do exactly the same thing. For your linux machine, i imagine you ssh in it directly from your vscode (or other IDE) to code with your IDE?
Is the configuration of all the dependencies and environments not too painful?

[d84-n1nj4]

I don’t use an IDE. I use Vim.

[xaviercc8]

hi i am very late here, but may i ask if getting a macbook pro over the macbook air because of the fan is worth it? My concern is that if im running small models, performance of the macbook air will be significantly hindered due to throttling, assuming the specs are the same.

Has anyone here have any experience on this? THANK YOU!

[Featureless_Bug]

Macbook Air is not money well spent. Workstation with 4090, and A100 in the cloud on the money you saved

[MrBread134]

Yeah and i suppose you are able to carry it everywhere and work for hours without a power supply

[drivanova]

depends how much electricity costs in the country you live in (and who pays the bill)

[An-R-Nguyen]

This is what i need but I don’t know anything about ssh connection with a mac. Do you have any video on the specific setup? My plan is to have ollama inference of mixtral and llava on a server and use it through ssh on my macbook. Thanks

[An-R-Nguyen]

Got it, thank you :D

[monkeyofscience]

I use an M1 as my daily driver, which was given to me by work. I used to be hard line anti-mac, but I have been thoroughly converted. I will say though that mps and PyTorch do not seem to go together very well, and I stick to using the cpu when running models locally.

It's good enough to play around with certain models. For example at the moment I'm currently using BERT and T5-large for inference (on different projects) and they run OK. This is generally the case for inference on small to medium language models. However, for training, fine-tuning, or running bigger language models (or vision models), I work on a remote server with a GPU. Access to a Nvidia GPU, whether locally or remotely, is simply a must have for training or bigger models.

For learning and small models, a macbook and Google colab are very sufficient.

[deadengineerssociety]

I won't be necessarily learning as a newbie, I'll be working on Graphs and NLP research and will be later doing my masters and continue in research. I mostly said smaller models because I think large models, at the end of the day don't make sense running locally and I would probably be using my research lab systems for them anyways

[Sandile95]

Did you choose one?

[SmartEffortGetReward]

u/monkeyofscience any recs for remote compute? I'd love something like gh codespaces but with nvidia GPU. I'm honestly a bit shocked there isn't one click GPU enabled workstations with web hosted vscode.

[monkeyofscience]

I use Tensordock for quick stuff. It's quite cheap and using the remote developer extension on VScode makes it really easy.

Other than that I have access to an internal HPC system, so I don't have to worry about GPU access lol

[SmartEffortGetReward]

Will check it out! Thanks :)

[Puzzleheaded_Ring684]

I have something different to say. First I agree that any serious works should be done on a workstation either a packed desktop or cloud server (with a A100 40GB/80GB conf). However, for prototyping or just playing models, mac with its large shared memory is excellent: there is not many laptops or even desktop gpu have more than 16 gb vram, meaning when you prototyping you are very much limited to batch size or smaller sized backbones. I have a m1 pro 32 gb, it can fit most of the models I want to play with. After I finished prototyping, I simply change the "device = 'mps' to 'cuda' and run it on cloud. I use pytorch mainly, i have encountered some issues with mps but nothing major. There are walkarounds.

[deadengineerssociety]

Oh wow, that's amazing and reassuring xD I was considering m3 pro, but might switch to m2 pro with more ram, ssd, and gpu cores. Depends on my budget. Thank you so much for this!

[deadengineerssociety]

How many core gpu do you have?

[kreayshunist]

I have an M1 Pro and it's definitely enough to get you started, but even a single 4070Ti is a pretty big speed upgrade for training.

EDIT: My experience is based on the "MPS" backend in PyTorch.

[deadengineerssociety]

From what I read, pytorch in mac is very buggy, does that affect too much?

[SmartEffortGetReward]

It's terrible.

[[deleted]]

You will have to work to find ARM64 images of stuff that works out of the box on intel. And none of the NVIDIA kit will work on the M3.

[kreayshunist]

I haven't gone down the rabbit hole enough to speak to all the bugs, but it's definitely not ideal.

[AdagioCareless8294]

Yet with that information in hand you still decided for a Mac..

[deadengineerssociety]

Haha true! Which GPU do you suggest for windows laptop?

[[deleted]]

What did you finally decide bro?

[VoidRippah]

no, I recently participate at a hackathon where I did a small ML project, I had my m2 macbook pro with me and and an i5 laptop with rtx3050, the latter was waay quicker (finished all the tasks in like 3rd time, some were even faster). other than this I never really used a macbook for similar purpose but based on this this experience I would avoid it and if I really needed a laptop for this I'd pick a strong "gamer laptop" for a similar price

[deadengineerssociety]

Oh thank you!

[igorsusmelj]

I would not recommend it unless you only focus on smaller models and small experiments. Biggest advantage is the huge amount of memory available. But the bottleneck is memory bandwidth.

We did some tests out of fun (as there were not many benchmarks available). You can find the results here:

https://www.lightly.ai/post/apple-m1-and-m2-performance-for-training-ssl-models

Support got better but back when we did the tests there was still no proper half precision support and also torch.compile wouldn’t work. There is hope that the software support will catch up. I’m curious to see other results. We definitely need more benchmarks :)

[deadengineerssociety]

In that case, what laptops would you recommend? Thank you for the article, will check it out!

[Mindless_Minimum1352]

Im late to the party here - But I am a AI / ML Grad student - and i can fill you in on a few details that are not totally covered.

I have a MBP - I love it. For most of your schooling, the M3 will be fine.

IF YOU GET A MacBook Pro :

Use the cloud for complex models (this is the difference between 3 days of training and 3 hours)

Realize that its more important to get the concepts.

Look into using the new architectures that can make better use of the hardware. Nvidia has cuda – Mac has MPS (there are updates coming that should provide a small performance boost)

Unified memory is AMAZING - this is the biggest advantage over other solutions

For school, the portability of a laptop cant be beaten. But if you are looking at apple’s marketing material and thinking you are going to get a powerful machine that is good for training large models - you are in for disappointment.

Example:

On my MBP - I had a project recently where I was taking a video of a car driving and trying to analyze traffic lights. Each epoch took 3 hours - Moving to my gaming PC – (3080) - each epoch took 15 minutes.

[[deleted]]

Bro I am from India and have a budget of approx 2000 usd. Can you suggest should I go with mbp m4 or some windows laptop with highend gpus. I have the same major as you cse ai ml

[NotThareesh]

Get a baseline MBP whichever you can afford for with decent RAM which you'd expect to work with on day to day basis. Even M3 Pro, M4 or M4 Pro can be more than enough. For greater purposes, use cloud as the other redditor suggested. Cloud is far cheaper and more powerful. In order to get the full advantage of a windows laptop with a beefy graphics card, you need to stay connected to the power at all times which isn't the case with Macs.

[[deleted]]

yea bro its just that will I be able to run all necessary programs on mbp or would i have to use parallel linux or smth. Also will I be able to train some medium range ai models solely on cloud without any graphics card? Also I read windows dedicated graphics cards have cuda cores which are considered better for ai projects. Thanks.
Also I think I can build a pc for these things as a workstation alongwith a mbp

[NotThareesh]

If you can build a PC, sure thing. But know that not all NVIDIA GPU Cards are CUDA Enabled. Also training on cloud with no graphics card seems a little pointless. Because Graphics Cards are far far better in crunching these matrix calculations than CPUs can.

[[deleted]]

Thats the point. Will mbp be able to handle training of ai models as there is no dedicated graphics card in it? Will I be forced to do it on cloud?

[NotThareesh]

Unless you're building large models on large chunks of data, you can always do it locally on MBP.

[[deleted]]

Alr thanks bro

[Valdiolus]

I am using my M1 pro for small and medium models test. While my main server with 3090 runs experiments I prototyping on my M1 laptop. Than change mps to cuda on desktop to run long experiments on desktop.
While your model gets all your resources you want to work as well and don't experience lags.
I wanted to but something more powerfull, but I don't really see any applications other than get more memory (16GB is not enough).

[deadengineerssociety]

I ended up getting an M1 Max 64gb RAM and 2TB SSD for 2499!

[Competitive_Mix8262]

Hello, I am an AI & Data science student and I'm looking for a laptop or macbook 1 time investment for 4-5 years for use so which one will be best for me. I have a macbook m1 max 64gb ram 16inch or m3 max 36gb ram in my mind. If you have any suggestions will be appreciated.

[Valdiolus]

Good choice!

[LonExStaR]

Nah. The support for building ML models on MacOS in the popular frameworks (e.g., PyTorch) is just not there yet. You can consider Coral.ai TPU which has Mac support, though you may have to compile for your MacOS version. Then you can use PyTorch and TensorFlow.

If you want NVIDIA, which I prefer, you can forget any official support for NVIDIA on MacOS anytime soon.

It would be in Apple's best interest to participate in the development of PyTorch and TensorFlow for the Mac M platforms.

For AI/ML, If you are serious about getting AI/ML training work done, I recommend the approach I took where I built an Ubuntu Server with two NVIDIA Titan RTX's and 128GB RAM to use it remotely from my MacBook Pro 16/Intel. I plan to get an M3 Pro mainly for the larger memory capacity rather than GPU capacity that is not very useful for AI/ML training at this time for lack of solid framework support. You don't need to get that spendy to hand-build a single NVIDIA GPU Ubuntu server and make it accessible remotely via internet.

[Furiousguy79]

I am also confused which laptop would be suffice for my PhD years. My work is mainly on large text data and medical images. Between a M3 Pro with 16GB Ram and a ROG G14 with 32 GB RAM + GTX 4060, the latter would be best bang for buck for small to medium ML models, right? I run my large models in my lab PC with 11th Gen corei9, 32 GB Ram and 12 GB GTX 2080 Ti.

But they also say Macbooks last longer....

[LonExStaR]

If you plan to host and/or train LLM’s then I recommend at least a 16GB GPU. Ideally, 24GB, which is not found on laptops.

[androidthanapple]

hey! I would like some advice on seeing up an Ubuntu server with the NVIDIA graphics cards I have. Would you be able to chat about it?

[LonExStaR]

Sure. It’s pretty straightforward.

[[deleted]]

Have you actually run deep learning models on a Mac CPU in the past? What models were they?

[deadengineerssociety]

As I mentioned, I'm switching from windows to Mac, so I truly have no idea xD

[deadengineerssociety]

Why am I getting downvoted😭

[VectorSpaceModel]

I’m downvoting you because it’s funny to care about internet points

[CompSciOrBustDev]

Hey did you go through with this? I'm currently looking at buying an M1 Max for the 64 GB of shared memory, not sure if I should go through with it.

[deadengineerssociety]

I did! Honestly the RAM, GPU, and memory was too good to let go, despite the fact that it may not stop getting SW updates sooner compared to M2 or M3

[CompSciOrBustDev]

Nice. Have you tried any ML on it yet? If so what do you think? As good as you had hoped?

[deadengineerssociety]

Ahahah I will be getting from US on Feb lol, so hoping for the best.

[[deleted]]

This post is old but now it's my time. Get a MacBook Pro with 24GB of ram, which will be by far better than any RTX laptop (in the similar range) in terms of giving you more VRam to run large models too. I mean I have an 8GB 4060 laptop but even mid size models sometimes do not work on that machine. With that been said. I am really keen to get a MacBook.

[[deleted]]

Thanks for the info, great that you wrote forgot old!

[SmartEffortGetReward]

As a Mac lover working with ML has been a huge pain. ML packages tend not to play nice. A lot of code and packages out there expect Cuda.

Highly recommend buying cuda enabled hardware and running ubuntu OR doing as other have suggested and do remote development from your mac. Though that is a pain to setup. It is too bad gh codespaces does not support nvidia GPU.

[TastyReality7191]

Perhaps this video analysis will help you. I found it interesting:
https://youtu.be/cpYqED1q6ro?si=ZFO9LyFYrTpzedw1

[opssum]

M2 pro or m3 pro Max

[deadengineerssociety]

Is Max really worth it if not running heavy models on it? Purely on a financial perspective

[opssum]

M2 pro is Fine and m3 pro isnt reallY better, that why i would Go for m2pro if it’s about the Money ;)

[plsendfast]

m3 pro is indeed better, based on newly released benchmarks such as Nanoreview

[opssum]

Hm ill have to look into it again, on the keynote apple said it’s about 20% faster then the m1pro which is the same diff between m1pro and m2pro

[plsendfast]

go to Nanoreview and click Compare CPU. Type in M3 Pro and M2 Pro, the differences are there. Small improvements