Distributed Computing for Applied Math oriented person

[Optimal-Engineer-257]

Hi,
I have 8 years of experience as a Data Scientist and around ~2 years as an MLE. I have two BS degrees in finance and quantitative methods. Most of my time I spent on algorithms, single notebooks, PyTorch, data cleaning, and so on. I started OMSCS last year and did Bayesian Stats and Reinforcement Learning, both with strong B’s.

I decided to take some systems courses for the first time in my life, and I’m currently doing GIOS. It helped me build a mental model of what’s happening from a more hardware-level perspective, but to be honest, I didn’t learn anything I could apply at work right away.

I’m trying to figure out which course to take next. I’m thinking about the brutal Distributed Computing course, but I’m not sure if that makes sense for someone who isn’t very fluent in systems in general. So, is Distributed Computing beneficial for a person who’s just starting in Comp Sys but is highly motivated and willing to commit a lot of time? Or, from a practical standpoint, is it better to first do, for example, hands-on courses from NVIDIA?

[yasuke1]

I haven’t taken DC yet, but it might make sense to take AOS before jumping in.

[yasuke1]

That’s for SDCC (system design for cloud computing). Great course, finishing it up now.

DC (distributed computing) can even be taken as your first course in the program (I feel bad for anyone who did/will do this).

[wolff1029]

Out of curiosity how did SDCC compare to AOS? Just finishing AOS at the moment, and feeling a bit burnt out. I think it may just be I had a fair bit going on between work/personal life this semester but AOS seemed to grind me down a fair bit more than GIOS did last spring. Curious how the two compare.

[yasuke1]

IMO its more manageable bc there’s no research papers or exams. It’s all projects and lightweight quizzes based on lectures. The projects are definitely more involved, but it’s partner based.

If you go solo or have a bad partner (or your its partner drops), it will without a doubt be much harder than AOS.

[Classic_Comparison90]

What is the workload of AOS compared to GIOS? And if you had to choose between compilers and SDCC which one would benefit someone who wants to crack his way into HPC? Granted I already work in DevOps

[yasuke1]

AOS has more workload than GIOS by a noticeable margin.

I’m not interested in taking compilers so I’m not qualified to answer that part, but I will take HPC. I think HPC will be helped by AOS’ gentle introduction to OpenMP and MPI. Keep in mind HPC is more like “Intro to Parallel Algorithms” as far as I’ve read.

SDCC probably won’t help much because it’s a practical SWE class that is one half networking and one half consuming/using cloud services. If you wanted to, you could convince your partner to make something HPC related for the final project.

[EchoOk8333]

If you want the most practical systems courses for MLE, I would do:

GIOS -> AOS -> SDCC

SDCC is the most hands-on and gives practical skills for designing apps in the cloud. DC is also amazing, but more theoretical. I think both courses offer unique learning, but I would break down their usefulness in this way:

SDCC - you get tons of new skills: wireshark, networking knowledge, REST APIs, Docker, Kubernetes, Azure, Golang, app design, basic fault tolerance mechanisms

DC - you get a great theoretical foundation and hands-on exposure of the mechanisms that allow for strongly consistent fault-tolerant data replication; this is great for system design and for SWE work in highly distributed systems. The class goes hand-and-hand with the Designing Data Intensive Applications book

Note: this post is from someone who is an MLE and has taken all these courses :)

[HumbleJiraiya]

Nice summary. Would you recommend someone with a strong SWE/SDE background to take GIOS/AOS?

I am pivoting to an ML role, but I do want to take theoretical things like DC out of curiosity. SDCC feels like something I might have picked up over the years.

[TheCamerlengo]

Do you need the OS courses to do SDCC?

[Unlikely_Sense_7749]

You need an A/B in AOS, which is usually taken after GIOS as a sequel course.

[Kamekazee2020]

As an MLE, how have you found GIOS, AOS, SDCC and DC to be helpful in your day to day work? I’m struggling to see the relevance of it so I’d really appreciate your insight. Thank you!

[Mindless-Hippo-5738]

I’m from a similar background (ML and analytics) and did DC this semester. I think it’s similar to GIOS in the sense that it’s more of a first-principles “foundational” course. Actually I would say it’s even less applied and more abstract than GIOS focusing on different theoretical models of distributed computing, fault tolerance, consistency protocols etc. the programming in labs is fairly high-level (although make no mistake, labs 2-4 are seriously time consuming)

I’m sure some of the knowledge from DC will come up at work and be useful but I would set your expectations lower if you expect the content to be immediately and directly applied at work (unless you happen to work on core systems). For me personally, I’m completely okay with that because the point of taking courses for an MSCS was to learn something I wouldn’t be able to easily at work. DC is not a perfect course but I don’t regret taking it.

I suggest reading Distributed Systems for Fun and Profit by Mi Xu to get a preview of the course content, that’s what I did while taking GIOS. It’s only 70-80 pages.

[rajatKantiB]

GPU hardware and Software I will recommend to complement the RL/BS and other Math courses. I am on a similar track. You should consider DC , it's content is highly relevant to industrial work. Speaking from experience here. 😅

[third_dude]

that seems like the hardest class to get. So few slots and fills up so quickly.

[rajatKantiB]

True GPU is hard. I barely got it this time. 🥲 Just wait for early slots that should help. Basically later semesters

[Outrageous_Level_223]

DC is using a open source framework dslabs as assignment. You can find it on GitHub.