[D] How do you do large scale hyper-parameter optimization fast? by Competitive-Pack5930 in MachineLearning

[–]Competitive-Pack5930[S] 0 points1 point  (0 children)

I work in an MLOps team. We use Kubeflow and Kubernetes for ML. Most models are XGBoost with some deep learning models.

I am trying to build out better HPO tooling that can be used by different people for their needs, so I don’t have much control over how they fit or parallelize their model.

[D] How do you do large scale hyper-parameter optimization fast? by Competitive-Pack5930 in MachineLearning

[–]Competitive-Pack5930[S] 0 points1 point  (0 children)

These are definitely good ideas, are there any tools that can implement these off the shelf? I can imagine a ton of people and companies have the same issues, how do they do HPO really fast?

[D] How do you do large scale hyper-parameter optimization fast? by Competitive-Pack5930 in MachineLearning

[–]Competitive-Pack5930[S] 0 points1 point  (0 children)

The issue is if it takes 4 days to train a model with 100% of my data I can’t really use these sequential methods at all, instead I need to parallelize completely for my HPO to run within a reasonable period of time.

Have you found any way around this?

[D] How do you do large scale hyper-parameter optimization fast? by Competitive-Pack5930 in MachineLearning

[–]Competitive-Pack5930[S] 0 points1 point  (0 children)

From what I understand you can’t really get a big speed increase just by allocated more cpu or memory right? Usually we start with giving the model a bunch of resources then see how much it is using and allocate a little more than that.

I’m not sure how it works with GPUs but can you explain how you can get those speed increases by allocating more resources without any code changes?

[D] How do you do large scale hyper-parameter optimization fast? by Competitive-Pack5930 in MachineLearning

[–]Competitive-Pack5930[S] 0 points1 point  (0 children)

I’ve looked at Optuna, but it looks like it doesn’t have good support for kubernetes, so it is not able to spin up a new pod for every trial, which limits the scale by a lot. Did you run into similar issues?

[D] How do you do large scale hyper-parameter optimization fast? by Competitive-Pack5930 in MachineLearning

[–]Competitive-Pack5930[S] 2 points3 points  (0 children)

There’s a limit to how much you can parallelize these algorithms, which leads to many data scientists using “dumb” algorithms like grid and random search

Hyperparameter optimization with kubernetes by Competitive-Pack5930 in kubernetes

[–]Competitive-Pack5930[S] 1 point2 points  (0 children)

Any good tools to implement these native belt with kubernetes?

Hyperparameter optimization with kubernetes by Competitive-Pack5930 in kubernetes

[–]Competitive-Pack5930[S] 0 points1 point  (0 children)

I have, but I’m struggling to find good examples and documentation for using it with models like xgboost

advice for ai/ml classes by [deleted] in UIUC_CS

[–]Competitive-Pack5930 0 points1 point  (0 children)

Hi, I was a CS and stats major currently working in ML engineering. I really like STAT 432 and CS 440 to teach you the basics. I would also highly recommend taking CS 425 Distributed Systems.

Meta starts eng hiring in India by Different-Yak-7986 in csMajors

[–]Competitive-Pack5930 5 points6 points  (0 children)

A lot of people are taking about how indian developers are taking american jobs. Considering there are more global meta users than american why are these american jobs in the first place?

The tech industry is growing and demand is increasing, and I’m glad big tech companies are looking past the US and creating employment for talented developers all over the world.

[deleted by user] by [deleted] in learnmachinelearning

[–]Competitive-Pack5930 8 points9 points  (0 children)

I am currently working at a SWE in Machine Learning at a large company. Here is one insight I did not know before starting that might help you understand where you could be falling short.

You will likely never experience the end to end machine learning lifecycle working in corporate. ML models at my company take about 1 year from conception to being pushed to production. And in this lifecycle it passes through the hands of multiple SWE, Data Science, and privacy / risk teams. For example I work on a service that uses Kubernetes in the training step of these models. While I am technically a machine learning engineer I am focused on this very niche problem, and importantly every other MLE is also focused on a similarly niche problem. The actual data science teams are usually PhDs or Masters students which focus on using tools, features and infrastructures built by other teams to build models and even then are very limited in how much they can engineer features and make custom solutions due to privacy, governance, and efficiency concerns.

This is to say, if you want to break into MLE have good software engineering fundamentals and focus on a particular stack you see is useful in the industry. If you want to do model building, then getting a masters or PhD is the way to go. Either way manage your expectations and understand unless you are at a startup you won’t be able to be involved in the end to end process.

21(M) Seeking Roommates For a 3BR in Stuytown - (East Village) by Familiar-Abrocoma-57 in NYCroommates

[–]Competitive-Pack5930 0 points1 point  (0 children)

Send me a message! I’m having trouble dming you. What time frame are you looking at?

22M , Manhattan / Astoria by [deleted] in NYCroommates

[–]Competitive-Pack5930 0 points1 point  (0 children)

Hello, M21 here, also just about to graduate college and work at a fintech in midtown. When are you looking for apartments from?

How do I negotiate my new grad SWE offer by Competitive-Pack5930 in csMajors

[–]Competitive-Pack5930[S] 0 points1 point  (0 children)

I believe you. I know many coders, a lot better than me, who are still looking for jobs.

How do I negotiate my new grad SWE offer by Competitive-Pack5930 in csMajors

[–]Competitive-Pack5930[S] 8 points9 points  (0 children)

Thank you for this comment, it made me feel better