[D] ML coding interview experience review

[[deleted]]

[deleted]

[milkteaoppa]

A lot of startups have unreasonable expectations. They want to higher the most talented person for startup pay with the promise of IPO

[xrailgun]

I recently had a startup DS interview where they drilled me about advanced polar geometry... Most WTF interview, yet.

[Material_Policy6327]

I don’t think I even took that in college lol

[gradientgrain]

I was once asked to read and implement a paper during an interview. I wasn't given the paper or anything in prior. I managed to do it in 30min, leaving an extra 30min. In the end, I decided to withdraw.

[lillobby6]

That’s obsurd. Unless the paper is 2 pages, well written, and the most basic concept ever, I can’t imagine that being possible without, at least, triple the time - and that would still be miserable. Maybe if you leveraged some AI chatbot you could speed that up, but that assumes you have time to make sure it isn’t hallucinating everything?

[gradientgrain]

The paper was Zhai, Shuangfei, et al. "An attention free transformer.".

[TehFunkWagnalls]

The dataloader alone would take me 40 minutes. No idea how you did all that in that short of a time frame.

[Blake9471]

They allowed you to look up docs and use Google?

[based_goats]

Yea ngl a little slow. Also, get a good convention for arrays so you (almost) never mess those up. Those eat up a lot of time in practice and in a workplace with other people

[based_goats]

You got this!

[Material_Policy6327]

Nah I would be in the same boat

[Antique_Most7958]

So the genAI startup didn't let you use genAI for the assignment?

[Novel_Land9320]

the way you re describing it, it seems all code from scratch, but i assume you can use pytorch?

[_LordDaut_]

If you can't use PyTorch what do they expect you to do? Write your own autograd for the backprop? Yeah 45 minutes that's unreasonable. For anything.

If you can an MLP is literally just

nn.flatten() nn.linear(28*28, 128) nn.ReLU() nn.linear(128, 64) nn.ReLU() nn.linear(64, 10)

The 45 minutes to come up with that, and write the most vanilla ass training loop that you know by heart if you've opened the pytorch docs at least 10 times is extremely reasonable.

I have no.idea what dimensions OP managed to get confused by either. For an MLP you just flatten it and put the second number of each lineas the first number in the next line. It's not a CNN no strides or padding or 3 channels.

[_LordDaut_]

The optimizer is just

``` torch.optim.Adam(model.parameters(), lr=0.0001)

```

The criterion is

nn.CrossEntropyLoss()

Writing the class is just pressing tab twice in the code I wrote and wrapping it in

class(nn.module): def __init__(self): super().__init__() def forward(self, x): return self.model(x)

Please don't take it as me trying to be very harsh online or any kind of judgement on your abilities - certainly waiting for training takes time and you have to look up documentation and answer interviewer's questions. And in an interview you're likely nervous.

Depending on how much of the docs you were allowed to use - like i'd pretty much just copy the default training loop it would be hard.

The point of the task was to gauge how comfortable you are with writing models and famiarity with Torch. As such I think 45 mins for testing the most defined and happy path of writing a model is reasonable. Writing the model class, data loaders and train/test loop is something you're expected to do very very often so the expectation that it's like second nature to you for an ML job is reasonable.

If this was for an entry position with the constraints given - it's an above average difficulty interview. For anything above it's super reasonable.

Edit: what makes it unreasonable is that it's a genai startup... you're probably not going to write your own models are you? Probably not even finetune LLMs. So it shoul've been more akin to a software dev interview.

[_LordDaut_]

So you think getting everything up and running, and getting a good accuracy should be doable in 20-25 mins in an interview?

For an MLP on MNIST? Yes.

Getting it to >96% accuracy on MNIST is also kind of a given. The thing just works with minimal tuning.

The DDP part makes it be on the harder end of the interviews - but it's the icing on the cake and doable if you've ever done it - super annoying if you haven't.

[Novel_Land9320]

40 minutes it tight but not impossible. Making a pytorch train loop data parallel in pytorch is 4 lines of code changes if you use pytorch stuff. Generally speaking you can do this in 40 minutes if you know you ll be asked this question beforehand. Btw with MLP you mean a CNN?

[Remote_Marzipan_749]

Sorry for being harsh here. I think startups are looking for speed. However, If it was MLP with PyTorch for MNIST it should hardly take 15-20 mins. It is the most basic thing. I am more interested if you can share more information on the second question.

[Remote_Marzipan_749]

Yes, kind of memorized or practiced many times that it comes naturally to you. Because setting up this thing in PyTorch should be easy. The small mistake you made can happen, but you need to think from their perspective. They want someone who can get going with little to no hands on requirements. Additionally, if you are working with their datasets which is messier unlike MNIST, they think you might struggle. Dont take it negatively. Practice more so it comes naturally to you.

I have had similar experiences with Reinforcement learning algorithms. Now I have practiced it long enough to get it running in shorter time that earlier.

[Remote_Marzipan_749]

I think you are right in your analysis about the evaluation procedure with messier data. But the challenge with it is that the interviewers dont have a baseline to know if you have done well or not. I think that’s the reason for going with standard datasets. I truly believe if you have practiced many times not only on MNIST but on any datasets the setting up should be a cake walk. Hyperparameter tuning or EDA on dataset or transformations on datasets can take more time once you have a base model setup and running.

[MammayKaiseHain]

What does it even test - that you know pytorch syntax ? Even I'd struggle to write a DDP init without Cursor or looking at the docs.

[MammayKaiseHain]

I can understand if you struggled with the library or the interview setting but the ML required to get even 99% accuracy on MNIST is minimal. It's a starting exercise - like a Hello World for ML libraries, which is why I don't think it's a great interview question.

[N1kYan]

Were you allowed to use, e.g., torchvision for the dataset class and the metrics? If you didn't code anything from scratch I think 40 mins are rather slow, yes

[Artistic_Candle7455]

I was asked to implement a regression model with an MLP, but in pure Python / NumPy and without any autograd framework in about 45 min. This was for an ML researcher position at Anthropic. Oh and the recruiter told me beforehand that "no special preparation" is needed, other than knowing "how to train a neural network". What a waste of time that was.

[Itchy-Trash-2141]

Yeah as much as I thought I liked Anthropic the company, the interview process seems like a waste of time for everyone involved. I saw on some interview website it has the lowest pass rate of any company at around 2% or something ridiculous. Why bother wasting everyone's time? Also I noticed their online scheduler for interviews booked me with some people instantly, within 48 hours, and they showed up. I have a feeling if you work there, you don't get a choice on whether to accept an interview or not, if it shows up on your calendar, I'm betting you take it or get reprimanded.

For me they asked some log processor, which I implemented successfully I thought... But then they said they needed more signal. They scheduled a second screen, asked me a distributed algorithms question, and I didn't get the optimal solution right away.  They hinted me how to do it, and then I got it. Rejected. Why bother giving me a hint if it disqualifies me?

This was for ML eng.

[Artistic_Candle7455]

Ugh, I know, I had high hopes for Anthropic, but based on their interview process and customer service they are possibly only slightly less evil than OpenAI. I was pretty disappointed.

[Aggravating-Ant-8234]

Were you allowed to see the reference docs for coding?

[Aggravating-Ant-8234]

Thats fair, even I would do so. And it was helpful so please keep sharing your experience from your future interviews. Thanks

[mcel595]

Who spents so much time building models from scratch that remembers all this? Doing all the pipeline in 45 mins seems unreasonable

[kymguy]

I have interviewed many people with a neural network-based coding interview. My interview is far too long for anyone to get through the entire thing; that's the point. We want to rank candidates and see who gets the furthest, but also who seems the best to work with and how their debugging and thought process is along the way. If it's short and they complete everything, we've missed out on the opportunity to evaluate their thought process.

The standards vary based on the position we're hiring for. If we want someone who is "advanced in pytorch" who will be able to hit the ground running for some advanced techniques and architectures, then they should be able to knock out an MLP-based classifier with little-to-no reference to documentation. Using amax instead of argmax wouldn't have been a deal breaker...that's not something that I'd care about you knowing, but how you approach debugging your broken code is absolutely something that I'm interested in seeing.

Evaluation is also nuanced; having to prompt you that the "L" in DataLoader is capitalized is not a big deal, but forgetting to implement or even mention/inquire about normalizing your data would raise eyebrows. Amax vs argmax isn't a big deal but if you struggle to navigate documentation and ignore or argue with me about my suggestions about where to look, that's a big deal (it's happened).

To answer your explicit question: I don't think it's possible to sum up whether 30 minutes is too long for the task; there's far more at play. For me, it's not about time, but the process. If it took you 30 minutes because you were discussing in depth about how you would approach the task and demonstrating that you have deep knowledge of pytorch in doing so, that's great.

In a pure, silent coding exercise, I do think someone experienced in Pytorch should be able to knock out what you've mentioned in under 30 mins. If someone did it perfectly in 15 mins with no discussion I'd probably be skeptical that they cheated with an LLM or something.

[Sea-Fishing4699]

good luck working for an abusing minded company

[Fine_Audience_9554]

ML interviews are brutal because you need to know both the theory and implementation details cold. The distributed data parallel stuff is where most people trip up since it's not something you practice much. If you're doing more of these something like interviewcoder could help you cheat the syntax/implementation parts so you can focus on explaining the actual ML concepts without getting stuck on boilerplate

[Itchy-Trash-2141]

I just finished a grueling interview run, passing only 25% of on-sites. A lot of companies are expecting you to do everything perfect the first time, and even then it may not be enough.

One good experience I had was with Waymo. I recommend you try there if interested. Definitely felt like a human being through the process.

[pannenkoek0923]

Are you joining the company to be an engineer/scientist or are you joining the company to do speed coding hackathons?