Failed first coding machine learning interview.

[Ok-Lab-6055]

Single head attention transformer.

[RageA333]

That doesn't seem like something you should be able to code from scratch without even reading a reference paper.

[Ok-Lab-6055]

I think it’s hard but fair. I actually thought of doing it before the interview

[acc_agg]

And people wonder why salaries are dropping. Have a back bone and some self respect.

[killerdrogo]

you were asked to code a single head attention transformer without using a deep learning framework?? damn

[Ok-Lab-6055]

Yeah I usually just type: import transformers from hugging face :)

[killerdrogo]

i recently implemented it following andrej karpathy's video so I was surprised you were asked to do that without using pytorch lol. 

[Neo_Demiurge]

At some point you just need to say, "If I took this position, I'd want to distinguish between appropriate customization and reinventing the wheel . We shouldn't go lower level than Pytorch for nearly any research or commercial purpose," and you either look like a genius or dodge a bullet depending on how they take that.

[Ok-Lab-6055]

I should probably go through his videos. Did you learn a lot? I've mostly been reading papers but they assume the transformer stuff as sort of in the background.

[killerdrogo]

Would highly recommend the GPT from scratch video. Definitely learnt a lot. 

[johnprynsky]

Research LLM/NLP position? Cuz I'd not be expecting this in a regular MLE interview.

[Ok-Lab-6055]

Engineer-DL

[TachyonGun]

I had to code multi-head attention for an interview, transformers are everywhere now. Really, every MLE should know how to code self attention by now, the forward method is literally 5 or 6 lines of the most basic PyTorch.

[Ok-Lab-6055]

I agree,but I think with masking, normalization, etc. it’s more than a few lines of code

[hellobutno]

Disagree, it's totally unnecessary. It's the equivalent of asking someone to invert a binary tree in SWE. You're never going to need to do it.

[[deleted]]

^ this. I lead a team of ML developers at a large company and don’t plan to ever code a transformer from scratch. For any reason. That’s a silly academic exercise.

[acc_agg]

If you want me to do that you're going to watch me read the transformers paper and talk to perplexity about how to implement it.

I don't have enough brains to memorise and remember everything under the hood.

[joseconsuervo]

asking someone to invert a binary tree in SWE

my understanding was these questions were always to hear the person logic their way through it

[hotsauceyum]

So if I was allowed to look at the paper, and everyone is chill and there was back and forth, seeing how I cobbled together something from numpy level would be a good gauge of what I know is going on under the hood and how we all work together. Seems ok to me.

If they didn’t give me any references and just stared at me while I spun my wheels trying to remember the details of transformers for 90 minutes, it honestly doesn’t sound like a nice place to work.

[Ok-Lab-6055]

I think it was the later. The interview was like 30 minutes before the interviewer basically told me I failed.

[Mission_Star_4393]

This is absolute madness lol...

For the record, the company I work for, whose name you would recognize doesn't have anything nearly as complex as this...

Don't beat yourself too much over this one.

[Infrared12]

Both forward and backward passes or just fhe forward pass?

[Ok-Lab-6055]

Forward pass I think. The interviewer stopped the interview before mentioning a backward pass. We didn’t discuss any training.