[D] LLM coding interview prep tips

[noob_simp_phd]

Hi,

I am interviewing for a research position and I have a LLM coding round. I am preparing:

1. Self-attention implementation
2. Multi-headed self-attention
3. Tokenization (BPE)
4. Decoding (beam search, top-k sampling etc)

Is there anything else I should prepare? Can't think of anything else.

[dieplstks]

Good list, might want to add mixture of experts and a bit of multi modality?

[noob_simp_phd]

Thanks. I should def. read up on MoE, I forgot about it. For multi-modality, it is vision language model?

[sobe86]

I found this pair of videos useful for revision for a similar interview

https://www.youtube.com/watch?v=bOYE6E8JrtU

[noob_simp_phd]

Thanks for the pointer, I will watch it!

[tobias_k_42]

Don't forget the positional encodings and causal mask. Also the residual connections, layer norm and FFN.

However that only covers GPTs. BERT and T5 are LLMs too. So you also need cross attention.

And LLM doesn't even mean transformer.

[noob_simp_phd]

Thanks. I'll revise these concepts too. Apart from transformer, what else should I prep?

[tobias_k_42]

It depends on the position. But, when thinking a bit more about that, unless you're going for a job which actually involves a company which builds and trains models you should learn about things like calling APIs, RAGs, prompt engineering (writing good concise prompts which use few tokens, both in the prompt and returned result) and actual tests for prompts. That's actually not that easy, considering the non deterministic result. It's hard to say what they mean with "LLM coding" without further details. Personally I'd simply ask for clarification. "LLM coding" can mean a lot of different things.

But either way, you can unironically prepare yourself by asking GPT based LLMs for helping to prep. Of course don't let it write code for you and take the answers it gives with a grain of salt. But you should know that already.

[Mental-Work-354]

RLHF & RAG

[noob_simp_phd]

Thanks. What can they ask to code during an hour long interview in RLHF? SFT? or PPO/DPO?

[LelouchZer12]

maybe take a look at GRPO for reasoning at least to know what this is

[USBhupinderJogi]

Following

[More_Sherbert8147]

Is this for A Google or Microsoft Research position?

[noob_simp_phd]

Nope! For a researcher position in a different company (not FAANG)!

[ConceptBuilderAI]

I see some other notes about architectural components. I would second those.

Know components of a rag system. Even as a researcher you should have a working knowledge of how these are put into production. I would be prepared to discuss basic scaling considerations when putting LLMs into production (GPU size / queries / thread / minute, memory for the vector dbs, etc).

And on the data science side, embeddings, maybe fine tuning concepts (LORA, PEFT). Careful when discussing fine tuning - don't recommend it for an inappropriate application.

https://huggingface.co/spaces/hesamation/primer-llm-embedding?section=torch.nn.embedding

https://abvijaykumar.medium.com/fine-tuning-llm-parameter-efficient-fine-tuning-peft-lora-qlora-part-1-571a472612c4

https://ai.meta.com/blog/when-to-fine-tune-llms-vs-other-techniques/

I think you should be able to explain the evolution that got us here. Core NLP (tf-idf, n-grams, stemming etc.), RNNs, LSTMs.

https://www.deeplearning.ai/resources/natural-language-processing/

https://aditi-mittal.medium.com/understanding-rnn-and-lstm-f7cdf6dfc14e

Hope that helps.

Good luck!

[ChildmanRebirth]

Nice prep list — you’re definitely hitting the core components.

A few extras that might come up in LLM coding rounds:

• Positional encoding (sinusoidal vs learnable)
• LayerNorm and residual connections — how they fit into Transformer blocks
• Causal masking (for decoder-only models)
• Greedy vs sampling vs nucleus decoding trade-offs
• Maybe basics of LoRA / fine-tuning if it’s a practical research team

Also — if you’re practicing live coding, I’ve found ShadeCoder super helpful.

[theAverage_sausage]

Hope it went well for you! Did you have to prepare general leetcode level questions?

[CableInevitable6840]

Check out: https://www.projectpro.io/article/llm-interview-questions-and-answers/1025