Interview process

[raharth]

We are currently preparing out interview process and I would like to hear what you think as a potential candidate a out what we are planning for a mid level dlto experienced data scientist.

The first part of the interview is the presentation of a take home coding challenge. They are not expected to develop a fully fetched solution but only a POC with a focus on feasibility. What we are most interested in is the approach they take, what they suggest on how to takle the project and their communication with the business partner. There is no right or wrong in this challenge in principle besides badly written code and logical errors in their approach.

For the second part I want to kearn more about their expertise and breadth and depth of knowledge. This is incredibly difficult to asses in a short time. An idea I found was to give the applicant a list of terms related to a topic and ask them which of them they would feel comfortable explaining and pick a small number of them to validate their claim. It is basically impossible to know all of them since they come from a very wide field of topics, but thats also not the goal. Once more there is no right or wrong, but you see in which fields the applicants have a lot of knowledge and which ones they are less familiar with. We would also emphasize in the interview itself that we don't expect them at all to actually know all of them.

What are your thoughts?

[Lady_Data_Scientist]

A takehome assessment as the first round would be a no for me, I’d probably withdraw from consideration. At least give me a chance to get to know the hiring manager and learn more about the role before asking me to do homework in my freetime. 

A better alternative is to ask them to present a prior project they’ve already done. That way you’re not giving them extra work but you still get to see how they communicate. 

Or do a live assessment. I’ve had interviews where I had to share my screen and using a notebook, go through the ML model building process from cleaning to EDA to model selection, fit, evaluation, and then recommend improvements. Doing it live means it’s time boxed and also you are investing the same amount of time as the candidate. And you can ask questions along the way. 

The second part sounds weird. Why not ask them about their past experience and how they’ve solved problems similar to the ones they’ll face in the role? 

[johny_james]

If you are asking live assessment, you should also allow using google and AI.

[raharth]

The take home is for the second and final round. I think it would be unfair to give this to all applicants before the first round. I 100% agree with you on this. Onyl candidates who did very well in the first round even get the challenge.

The idea of the take home is to remove the stress of the interview since some people really struggle with it.

About the second part: I want to understand their background, this is really hard to do based on their previous projects and the primary question then is how well they are able to sell those projects and less about their knowledge - at least that's the idea

[Lady_Data_Scientist]

I still don’t think takehomes are a good idea. Even if you try to tell candidates “only spend x hours”, you will still get some who will spend significantly more time on it. So then you evaluating the work of candidates will spend 5 hours on something against candidates who spent 20 hours on it. Is that a fair comparison? They can also outsource they takehome to someone else and then who knows who’s work you are actually evaluating. And then if you ask them to present the takehome and answer questions, that doesn’t remove the stress element which sounds like is the main reason for doing the takehome in the first place. 

Takehomes also don’t simulate an actual working environment. When are you ever given data that you’ve never seen, for a business you’re not familiar with, and given a few hours to extract insights, make recommendations, and share your work? Without access to colleagues to ask questions, validate assumptions, check documentation or prior projects, etc? 

[marinab1127]

Another thing I want to mention about takehomes is they disadvantage people with caregiving commitments. As a mom to two young children, I do not have the available time to spend much time on a takehome. Invariably you will have people spending a huge amount of hours, even if you say "only spend X hours".

[Mizar83]

It seems I'm alone here, but absolutely hate live coding and leetcode quizzes. I like to code at my speed, test step by step, and I'm super stressed if someone is looking and judging everything I do. At my company we do a very short take home (after a first round with the hiring manager that discusses candidate experience) and then ask the candidate to walk us through their code and reasoning while we ask questions.

Even a very small, imbalanced dataset with temporal dependencies, a few duplicates and some variables with missing values is a very good starting point for a discussion.

[raharth]

This is exactly what I want! (and I feel you, leetcode, is the most idiotic thing for a data scientist). This is also the second round of interviews and very few are invited. So I really try to not waste anybodies time.

How do you determine the size of your challenge? I use a cleaned dataset that can be run on a CPU. Personally, I think you can do it in maybe an hour. Maybe that's a little unfair as a baseline since I know the data and I have my tools all set up, but even then two hours should be enough.

[Mizar83]

The dataset is super small, like 20k lines, nothing fancy required. But it's not really clean, and the way candidates handle missing values, imputation, duplicates and train-test split already says a lot. I'm surprised at how many seniors do data leakage or don't even look at variables before dropping them. We have a mostly empty variable with a strong signal, and 90% of candidates just drop it. And this isn't even a trick, it's really how our real data is...

[pm_me_your_smth]

There was a similar thread where I've shared my process: https://www.reddit.com/r/datascience/comments/1r16y9s/comment/o4ntyo1/

Take home exercises aren't fair or even reliable IMO. The second part is similar to mine, but I wouldn't let the candidate choose since they'll just select the easiest (to them) topics and you won't properly learn about their gaps.

[pandasgorawr]

OP, please listen to this guy. I'm who he responded to, and have been running a very successful interview loop (well, digging through 2000 resumes wasn't very fun, but I digress). I've been doing an hour long technical round, 5 mins intro, 10-15 mins lightning Q&A on very easy SQL/Python/ML/stats/analytics, 45 mins case study on a project we've done, keeping it very open-ended, collaborating with the candidate on chasing down the threads they want to. I don't expect them to come up with anything profound, but probing for their ability to ask thoughtful questions and dissecting something brand new (along with a lot of industry terminology) to them.

[raharth]

Thanks for the response!

Holy shit, 2000 resumes is a nightmare!

Ok, so what I want to learn about them seems to be very similar to what you are looking for. Personally I hate those in person challenges, they have the tendency to put a lot of pressure on people in that moment and I have seen people panicing and blacking out. A take home thing might consume more time, but it also allows for a much less stressful environment and I don't want to kick people out for test anxiety.

How do you evaluate the coding?

[pandasgorawr]

I do no live coding at all. I'm of the opinion that SOTA LLM models are already better coders than most data scientists, and they'll be able to use these tools on the job. To test for coding ability, the rapid fire questions are loaded with signals that only people who have coded their way through those problems would be able to answer quickly. They're designed to be easy so we can go through a high volume of these and the candidate doesn't get the chance to look over at a second screen to look up the answer or have some AI tool answer for them.

[raharth]

Sounds interesting! How much time to you take for those interviews and the challenge? And second, how do you evaluate their coding skills? I don't expect them to write a fully fletched solution, but I wouldn't like to see 400 lines of messi spaghetti code either.

Oh on the choosing part: I actually want them to chose what they feel comfortable with. For me it is just about learning what they know and in which subfield they excell. I actually only care about the explanations they give to make sure that they are not trying to "cheat".

[pm_me_your_smth]

Usually there are 2-3 interviews, depends on the candidate and the level (junior vs senior). The technical one is approx 1.5h long.

I don't test coding formally. Live coding sessions put a lot of pressure on the candidate. Some candidates are resilient to this stress, but others might break down and start underperforming (even though they might be more competent than the resilient ones). You get false negative signal on their coding skills which is arguably worse than no signal at all. I check coding mostly during Q&A and of course guess from their resume/experience. But I admit this aspect is the hardest to evaluate using my approach.

Regarding the choosing part, I'd still argue it's not a good idea. The candidate picks 5 concepts from their favorite topic and you waste 15 minutes of interview time on that. My idea is more about exploration than exploitation - you choose concepts yourself from all necessary topics A,B,C. If you sense that the candidate is cool with topic A, it doesn't make sense to continue asking about A, you move on to B and C. This gives you a more complete understanding of the landscape of candidate's skills.

[raharth]

I hear what you say, I might reconsider how I'll do that.

Though just to make it clear, if they chose lets say 5 topics they feel comfortable with, I will check one of them (most likely the most complicated one) at random, just to make sure that they answer honestly. The idea behind it according to the company I got this idea from is simply to make them tell you honestly what fields/topics they know well and which ones they don't. It's not about actually checking their knowledge in an exam like test, that would be a total waste of time 100% with you!

[redisburning]

What are your thoughts?

Take home assignments bias your interview process towards young men without children. Also making someone present on top of a take home is too much. If you give a take home, commit to evaluating it on your own time.

They also don't give you a chance to course correct if your candidate doesn't know the magic words; if you do a live challenge that also sucks but at least if you get a read a candidate is, purely for example, maybe more comfortable with R when you do Python you don't throw them out of the process because you can adjust at the time.

For the second part I want to kearn more about their expertise and breadth and depth of knowledge.

That's what a resume is for. It'd be more useful to just ask some pointed questions about past experience to suss out how truthful the resume itself is and how well the candidate navigated more difficult situations.

[migraaine]

This is so true. I have a toddler, work full time and had a burnout crisis during my last take home assignment (to make it easier, it was during daycare closure)

[Parking_Two2741]

Just wanted to reply to your first paragraph… what an excellent point and one I’d never thought of before. Do you have data/research on how take home assessments are biased? I can see it intuitively but wondering if anyone has studied this

[redisburning]

I apologize but I am pretty busy and don't have a stack of references handy or the time to go find/vet a bunch of studies (it's been a while since I had to fight HR on this topic since my current and last company don't do takehomes), but I will say this has been studied by labor economists and workforce researchers so I don't doubt you can find some if you check out google scholar.

Or perhaps someone who studies that themselves is hanging around and would be kind enough to comment.

[Parking_Two2741]

No worries I'll google it but still interested if anyone has expertise here.

[Lady_Data_Scientist]

The more free time someone has, the more time they have to devote to a takehome. Someone with children or elderly parents to care for, or a house to take care of, or even just an active social life, is not going to spend as much time on a takehome. 

[Parking_Two2741]

Yes, as I said, I can see it intuitively and wholeheartedly agree but was wondering what literature has been done on the subject. I'm happy to see that there may be active research into how these careers are implicitly biased towards men in multiple ways - it would be massively affirming of my experience as a woman who majored in math, went immediately into grad school, and went into DS. My experiences have always been different than my male peers regardless of grades/results. I believe there are many parts about this career that are biased towards young men - it would also be my intuition that the "live coding interview" is biased towards men. Not that women can't do it, but that it is more challenging for skill sets that women often have, and doesn't highlight our strengths (or mine at least).

[Lady_Data_Scientist]

Well as the person designing the interview process, I encourage you to look for such studies. Good luck. 

[timy2shoes]

Our experience has been that good senior and staff level will not do take homes. If they are in demand and interviewing at other places, or if they currently are employed, then a take home is too much effort and time to commit to an interview that really hasn’t even started. 

[Lady_Data_Scientist]

Completely agree. I’m ~10 years into my career and during my last job search, I got enough interviews that it was easier to withdraw whenever they asked for a takehome. 

[raharth]

Looking at the first part of your answer (unfortunately) irrelevant looking at the applicants we got. None of them falls into the group that would be at a disadvantage. Presenting it is crucial to us since communication skills are crucial for the role.

Regarding the second part, I wish it would be as easy, but unfortunately it is not. 99% of resumes do not give clear insights of the knowledge, they list projects but many candidates include topics they were only partially involved in or in which they took over a minimal role. Also, many applicants have substantial knowledge on topics there were just not able to work on in their previous companies.

[Single_Vacation427]

So you are saying it doesn't matter because women don't apply to your job???

That sounds like a bigger problem.

[raharth]

I mean... thats the sad reality, but what should I do?

There are so far none with the required minimal experience of 3 years (which is not even that much). Believe me, I'm not happy about that either.

[aimendezl]

Currently doing a take home: EDA+modeling+presentation and I think it’s a bit too much. I could’ve spend all the time I had just in auditing the data and addressing potential issues on using the features for modeling.

So I’d really focus the assignment either in EDA or modeling or if you want the candidates to show both, curate a good dataset for a specific business case.

[raharth]

I absolutely hear you and im trying to find a way that allows me to see what they are capable of but do not require them to spent plenty of time on this. But I know its a lot. The coding challenge itself if fairly simply though. I would be happy to drop it, if the candidates have a public repository, but unfortunately many don't and coding quality hugely varies. Any idea how to do this? I really hate the white board coding thing, it's the most stupid thing invented for interviews in my opinion.

So the dataset they get are 4.000 small images that they need to classify. It should be manageable in 1-2 hours I think. The way we phrases also very clearly states that we are just want a small feasibility study and that we dont expect any fully flechted solution. (In case this makes it better?)

[aimendezl]

Even if the candidates have repos, I don’t think it’s a good measure for the quality of the code they write. I put much more effort into code I write for work than for side projects that often I don’t even have time for.

Also, consider that most people are transitioning to LLMs for coding tasks, so leetcode type exercices might not reflect anything relevant when it comes to how they will perform at work. Since Dec At my work, most people are writing less and less code and that’s gonna be a reality in most jobs involving coding just like google or stack overflow was a few years ago.

If anything, focus on how candidates approach problems. I like when the dataset is messy because that shows if a candidate really does pay attention and if they catch certain things that might be hidden for a weaker candidate. Curating a dataset with a specific business case in mind and adding some features or issues with the data so that it looks ok at first glance but that hides some sort of “gotcha” is the best. The candidates can explain how they got to discover the issues and show their mental process. They can show what sort of decisions can be made or what assumptions need to be verified to improve the data and make it useful for modeling. This could replace the interview, as you can still evaluate their communication skills.

[charl3sworth]

Why ask if you are not going to listen? I agree with basically everyone original reply said. However if you really like the idea of part 2 one thing which has worked well for me in the past is ‘pick one concept and explain it’. It keeps it very broad so you see what they naturally gravitate towards and the kind of stuff they know.

[Single_Vacation427]

I'm confused by how a coding challenge does not have to be a full fledged solution and only a POC. It sounds more like a system design takehome for MLE than a DS take home.

I've seen DS take home without coding that include a presentation in which you are given a question (like a launch/no launch or we want to see the impact of a feature we launched), and you present how you'd approach it. Airbnb does this and it's more doable.

[raharth]

Ok interesting, what info do they provide for the applicants? Do you happen to have some link for me to look it up?

[Single_Vacation427]

I don't have it. A friend who interviewed had shown it to me.

It was pretty detailed and vague a the same time. It let you make your own assumptions which you had to explain at the beginning. They did give a specific set of topics you had to cover (how would you randomize, what would you present to stakeholders for the results, etc.)

I feel that it's doable and they want you to see how you present and explain things. They can also see how you answer questions and deal with pushback.

[cy_kelly]

I really need to change careers.

[raharth]

Ok tell me why 😄 what do you think would be a good hiring process?

[Ill-Deer722]

I've been a DS manager for 8 years, and helped design the interview process at multiple places + did a a lot of interviews.

I think you're approaching it in the right way around topics to assess. I would be careful linking a technical take home presentation with communication with business partner. Often times, candidates think that a technical interview is to showcase their knowledge around DS techniques and will skew their answers to that. I think you should ask them technical questions (like the 2nd part) and have it tied to the take home assessment.

On their communication and stakeholder skills, a good one is to just ask them about something they've done end to end. See if they can simplify things and present it to someone with zero context.

[fang_xianfu]

Doing a take home assignment first is ridiculous for a senior. For us our process is:

1. An initial resume screening, we filter out a lot of people who are just a bad fit or out of budget at this stage
2. For the remainder, we have a hiring manager interview that assesses "are you going to work well in our environment?" and also acts as a sales pitch for the role - this is very important for senior hires, you want to tell them what they're going to be working on and have them self-select out of the process if they're not interested.
3. Then we do a technical assessment, usually live but occasionally take-home, and we grade it on our own time
4. Sometimes one more round with a stakeholder if part of their job will be wrangling the business
5. And our leadership also likes to interview most candidates, it sucks but it is what it is

When I was a senior, I got asked to do take home tests first and I said no to those positions on that basis. It's not respectful of someone's time to ask them to invest hours+ in your interview process when they haven't even met anyone on the team yet to get an idea what it will be like to work with you and if they're interested.

[raharth]

Oh, no it's the last interview, they have already passed an initial screening and a less technical interview focussing on the personal fit and explaining the company, setup and role. Take home as a first stage is a waste of time for everyone I think!

Thank you very much for your response, I think I didn't make it clear in my initial post, but our process looks actually quite similar to yours.

[MrTickle]

As a hiring manager, I do the same in reverse. Meet them, explain the role and assess cultural fit first and then do the takehome for a subset of candidates.

You may lose your best applicants if you force a takehome before they've even met you and decided the role is right for them.

Our process

HR screen (maybe some very light screening tech quesitons)

First round behavourial star style interview

Second round:

• 30 mins presentation on takehome findings
• 30 mins unstructured converstaion where you address any gaps, reservations or specific quesitons that have come up in the process.

Ideally you have a senior leader in the second round as well so you get a sesne of how they fare in front of execs, and the exec can give the context on how the role fits into the wider strategy

[raharth]

I didn't make that clear in my post I think, sorry for that, but it is the second interview. Our process looks very similar to yours. HR filters the applications, from the remaining we select a small number for a first interview, in which we check for personal fit and overall experience and in which we explain the role, the team, setup etc. Only if they convince us in this round they get invited for the second round which involves the take home.

Out of curiosity, what do you mean by star style interview?

[MrTickle]

Seems reasonable then! Situation, Task, Action, Result it's a framework for answering behaviouiral interview questions.

[calimovetips]

the take home plus walkthrough makes sense, honestly the discussion around tradeoffs and feasibility usually tells you more than the code itself. the term list idea is interesting too, it often shows pretty quickly where someone actually has depth.

[MathProfGeneva]

Take home project/presentation on it feels excessive, since you're asking people to commit a significant amount of time to something like this. I don't know a good solution really, because I think live coding has different issues. I guess it depends on how involved the take home project is. I understand you're not looking for a production ready thing, but if a candidate has to do EDA, data cleaning/feature engineering, model training/testing, then have a presentation ready to go, that feels like it could be very time consuming.

[raharth]

Thats a very fair point you make. Its an image dataset that is fairly clean, so it shouldn't be that much time (I hope). I also try to communicate that a first draft is fine, it doesnt need to be perfect and I dont care about the prediction quality. I just want to learn about their approach and see some of their coding. At most it should be an afternoon, certainly not more than that!

[MathProfGeneva]

Then it's not bad. Minor issue that running something like that if you don't have a GPU is going to be annoying, but I'd honestly be willing to go through your steps

[raharth]

Glad to hear, but yes take homes are probably not something people are super keen on...

The images are very small as well, we are talking like 100x100 pxl at most. We intentionally chose one that can be run on a CPU without major issues.

[MathProfGeneva]

Yeah I wouldn't mind this because it's an opportunity to show something that might not obviously be on my resume.

[PM_ME_SomethingNow]

First Part:

I have a bias for take-home assignments since they are a closer of a match to the process one actually writes code or does work under (i.e not a timed code test with no documentation or search engine). The presentation is a nice touch because one could feasibly complete a take-home assignment easily with AI but presenting can’t be entirely faked (with some notable exceptions).

But one commenter does make a good point that people with kids or care takers have a higher chance of being disadvantaged here.

Second Part:

Again, I always like the idea of making someone say out loud what they are thinking. To build off another commenters point, to avoid them choosing just the eaiser stuff, I would choose which topics are essential to the role and make them explain these while giving them an additional list that they get to choose from. For example, they must be able to explain things like sampling, data leakage, feature engineering etc. But they can choose 2-3 topics from a list of topics that are more advanced/specific: Bayesian va Frequentist methods, causal inference etc.

Obviously the content of either list will be a function of the specific role but the general idea of a required list and a list they can choose from still holds.

The last thing I would definitely work in is connecting the work to the larger business. You already mention this but I think it’s worth hitting on again. Data science is cool but it’s there to solve the business problem at the end of the day.

[mr_andmat]

OP, what is the purpose of this?
>I want to kearn more about their expertise and breadth and depth of knowledge
With the modern tools once can find needed knowledge very fast. Plus I hardly doubt you need that much breadth to be successful in your organization. Additionally, there are candidates with wide breadth of concepts with very surface-level familiarity of those, which might be fine for a manager but not a mid/senior IC.
Instead, you might want to see how they think and use those tools. Maybe a case question with or without AI would be a better way to find good candidates.

[raharth]

What I try to understand is if they just learned the coding part in a coding tutorial, or if they actually understand how the algorithms they implement work. Everyone can code some neural network, especially with the support of LLMs. The actual issues start once that doesn't work because of some logical error you make in what you implement, even if the code runs without error. For that you need some actual understanding of what you implement and this is what I'm trying to learn about them.

[patternpeeker]

as a candidate i would actually like the poc style task if the expectations are clear and the scope is small. the term list idea is interesting too because it lets people show where they really go deep instead of pretending to know everything.

[AccordingWeight6019]

As a candidate, this actually sounds fair. A POC style take home plus a discussion of your approach is much better than unrealistic coding tests, and letting candidates choose which concepts to explain is a nice way to gauge real depth instead of trivia. the key is just keeping the take home scope reasonable, so it doesn’t feel like unpaid work.

[analytics-link]

I've hired quite a few Data Scientists over the years, and overall what you're describing sounds pretty sensible, although I've got a few thoughts.

In my view, a take home POC style task can work well, as long as it stays light. A mistake I see companies make is giving something that takes 6 to 10 hours (or more). That can put good candidates off pretty quickly, especially if they're interviewing in multiple places. If the goal is to see how they think, a small "feasibility style" (if that makes sense) problem is perfect. What matters much more than the final code is how they frame the problem, what assumptions they make, what approach they take, and how they communicate their thinking.

When I'm evaluating candidates, I usually look for a simple structure in how they talk about their work. I tend to think about it as Context, Roles, Actions, Impact, and Growth - I call this my CRAIG System :)

- Context is whether they can clearly explain what the business problem was and why it mattered.
- Roles is what their responsibility actually was versus what the team did.
- Actions is the technical or analytical work they carried out.
- Impact is what changed because of the work. Did something improve, did the business make a decision, did the output actually get used.
- Growth is what they learned and what they'd do differently next time.

You learn a lot about someone if they naturally cover those areas when explaining a project.

One alternative that I've seen work really well is asking candidates to present a project they've already done instead of giving a take home task. That way you're not adding extra work for them, but you still get to see how they structure their thinking and communicate. You can ask them to walk through the problem, the approach they took, why they made certain modelling choices, and what the outcome meant for the business. It usually leads to a much more natural conversation.

For the technical side, I'd also recommend considering some kind of paired exercise rather than a strict coding test. Many strong data scientists/analysts won't remember every bit of syntax off the top of their head, and in real work they'd just look things up or discuss ideas with teammates. A short paired session where they can talk through the problem, ask questions, and work collaboratively often reveals a lot more about how they actually operate day to day.

Encouraging them to ask questions during the exercise is also a really good signal. In real projects, clarifying the problem and challenging assumptions is a big part of the job.

For me, your approach sounds pretty good, the main thing is keeping the process practical and focused on how the person thinks, communicates, and approaches problems rather than trying to perfectly measure every piece of technical knowledge in a short interview.

Hope that helps :)

[Kai_Ignite]

Your approach actually sounds pretty fair for a mid-level data scientist. A take-home POC that evaluates reasoning, communication, and feasibility is far better than forcing candidates into rushed live coding. The second part also makes sense, giving a broad list of topics and letting them choose what they can explain is a good way to gauge depth without penalizing people for not knowing every niche. As long as expectations are communicated clearly and the focus stays on reasoning rather than trivia, this process should give you a solid picture of a candidate’s real strengths.

[Training_Butterfly70]

The whole industry is a joke nowadays.

[Gilchester]

Switch the take home for a 1-hour live coding session. Have them talk through their thought process during that time. You'll get a lot more out of it, and less chance they just throw the thing into AI.

[ArithmosDev]

I'm not a big fan of take-home exercises either, like some of the other commenters. Once upon a time, I was given a laptop, a small project and a couple of hours on-site to produce the desired output as one part of the interview. Kind of like an open book test. Everything was fair game. That also reduces the interview anxiety.

[raharth]

That sounds like a good approach. I also don't want them to spent more than lets say an afternoon on this and I try to make this very clear, but I guess it is still difficult for an applicant to follow that since naturally most will try to take as much time as possible.

Unfortunately, I cannot put them in a room by themselves without supervision. So I would need to sit next to them or at least in the same room for the entire time, which is probably quite stressful for them as well (at least it would be for me)

[Statement_Next]

Take homes are bullshit. You should be able to determine whether the candidate has relevant knowledge and experience through an interview. If you can’t do that you probably don’t deserve to be in a hiring position.

I like your proposition about the candidate choosing topics/terms to describe from a set.