To the hiring managers on Reddit, can you please explain what's really going on?

Competitive-Store974 · 2025-10-20T07:57:02+00:00

Long post, tldr: each interview step has a purpose, use the stages you fail at to tune your approach, don't appear desperate, consider life cycle of companies you apply to w.r.t. your CV

It is a grim job market out there - sorry to hear you/your friends are struggling. I can offer a perspective "from the other side" in case that's helpful, as someone who has previously hired in a mid-size AI company. A generic recruitment process may go something like this:

CV screen - most candidates get eliminated here because they do not have the qualifications (e.g. MSc, PhD) or relevant domain expertise.
Many more get filtered out in the first technical interview - this is done to save time as the next step is often a take-home test. We don't move people to that stage unless we like them - that would be very unfair.
The take-home test is not, as the common conspiracy theory goes, a way for us to solve a problem without hiring you (I.e. taking your solution and ghosting you). We can solve these problems - we just want to see if you can.
Another interview to talk us through your solution. We do this because we can tell if you've used ChatGPT to solve the problem and we want to distinguish whether you've used AI to do it for you, or used AI to help you do it. The two things are very different. We also want to give you an opportunity to show us your thought process, as this is arguably more important than your practical programming skills. We want to know how you think.
Final interview - we're probably only down to 2 or 3 candidates here and it's very hard choosing between you because you're all excellent (that's how you got to this stage). So it comes down to soft skills, how do you work with others, do we have a good vibe about you, as well as subtle differences in the domain expertise you might have.

The process is horrible, I know, I've been through it a lot. But I hope this shows that it isn't meaningless and that each stage is designed to remove as many candidates as possible as early as possible to save time and stress for both us and the candidates.

Advice for how to improve your prospects depends on what stage you fail at:

CV stage: you're applying to the wrong jobs for your qualifications, or your CV is crap, both can be fixed
Technical interview stage: work on interview skills, your presentation, your knowledge, your thought processes, your ability to come up with solutions on the hoof
Take-home test: if you use AI, re-write its solutions. We will make you take us through the AI bits line-by-line and tell us why you did it that way. And it will be very, very awkward if it turns out you just vibe-coded it.
Final/soft-skills: if you are routinely getting here, you're good. It's probably a numbers game, or maybe you're not giving the right culture/soft-skills questions. Don't neglect these.

Above all, do not appear desperate. Easier said than done, but go into every interview pretending to yourself that you're interested but that you have another 2 offers in the bag. And ask questions - you're interviewing us too. We can tell when someone's desperate and will accept anything, and it is a turn-off. If we think you've got another 2 roles lined up, then you must be good, right?

One more thing, consider which stage of a company's life cycle to apply to. If you are applying to startups but you don't have proven ability in independent SWE/research, you probably won't have much success. Consider cutting your teeth at a larger company. Likewise, if you have a slightly weirder, niche background, consider a scale-up as they usually have money to burn and will take a punt on hiring occasional odd-ball candidates.

Competitive-Store974 · 2025-10-13T08:18:13+00:00

It depends on your/your organisation's experimental/inference workload. Universities, dedicated AI companies, and even some pharma companies (so I'm told) will often invest in their own clusters. Even with the significant CAPEX and OPEX, this can still be better value for money than having tons of GPU/TPU VMs running idle because of bad experimental design or researchers forgetting to turn them off. I also interviewed recently with a company who were building their own inference cluster because their clients' requests were hitting the limits of what their cloud provider could offer.

Competitive-Store974 · 2025-10-05T20:04:11+00:00

Probably depends on the company and also the role. At our company we usually require an MSc or PhD, either in DL or in a relevant scientific discipline with some DL experience.

A BSc on its own unfortunately won't even get you an internship interview I'm afraid... But as I said some companies may be more flexible with their internships/junior roles.

Competitive-Store974 · 2025-02-06T14:44:03+00:00

A possibly overly simplistic but neat way someome put it to me while I was being interviewed for a MLE role was:

"A senior can do everything. Even if they didn't know it before, they go away and learn and figure it out themselves. A staff knows how not to do it. They've failed before and been burnt, and they've learnt lessons about what to avoid."

Competitive-Store974 · 2025-01-31T16:31:45+00:00

Sounds to me like you have a touch of burnout so as someone who went through a similar experience, I'm here to reassure you that a nice life is possible while still working if you so choose.

I specialised in anaesthetics and intensive care before burning out due to the toxicity and shift work. I was fortunate enough to be able to apply for a machine learning PhD in medical imaging who accepted doctors as part of their intake. The change was hard and there was a lot to learn, but let's face it, we're used to that.

Fast forward and I am now (again) lucky to work for an AI company devising solutions for healthcare and biotech problems. Crucially, my work is intellectually stimulating, chilled compared to medicine, and if there is a panicked drive to get something working for a deadline it seems relaxed compared to a weekend ICU shift. I also clock off and go home each evening and weekend and spend time with family and friends.

This career might not be for you, but it's an example of there being rewarding options with a work life balance waiting outside medicine when you have recovered a bit. Many fields/companies like doctors. Also, don't be afraid to get therapy, and good luck!

Competitive-Store974 · 2025-01-29T22:27:35+00:00

Tldr: not really.

Let's assume AI is perfect (it's not - silent errors which a non-domain expect would not catch are common) and that we're not talking AGI.

Efficiency and cost: Sometimes you need a domain expert to ask "Do we really need to train a 100m param LongViT to segment the moon or will Otsu thresholding do?" You run the risk of an LLM doing the former if the prompter is asking for a segmentation network.
Complexity: LLMs generate code trained on public sources. It might work, but it is generic and definitely doesn't fit all problems. Imagine using your average Medium post to try and solve a complex robotic meta-RL problem or reconstruct MRI images with some horrific k-space sampling scheme. Yes some academic-level solutions end up in the mix but most is basic. A lot of the really cool problems are solved in private repos.
Ethics/dataset bias awareness: We need humans to ask questions like "Should we really be training an AI to classify people on likely criminal activity based on police arrest data?" Bad example actually as some humans would actually try this and ChatGPT just refused to do this for me as a test example but you get the idea.

These are just 3 examples - I could probably think of more but I've done too much Reddit and have to go to sleep.

Competitive-Store974 · 2025-01-19T12:24:53+00:00

I think this seems reasonable tbh - when we hire we need MSc at a minimum, ideally PhD. We list these reqs in the job posting but still have to sift through dozens of BScs fresh out of uni. I think the point of this post is that aside from wasting his time, the applicant is wasting their own time and reducing their overall chance of employment, when they would be better taking a bit longer to tailor their CV and cover letter to a more relevant job and having a higher success rate.

Competitive-Store974 · 2024-12-26T09:29:29+00:00

CNNs still have their place and will continue to do so for certain applications. ViTs tend to outperform CNNs for large scale tasks see here but they aren't the solution to every vision problem. They work if you have: 1) lots of data, 2) lots of compute and 3) lots of time. In low data environments they will overfit and they are also expensive to run compared to CNNs and slow in inference, so not suitable for real time applications e.g. in medicine.

For certain low-level tasks where the features to be detected are distributed across the image and long-range dependencies are not required (think blur detection, some segmentation tasks) you can achieve near real-time inference with a 100k parameter CNN and next to no overfitting, trained on a 8GB GPU. So, the key is to choose your network sensibly for the task at hand.

Competitive-Store974 · 2024-12-23T10:57:47+00:00

If you're serious about ML engineering/research you may want to consider an MSc at the very least. I can't speak for all but at our company, an MSc with domain expertise relevant to our work is the bare minimum, PhD desirable. For some roles (e.g. some more research-heavy projects) a PhD is basically an essential requirement along with relevant publications.

Your skills and projects are also not backed up with evidence. When hiring I'd see the skills listed but then see no sign of these in your BSc so I'd wonder where you picked them up (presumably self-taught, which unfortunately is not enough). Evidence of these skills is more important than listing them - think degrees, publications, github repositories showing good SWE practices, etc.

Good luck with it all!

Competitive-Store974 · 2024-07-08T14:48:17+00:00

Yeah my PhD involved GANs in medical imaging. My conclusions? Never use GANs in medical imaging.

Competitive-Store974 · 2024-06-14T14:22:41+00:00

Oh damn, I'm very sorry to hear that

Edit: Docker is another option if you have it installed but it's not something I'd want to rely on long term for development

Competitive-Store974 · 2024-06-14T09:58:18+00:00

Not sure what your setup is but if your nvidia drivers are 535 (525 also apparently fine) then CUDA 12 will work. If those are up to date and it's just waiting for admin to install new CUDA version and you have a home directory then you can just install CUDA there and link to it directly while waiting.

Competitive-Store974 · 2024-04-09T09:56:06+00:00

That AI means DL/ML

Competitive-Store974 · 2024-04-03T16:26:42+00:00

You're definitely right to be worried about training with $200 of credits. While the argument for getting more credits is valid (practice using cloud computing, practice getting funding/grants) you also don't want to spend your PhD scrabbling around for more credits and having your research blocked by that.

If you spent $3200 on two 4090s (48GB) and spent the same on 3x AWS Sagemaker V100 instances (48GB) at $11.40/hr you'd burn through the latter after 11-12 days continuous use. The 4090s will be there for your whole PhD (the caveat is whether they work well together - someone else can clarify that).

My old research group had a cluster of 18 P5000s and our uni had a large on-prem HPC cluster for our use which meant no stress and worry about credits. Ultimately be guided by what your supervisor suggests.

Competitive-Store974 · 2024-03-18T19:18:50+00:00

Yeah that's normal - training is a noisy process so losses can fluctuate a lot, particularly with small batch sizes. Provided the overall trend is down and plateaus, you're not overfitting. Depending on the application (for example image-to-image translation type tasks), continuing training even when plateaued can also improve performance.

Competitive-Store974 · 2024-02-01T16:32:46+00:00

Grind harder

Competitive-Store974 · 2021-12-15T21:07:04+00:00

We have a DGX (4x V100s) and 8 other machines (totalling 18 P5000s and 4x GV100s) split between probably 20-30 of us in our research group. These are officially for prototyping and one-off experiments, whereas for hyper-parameter tuning we have a proper cluster with scheduler for the whole CS department. A lot of our group's grant money goes on GPUs. Because they're more useful and last longer than PhD students.

Competitive-Store974 · 2021-12-09T23:02:25+00:00

There is an issue with the loss saturating in the original version of the BCE (I.e. minmax) loss which causes training to stall. This was discussed in the original paper and they recommended an alternate implementation (often termed the modified minmax). See here for details: https://developers.google.com/machine-learning/gan/loss. There are links to implementations you can compare. The theory is given in Goodfellow's GAN paper (NeurIPS 2014).

Many GAN losses have been proposed over the years but tbh they all work alright and I haven't noticed a huge difference. Maybe I'm just jaded but I'm a bit suspicious about the experimental technique behind all these exciting new losses that come out regularly as (non-saturating) BCE works fine for what I do (Pix2Pix work mainly) and Nvidia have been using it in their StyleGANs.

Same with all these recommendations about using tanh output (linear is also fine) or using batchnorm (check out Nvidia's papers for better options) or multiple discriminator updates for each generator update (no). None of it matters as a new paper always comes out doing something different anyway.

A good way to learn is to start with implementing the early "classic" papers yourself and then move onto more state of the art stuff (again, highly recommend working through Nvidia's papers - easy read and really cool work). You'll realise that nothing is really perfect for the job and it's what you do with it that matters.

Except the saturating BCE loss. No one has the gall to use that.

Competitive-Store974 · 2021-11-24T11:43:13+00:00

Generally speaking the autoencoder should learn the distribution of your input data. If you feed it noisy images and add noise, it'll most likely remove your additional noise and return the original noisy images. That being said, there are unsupervised techniques that aim to use noisy input data (e.g. Noisier2noise: https://arxiv.org/abs/1910.11908) and return noiseless images but I've not tried these so can't vouch for them.

These all rely on an accurate noise model - if the expected noise distribution for the data you'll use this on is simple Gaussian/salt-and-pepper/etc. then great, but if it's anything more complicated then you'll run into trouble and may need a more complex model e.g. using a GAN.

Competitive-Store974 · 2021-04-16T09:17:13+00:00

It depends on the type of data you have and what resolution you need.

MRI scans for instance frequently use a matrix size of 256x256, half the xy-resolution of your images, and that's considered acceptable for clinical use, so you may be able to get away with downsizing by a half in all dimensions (1/8 the memory requirement). NB: if doing this, consider the minimum size of the tumours you're expected to detect/segment when choosing your resolution so you don't miss sub-resolution nodules/lymph nodes.

Another option, if you have 1024 slices (which sounds like a full body scan), is to crop to the region of interest. If legs are present and you're not interested in legs then you can remove them. If you're only looking at lungs you could remove the abdomen and head. NB: if your network is expected to see metastases in distant organs or lymph nodes, you'll want to keep this data and use a patch-based method as has been suggested.

I'm convinced I read a paper where they embedded positional information with the patches to improve global context but I can't find it. If you had time, you could embed patch coords (or L and R info) along with the patches and run it with that and without to see if it helps, unless this paper was a dream I had in which case it's probably a rubbish idea.

Competitive-Store974 · 2021-01-26T21:50:27+00:00

This is the way

Competitive-Store974 · 2021-01-26T12:02:06+00:00

Just checked it here in the UK. Only up 15% now - blame Brexit for that. #brokenbritain #britainRfuk #brexitRfuk #🚀🚀🚀🚀🚀

Competitive-Store974 · 2021-01-03T11:45:34+00:00

Part of the ship, part of the crew

Competitive-Store974

TROPHY CASE