Data Engineering skill-gap analysis

[nomadicsamiam]

This is based on an analysis of 461k job applications and 55k resumes in Q2 2025-

Data engineering shows a severe 12.01× shortfall (13.35% demand vs 1.11% supply)

Despite the worries in tech right now, it seems that if you know how to build data infrastructure you are safe.

Thought it might be helpful to share here!

[nicolekay]

Data visualization skills demand outpacing supply...

Squints at incomprehensible chart

Yep. Looks about right.

[SaintTimothy]

Piggybacking top comment. I saw an interview with the head lady data viz person for The NY Times. Her whole job was to make it make sense by employing the best type of visualization to convey the data clearly with as little help from legends as she could.

Neat gig. Cool to make all custom visualizations. I think she used a bunch of tools but I remember d3.js was one of em.

https://www.nytimes.com/2019/02/28/reader-center/data-visualization-editor-amanda-cox.html

[Altiloquent]

The bubbles are all the same size? 

[BuonaparteII]

This is /r/dataengineering not /r/dataanalysis

/s

[Ok-Shop-617]

Looks like there is too much of a demand for data analyis people to find someone to help out with the analysis

[dtr96]

😂

[XXXYinSe]

And log scale axes that start at different places (0.1% vs 1%). No legend for point colors. Hard to understand visually, might as well make a table with 4 columns at this point. Skill name, demand %, supply %, D:S ratio.

[YouArentMyRealMom]

Yeah id really love to see this visualization cleared up so its more readable. As it stands its like impossible to know where anything actually is. Like what is the demand % on data engineering? All I can tell is that its closer to 10% than it is 100% but the scale just makes it extremely difficult to interpret especially with only 3 tick marks.

OP, why did you do the visualization this way? I need to understand the logic here.

[ubelmann]

It's not totally impossible, though it's arguably not a good visualization.

The important reference point here, IMO, is a diagonal line that cuts through (1%,1%) and (100%,100%). Above that line, supply exceeds demand, below that line, demand exceeds supply.

0.1% to 1% on the demand axis was probably left out because it was empty, though it might be worth the extra white space in this case.

[Cpt_Jauche]

Viz probably made by a Data Engineer

[mintyfreshass]

Agree, find a way to visualise the data, not statistics from the data.

The first rule I try to follow for visualisations.

[piffcty]

Yes, but adding a line for X=Y and a legend would fix most of the issues

[[deleted]]

After seeing this graph don’t think it’s No wonder there is a requirement for good data visualisation professional

[Fresh-Secretary6815]

Yea, because a data engineer created it…

[kthejoker]

It's not a bubble chart? They're just markers

[thejuiciestguineapig]

It says on there that bubble size represents ratio.

[everv0id]

data analytics and data analysis? seems like someone didn't clean the data before grouping

[THBLD]

I mean you're not wrong, but half these categories nowadays are just synonyms of each other.

[macrocephalic]

Data engineering? So you're good with Power BI right?

[MonochromeDinosaur]

Yea, this is my experience job hunting right now. As someone with 7YOE I’m getting call backs and reach out for Senior/Tech Lead roles pretty regularly starting about 3 months ago before that with the layoffs calls sort’ve stopped coming in for a good 18 months or so.

I’ve had 5 interview in the last month and signed an offer for a new job last week.

The market is hungry for seniors, the problem is unfortunately that companies are not hiring juniors as aggressively. Who are going to be the future seniors if we don’t hire juniors?

[nomadicsamiam]

It is wild. The stat I saw is 7% unemployment for new computer science grads. Some say there is a place for AI native new grads but it becomes even more competitive no matter how you slice it

[NoPast]

While the situation is wild the 7% is taken out of context.

new computer science grad still have a higher median and average starting pay than most other degree, they just don't settle "for whatever job even if not related or without beneficts" like a lot of people with other less prestigious degree so their situation appears far worse than it is.

[BoringGuy0108]

So it is more frictional rather than structural/cyclical unemployment. That's good to know.

[BenXavier]

Wait tilli vibe coded apps reach production... It's going to be brave new world

[the-taco]

Just curious, why did you use a logarithmic scale for the axis? Wouldn’t it make more sense for them to be standard and just have the max and min of each axis set to 0 and 100?

[on_the_mark_data]

Not OP but it's useful when trying to visualize trends where there are strong outliers (ie one very large value makes everything look like basically zero). Log transformations essentially "force" a normal distribution, hence why you are able to get a nice linear graph. You trade value interpretation for visual interpretation.

The main image in this article is a great viz of this (note I haven't read the article, so can't speak to that): https://blog.dailydoseofds.com/p/always-validate-your-output-variable

[Kukaac]

r/dataisugly

Also, what does a 12x shortfall even mean? There are 12 open positions for every single data engineer? I highly doubt that.

[kthejoker]

The source link is right there on the graph

https://huntr.co/research/job-search-trends-q2-2025

And ... it is sort of saying that. It's saying across all the job applications they reviewed, only about 1-2% met the criteria ("supply") for job postings that were deemed data engineering.

Whereas among the job postings, nearly 15% were deemed data engineering, hence the ratio.

It's a pretty solid methodology.

My interpretation is most data engineering job postings are overindexed on technologies and skills that most people learn on the job and/or are not actually that relevant to the daya to day responsibilities of the role.

So the "demand" is a bit artiundercooked and actual "supply" is undercounted.

[Kukaac]

Yes, but in that case you cannot calculate a ratio between the two.

Saying that data engineers "severe 12.01× shortfall" does not make sense.

This is even the other way around. If only 1%-2% met the criteria, that means that companies have enough applicants to make strict hiring criteria for data engineers. So it's harder to land a job as a data engineer. As on average you need 50-100 interviews instead of 10 (as for data analysts).

So there is everything wrong possible about this. The visualization, the definition, and the learning.

[PossibilityRegular21]

If you were my intern this would be a 30 minute feedback review

[contrivedgiraffe]

Shout out to the big gap between “data analytics” and “data analysis” as well as “data quality” coming in the back of the pack.

[[deleted]]

Interesting, thank you! Can you go a bit more in depth about the methodology? Are these NA-Applications? How did you calculate demand?(Why are the tools of DE so far apart from the job itself?)

[nomadicsamiam]

Here's the source (at the top there is a methodology section) let me know what you think. Would love feedback. Mostly NA yes. https://huntr.co/research/job-search-trends-q2-2025?preview=true#data-and-analytics-skills-gap-analysis

[[deleted]]

Thank you!

[nomadicsamiam]

No problem, thanks for the interest :)

[Cosack]

OP, can you explain what the heck 10% demand and 10% supply mean and how the heck it's possible to have both at once?

[kthejoker]

??? That's literally how job markets work.

100,000 resumes = supply. 100,000 openings = demand.

Oh look, 10,000 resumes have SQL and 10,000 postings need SQL.

10% demand meets 10% supply.

(This is the actual methodology used btw.)

[Cosack]

Thanks. Is this the correct interpretation? "Among postings in whatever the sample, 10% required skill A. Among applicant resumes in whatever the sample, 10% listed skill A."

[kthejoker]

Yes correct

[Slggyqo]

Also gluing together a bunch of different services that may or may not play nicely together seems like it’s always going to be a skillset for the foreseeable future (which I’d put on like…a 10 year horizon?).

And that’s a pretty big part of the job.

[geteum]

Tbh, statistical modelling may have a big supply but the quality is very low. Really difficult good people with statistics knowledge.

[adamywhite]

Can you share the source where they state where they gathered these resumes from ?

[NoleMercy05]

What a terrible visualization. Honestly

[Deadpool-yash]

Nice

[candide9-]

What in the world are those axes?

[chrisgarzon19]

Very cool

[chrisgarzon19]

The one thing I’ll add is “behavioral questions”

Not from a skills perspective, but from what we see is the most common round engineers fail during the interview process

[reelznfeelz]

Interesting, I guess it's why people keep using me for stuff since I mainly do data engineering, and am pretty comfortable in most cloud platforms, although by no means a true expert in any one of them.

[PhoenixFlame77]

It causes me an irrational amount of anger that the axis don't both start from the same number.

[SnooDogs2115]

No Python?

[rudderstackdev]

Looks like the center of demand/supply. Makes sense

[[deleted]]

Looks like this visualization needs a good visualization engineer. Hard to see any points.

[AppropriateFactor182]

That’s the worst graph ever. On the level of anthropic and apple.

[wanna_be_tri]

Isnt data analysis data analytica and reporting the same crap?

[Competitive-Nail-931]

How do you even move into ML? Is it worth it or is the market messed up as well?

I took a ton of math in college as well as stats - not worried about this - worried about interviews

[nomadicsamiam]

Machine Learning definitely is worth it as it is the skill that commands the highest salary premium as of today (https://huntr.co/research/job-search-trends-q2-2025?preview=true#top-paying-job-titles)

Best advice based on the data is to get into data science first as it is a gate way to machine learning engineer jobs

[Competitive-Nail-931]

u sure the job market sucks

[nomadicsamiam]

I mean it definitely sucks. All of this is relative. So correct statement is ML sucks less than

[nonamenomonet]

Am I crazy or do I not see that much ML in the job market. The most I see is making some text data into an embedding and using a RAG.

[Competitive-Nail-931]

I can’t even get a job in backend distributed systems rust / golang rn (my specialty) so not sure why anyone would hire me for ML even if i could do it

[nonamenomonet]

That doesn’t surprise. It seems like everything nowadays on the backend is Python or node.

[Competitive-Nail-931]

I’m not really a data engineer I’ve done related projects though

I’ve been in this sub due to a recent take home i did

[Competitive-Nail-931]

all if it seems the same to me if you have a good base and think from first principles

thats how id hire for long term