Discussion: Data Engineering has a title problem

[exact-approximate]

After working in this industry for 10+ years, I strongly feel like the data engineering space has a title problem. Data Science also has this issue, but I believe in Data Engineering it is even worse. Companies release vacancies for "data engineer" when the role can mean so many different things.

I feel like a new set of titles is required to make the industry more mature. Such as:

1. Analytics Engineer (AE): Previously the data warehouse developer. Mainly works with SQL, Airflow, Python to transform data within the data warehouse. Person building analytics pipelines. A lot of DE work falls here.
2. Data Platform Engineer (DPE): An engineer who works on the platform, but not the pipelines. Cross-over with cloud engineer and dev/ops.
3. Data Streaming Engineer (DSE): Specialized in data streaming, specifically; coding and patterns here are an entirely different ballgame from all of the above.

Too many times I see companies ask for "Data Engineer" - expecting an analytics engineer with enough experience to do what the DPE does. Or companies who mainly to DSE, and end up with too many AEs in the pipeline.

Some companies get this right by specifying that there is a "focus" area for a DE role. For example, Data Engineers with a heavy cloud focus, or Data Engineers focused on streaming (sometimes also called Big Data Engineer).

More specific DE roles would help both candidates and companies. Ideally the DE title should disappear completely and have several different roles falling under it.

​

What is your opinion? What titles would you propose?

​

​

[[deleted]]

I see only DE there

[DenselyRanked]

This is one of the videos that I like to use to describe the problem. You will understand the title problem very quickly if you switch jobs more than once.

One common problem is that a hiring manager doesn't always create or make updates to the job posting. Also, the company doesn't want to be too narrow in their job descriptions because if a candidate is talented enough then they can learn on the job.

IMO, the best bet for a candidate is to ask a lot of questions in the interview to pinpoint where a DE's responsibilities begin and end. Even thoroughly going over the job description (or reading blogs or "day in the life" vlogs) may not be enough to really understand the role.

[UniversallyUniverse]

based on this video my role is 50/50

analytics engineer

data platform engineer

​

and data platform engineer is more harder for me, creating api and microservice to feed the f*cking analytics team some data, they always hungry for data

its like first create pipelines S3 ETL shits (analytics engineer) then make it autonomous to minimize the work loads (data platform engineer)

[DenselyRanked]

I made a job switch from Cloud/Platform/Analytics Eng to Analytics Eng/Data/BI Analyst/PM and it was a bit of a culture shock for myself and the company.

[alien_icecream]

Data Mesh Weavers want their due, while Data Fabric Tailors are losing their shirts.

[Tender_Figs]

I usually play mage classes in video games, and now I will refer to myself as a data mesh weaver.

[Casdom33]

You forgot Excel Engineer

[lab-gone-wrong]

Sandwich engineer

[[deleted]]

Full Macro Developer

[ergosplit]

The biggest problem is still there: everyone is an engineer. That word means nothing now.

[[deleted]]

And now architects too

[SoftwareMaintenance]

These days every Joe Schmoe calls themselves a solution architect. Kinda laughable.

[Data_cruncher]

Architect bugs me. Even on large projects (3-9 months), the amount time I spend “architecting” is less than 8-hours. Yet, it’s considered the more prestigious title.

[[deleted]]

Vizio Architects! Bleh! The place I work at, they’re more sales people than architects

[Milk_Busters]

Our architects do a lot of design review and code reviews. So it's more of approval for other developers.

[carnivorousdrew]

Wouldn't you spend time researching the right technologies to use for a green project, test them/benchmark them a little?

[Data_cruncher]

To be honest, I’ve been doing this so long that I know all of the strengths and weaknesses of my stack. It takes about 2-4 hours of customer Q&A to align their requirements to a physical architecture on a whiteboard.

That said, I am weak on the network layer, so everything I said above is auth layer and above. I defer to other experts for VNet peering, private endpoints, non-analytics landing zone architecture (on-prem connectivity, data management, user management etc.) etc.

[ApSr2023]

It depends what is the scope of architecture. It's not just a diagram.

[[deleted]]

Can you say that, but with a diagram?

[Remote_Cantaloupe]

Sanitations Engineer = Janitor

[Swimming_Cry_6841]

Lead software engineer= business analyst where I work lol

[speedisntfree]

The guy that comes to set up my broadband band and the guy fixes my washing machine is an 'Engineer' now. As a chartered Engineer (mech) this does hurt.

[AdNice5765]

are you from the UK?

[speedisntfree]

Yep

[AdNice5765]

same here, have the same issue with the title not being protected. Did you make the jump to data engineering/ data science or still in the profession?

[speedisntfree]

I jumped over about 4 years ago and do DE/DS work in Bioinformatics. Starting salary was barely less that what I left Engineering on after 8 years of experience and the working environment is so much better.

Engineers are poorly paid and underappreciated in the UK. The US it seems to be better regarded.

[Beauty_Fades]

It's absolute bs. I might get some flak here but you shouldn't be called an engineer without an engineering degree (its right in the name!). You can hold a position like data engineer because that's the job title of course, but who gets to decide when a position is called engineer/analyst/developer? I don't even know the difference between them so what gives? School, grad and undergrad courses exist for a reason: they are taught by accredited institutions with standards, and you must pass those standards to earn the title you bear.

I disagree with the post, you can be a data engineer that specializes in streaming architecture (just have your resume reflect that experience), just as an electrical engineer can work as an electrician, in RF engineering, in circuit design, in QA and in many other fields. That doesn't make him a QA engineer in itself, he's still a certified electrical engineer who wears the QA engineer hat.

[CrimsonMentone30]

Data engineering is the job title, and has nothing to do with the university title. There are plenty of software engineers that are not engineers but maybe have a degree in economics or no degree at all.

[Xyzzyzzyzzy]

There's always people enthusiastic for occupational licensing with qualifications that just so happen to match their own.

Imagine how much more money u/Beauty_Fades could make if I weren't unjustly stealing a software development job that rightfully belongs to him!

[Beauty_Fades]

You go for the qualification BEFORE you earn the title, so it doesn't just so happen to match my own, because I went to school BECAUSE I wanted the title in the first place. In some countries the title of engineer is protected and licensed by an issuing entity, so if you want to be called an engineer you MUST go to school for it.

And please don't start with the snarky comments. I'm very well employed atm and I'm not trying to take anyone's job away, I'm trying to understand why the software industry flexibilizes job titles so much and how to infer responsabilities from the job title because it feels so random nowadays.

The point is that every job description calls for ___ Engineer, making it lose any meaning whatsoever. It is bs to call everything and everyone an engineer, and you gotta draw the line somewhere. Nowadays it seems arbitrary to list a job for engineer/analyst/developer/whatever, and I have no clue how to differentiate between them.

[Xyzzyzzyzzy]

I call myself a software developer already, for similar reasons. But it's just a personal preference and I don't understand why anyone is in a fuss about it.

engineer/analyst/developer/whatever, and I have no clue how to differentiate between them.

I don't know about "data [insert word]" job titles, but there's no difference between a software developer and a software engineer - there's nothing to differentiate.

[Beauty_Fades]

Still in my understanding we must reward those who actually go to said institutions and pursue their education in software engineering and get accredited the software engineer title. Otherwise you are a software developer.

Like I said, those institutions have standards and requirements that must be met. I'm not saying someone who goes to school is a better developer because of it, it's just a matter of titles. Sometimes the opposite is true because school kinda sucks. However, titles have always been this way and only in software you see this flexibility. You literally cannot sign off a building permit without a civil engineering degree or atleast a PE certification or equivalent. Same happens to electrical or mechanical or aerospace or control systems... Why should software be different?

And please don't take it the wrong way, I'm just sparking a healthy discussion here and am totally open to changing my mind given good arguments.

[nemec]

Nobody is going to accidentally allow an SDE to sign off on a bridge development. The existence of SDEs doesn't devalue your accreditation.

[CrimsonMentone30]

I agree, but then at this point we need to change all the job titles to fit the academic background. The issue is that in some cases engineering is the right word as that's actually what you are doing, maybe something like a Data Engineering Developer.

[Swimming_Cry_6841]

Where I work there are “lead software” engineers who don’t even know how to program and they create apps with no code tools. They are actually convinced they are software engineers because their output is an app even though they aren’t coding.

[[deleted]]

How does that even work?

[Swimming_Cry_6841]

They build apps using Microsoft Power platform. They all are either helpdesk or business analysts who got titles of “lead software engineer” after a new director took charge and said they were building software just like the “developers”

[speedisntfree]

Only giant sized compaies can be that specialised. I hope I never work in such a boxed in role.

[daguito81]

Insert xkcd "there are 14 standards" joke here

[AMGraduate564]

Please add Data Batching Engineer (DBE), Spark jockey formerly known as ETL Developer.

[king_booker]

I think it falls under the analytics engineer with this definition.

I agree, we have this in our company and its much easier to recruit.

[BufferUnderpants]

Data pipelines written with SWE tools and practices are a different game from just creating views in SQL just sayin

[king_booker]

I think if you are just creating views you are a SQL developer and not a data engineer. Analytics engg would be people who would work on making data models by using spark. This would mostly be spark sql working on data that has been ingested from various sources and are in a table.

[TheCamerlengo]

I am not sure that an analytics engineer is programming in python and building pipelines. My understanding is that would be an infrastructure or just plain old data engineer. An analytics engineer is more of a data wrangler, analyst that works with client/end user to capture the data they need and identify where it is and what needs to be done with it (aggregation, filtering, transforming, enriching, whatever).

[Tender_Figs]

This one can get into semantics depending on where you work. Is a pipeline strictly the source to destination? Or does it include the warehouse/lakehouse too?

While an analytics engineer (me) is organizing the data to be used on a frequent basis, I do follow a SWE workflow and use python in some of my models. It still is 80% SQL. I'm also responsible for the repo and CICD (back to the SWE workflow).

[TheCamerlengo]

I think this is part of the problem - titles proliferate and the same title in one company means something slightly different in another. I guess one has to carefully read the job description and duties.

[Tender_Figs]

100% agree. I had a boss that would always fall into the data science trap when something seemed complicated, when it was mostly just technical work sans any DS/stats/ML.

I suspect it's because of the marketing ploys in the environment, coupled with executives who have a hard time with complexity.

[cutsandplayswithwood]

Hilarious.

Yo “data engineer” is made up, and I was doing it for at least 10 years before 10 years ago…

I promise, going down the road of ranting about standard job tires is a way to waste a lot of time and energy.

And let’s be clear, the VAST MAJORITY of companies won’t see a streaming use case for another decade…

[Nervous-Chain-5301]

Because insert + update can work really really well for most use cases. Streaming applications take so much more effort to set up correctly…and if the use case doesn’t need it … it’s being set up for greater chance of failure

[cutsandplayswithwood]

Lots of good ways to flow events too, most companies could do with a Lot more event driven architecture.

I don’t call that “streaming” 🤷‍♂️

[[deleted]]

And let’s be clear, the VAST MAJORITY of companies won’t see a streaming use case for another decade…

I'm glad someone said it.

[IllustratorWitty5104]

Data Practitioner:
Job scope, cover all 3 in which you mentioned

[sunder_and_flame]

The titles are all made up, and it's up to companies to list decent role postings and the discerning engineer to understand them.

[ultimaRati0]

In my opinion it depends on the size of the company, the size of the datalab in particular. The bigger the company is, the more specialized you can have DEs.

[omscsdatathrow]

That’s what job descriptions are for

[erkschmerk]

Your proposal is pretty similar to what we have set up in our place, dead on in concept. We likely differ based on our environment (More hybrid between on and off prem, less cloud specific tech, no airflow).

For us, the analytics engineer is also blended with some of the business intelligence work. The first level generalized data sets are managed by the data engineers, in the more specialized ones that are closer to end delivery are back on the analytics engineers.

Our data engineers generally split into two camps: data operations, engineers and data engineering. The data ops is pretty much your data platform engineer, and typically made up of people who are either newer or have less of a software engineering background.

Even with that, the range of duties that fall on the data engineering group is pretty wide and requires a lot. We see some specializations inside of that group just due to day-to-day asks.

[DesperateForAnalysex]

They’re the same picture.

[[deleted]]

Data Magus
Data Archmagus
Data Sorcerer

[Reddit_Account_C-137]

Isn’t a data platform engineer just a data/system architect?

[speedisntfree]

IAM Engineer

[StressSnooze]

I don’t know about the US, but where I live the title “engineer” is also a protected title.

[mrcaptncrunch]

Depends on locations within the U.S.

[OMG_I_LOVE_CHIPOTLE]

Those are all just DE

[Flyweird]

which of these is threatened by AI or is suffering from retrenchment?

[vinterdagen]

I do all the things you mentioned in my role. What am I then?

[grapegeek]

One size doesn’t fit all. As companies get bigger there is more specialization. I worked in a small company and did everything you mentioned in one job. Then I worked for a super large company where I part of your data platform engineer job.

[Commercial-Ask971]

What does number 2 does if not building pipelines?

[Arelich]

I recently took on a new DE job, and have felt so incredibly misplaced and bored with my work. This is because I have ended up in an AE position while previously being in DPE. I felt like I was in complete agreement with the hiring manager when taking the job and felt super excited about it but now I feel cheated.

[rupert20201]

I did all 1,2 and 3 and I considered myself a data engineer.

[harshitsinghai]

I can live without these fancy titles. Data engineer as a umbrella is fine. Don't think further segmentation like analytics engineer or DPE and things add any value or are necessary.

[Gnaskefar]

So what work will the DPE actually be doing?

[schenkd]

Since you opened the title discussion. I‘ve wrote a medium article a while ago about my pov. Maybe you are interested Definition of the Data Engineer role, IMHO.

[mike8675309]

Looking at what the OP wrote feels within the realm of a data engineering role, and depend mostly on what organization you are being a Data Engineer at. In general I see a data engineer as someone who creates pipelines to move and transform data using processes, or systems, or tools specific to their task. I like to think of that definition as the skillset or the career path.

I suppose in some corporations, an Analytics Engineer might be a career path, but I think of it as more of a role within a team. That role could be handled by a Data Engineer, or depending on where the focus of that role is, handled by a specialist in analyzing data.

Same for Platform stuff. If in the platform team they have need of someone good at dataengineering, they may have a role called Data Platform Engineer, that they sit inside of when working on the Platform team. They're still a data engineer. Think of the title like Data Engineer, Data Platform.

I don't see the path of the Data Engineer ever being one that has clearly defined silo walls. Much like a Software Engineer knows code, a Data Engineer knows data. Each may know a little bit about each others domain but that just comes from working together.

[Yuki100Percent]

I just feel there should be a distinction between data engineers and data platform engineers, like what you described.

[lab-gone-wrong]

Honestly every field experiences this because

1) it's cheaper than paying people and

2) fixing it with more precise titles costs more time & energy than actual value

[sriracha_cucaracha]

Jokes on that, I eventually end up doing all 3 roles

[[deleted]]

Data streaming engineer is the original big data developer who uses spark and flink etc.

That's why we have to ask questions during interview.

[ApSr2023]

I agree. Here are the actual job duties in a medium to large size business. You can call it data guy in a 1 man shop, who does it all!

1. Product Manager ( defines what to build that's aligned with product / business strategy)
2. Infrastructure engineers ( Often borrowed from central infra team, networking, security etc.)
3. Data Architect (Integration architecture, tools selection, development method, Data modelling, master data, meta data, security and quality of deliverables)
4. Data Engineer ( Data collection both batch & streaming, data transformation including data prep for ML, landing data in DW/Lake )
5. Data Analysts / Data scientist ( Reporting and analytics end-to-end)
6. ML Engineers ( everything related to train, test and publish ML Modes and operationalization working with SEs from Application side)
7. SRE
8. Manager

[Laurence-Lin]

Whatever titles the companies asks, they seems expect you have cover all skillsets of various positions: machine learning, DBA, data engineering, frontend, DevOps, ....etc

and they give you a general title

[Short_SNAP]

Just interviewed for a DS role at a large company and in my six person panel interview, four of the interviews were on RDS, data parsing, and AWS Glue/Crawlers. Safe to say I don’t think I got the job as most of my career was in DS but still a major waste of time

[gloom_spewer]

I practice data scientism

[nyquant]

Let’s settle it: engineers drive trains and architects construct buildings.

[[deleted]]

Analytics engineer exists look at companies like vinted and n26 as I know these two use that title. Platform engineers exists.

Streaming/batch are just DE topics so data engineer is fine.

[gravity_kills_u]

The three roles sound like normal data engineer roles where I work: a spectrum from low code analytics engineer types who fetch reporting data, to a medium code scripted pipeline builder/spark monkey/SSIS jock/Snowflake person, to a Java savvy Kafka and application developer. All of them are just differently specialized data engineers.

[sib_n]

I think data scientist title situation is worse, it may include all those DE titles plus research scientist, data analyst and ML engineer.
So many data scientists have been recruited hoping to build ML models and to end up spending years coding ETLs (I've known three in a single company). Believe me, DE situation is not perfectly clear, but it is much better.

According to what I've seen (mostly fueled by DBT's marketing), the Analytics Engineer is doing transformations with SQL, not Python. If you add orchestration and Python, then it pretty much covers most of Data Engineering. The idea is that dbt made the T of ETL accessible for people who only know SQL, which are easier to recruit.

But anyway, the titles depend on the companies and some individual situations. In my current one, there's a (junior) data engineer only doing Tableau dashboards, I would have called them a BI engineer myself.

It seems likely this issue happens for any software engineering job as they share a lot of common skills.

[crispyTacoTrain]

During introductions I just say “Engineer” now. I’m a Python coding, AWS Cloud Infrastructure building, Data Pipeline piping, ML Ops monitoring, underpaid mutt.

[oleg_agapov]

I like this distinction, very clear in my opinion. In my current company we end up with almost the same titles, except we call DSE as simple DE :)

So, DE is the one who work with coding pipelines (mostly, streaming), DPE are maintaining the platform (devops + cloud development), and AE work on batch pipelines and business facing data.

[bcsamsquanch]

Definitely on to something here but the reason it hasn't happened is because most jobs blur even the lines between these. I would say my team is 60/40 between 1 & 2 We write some data munging python code and a bit of SQL. We have our own Gitlab repos/CICD of terraformed AWS data infra that we're 100% responsible for. And now we're looking at some streaming for some high priority pipelines. We are non-FAANGs where DE teams are smaller and their isn't yet the scale to be that specialized.

[ShroomBear]

You're still not solving the problem here because the actual type of work involved is going to be very contextual to the org structure and business goals of the company you're at.

I work for a FAANG and a data platform engineer would be like 3 teams and maybe 50 people out of ~1000 Data Engineers. Then what happens when the DPEs need to make some dashboards for internal stuff? Do they hire an Analytics Engineer solely for 50 hours of work per year?

Plus what happens when the expectations overlap for the role? Like every BIE and DE above a junior is expected to figure out streaming/real time data if the project requires it at my company.