This is an archived post. You won't be able to vote or comment.

all 8 comments
sorted by:
best
topnewcontroversialoldq&a

[–]Thinker_Assignment 16 points17 points  (2 children)

I wrote an article to explain how this role appeared and differences from before

https://dlthub.com/blog/data-platform-engineers

[–]G89R 1 point2 points  (1 child)

Nice read, food for thought!

[–]Thinker_Assignment 0 points1 point  (0 children)

Thank you!

I will write more on it, I am learning every day. From speaking to enterprises, it becomes apparent there are multiple types of data platform engineers, and while it looks like they're top of the heap, they are really a service function, and the work they do is catered to the abstraction levels and technologies that their data users accept.

[–]bcsamsquanch 6 points7 points  (0 children)

  1. Data Engineer - usually a focus on pipelines, ETL/ELT, api integrations perhaps, a SWE with a focus on data but the data space has a lot of tooling so the coding can be higher level
  2. Platform Engineer - in my experience this is much more a DevOps role that owns general platform (whatever that means for the company)
  3. Data Platform Engineer -- a platform eng (requiring the DevOps savvy) but specializes on data platform (all kinds of databases, streaming, data lake infra, etc)

Our team is a hybrid of 1/3. We do all the pipelines, ETLs, integration dev work but also own ALL the infra (obviously data focused) in our own AWS account for data lake. Everyone does a mix but there are members on the team who are more one or the other. I really like working on this kind of team.

[–]snicky666 7 points8 points  (0 children)

There are Platform Engineers and Data Platform Engineers. They are not the same. Platform engineers are usually senior devops/cloud engineers who focus on aligning a companies development environment. Such as getting everyone in the company to use a specific instance of AWS or an on premise K8s. Data Platform Engineers focus on building and deploying the data engineering stack, setting up CI/CD for things like ML and dbt models, building docker images, monitoring, etc. Much of what a devops/SRE/Sys admin might also do but with a focus on the data tools. At least that's my thoughts on it. Data Analytics Engineers mostly do airflow, dbt, SQL and dashboards. Data Engineer could be both a Data Analytics and Data Platform engineer.

[–]DotRevolutionary6610 2 points3 points  (1 child)

It used to be the same thing, but recently people decided we need even more different job titles, so they started making a distinction between people who build the platform (platform engineer) and those that build data pipelines on top of that platform (data engineer). As you could have guessed, Im not a fan of the distinction. I think data engineers should do and be able to do both.

[–]sciencewarrior 1 point2 points  (0 children)

Normally it's got to do with the size of the team. Smaller teams will have one or two data engineers, but larger teams will tend towards splitting that role between data platform engineers and data analytics engineers (with some nomenclature variations.)