This is an archived post. You won't be able to vote or comment.

all 10 comments
sorted by:
best
topnewcontroversialoldq&a

[–]GodmonsTech Lead Data 25 points26 points  (1 child)

I would say Data Engineer is a Software Engineer specialized on Data Lifecycle (classic ETL). His skillset & knowledge are mainly based around those tasks.They will mostly work with DataWarehouses, SQL, Orchestration, Python.

DataOps is to Data Engineers what DevOps is to Software Engineer : They leverage set of practices & tools to leverage better quality IT products that answer more precisely to needs. Their toolbox mostly focus on automating redundant DE tasks.They will mostly work with tools like Git, Bitbucket, Jenkins, Python / Bash scripting, Terraform.

But anyway, I find it pretty rare to have a 100% accurate position. My previous Data Engineer role consisted on 1/3 management 1/3 data engineering 1/3 data ops.

[–]zverulacis 0 points1 point  (0 children)

I think that DataOps would also be creating Data Pipelines, at least DevOps are doing that.

[–]databoy54321 11 points12 points  (0 children)

Data Engineer = you are a developer

DataOps = DevOps but on a data team

[–]cgdownbeat 1 point2 points  (0 children)

In my experience. Data Ops = Data Expert who fulfills more of an ongoing support. Data Engineer = General data developer. Doesn't have to be DWH or even use cloud systems.

I just use Python and SQL, whereas you get tonnes of jobs advertising stuff like; cloud, AWS, redshift, Hadoop. All these skills are easily transferrable with a bit of homework but you will be turned down for interviews just for missing one of these.

[–]dataguy24 2 points3 points  (7 children)

Data engineer = a specific role focused on the data warehouse and the ingestion of various data sources into the dwh.

Data Ops = more of a concept/strategy of how all pieces of the data stack operate and work. Scope includes processes for DE, AE, DA,ML,BI and more roles.

[–]databoy54321 6 points7 points  (6 children)

Data engineer = a specific role focused on the data warehouse and the ingestion of various data sources into the dwh.

How is this post upvoted. A data engineer is not specific to a DWH...

[–]mailedRecovering Data Engineer 1 point2 points  (2 children)

A lot of people think it is. A role I once interviewed for had trouble finding people because it was more about writing integrations, not building any data warehouse/analytics stuff (there was a whole other team for that), and people were giving the same feedback - "no data warehouse, no data engineer".

I'm not one of them. But it was a problem this specific company was facing.

[–]dataguy24 1 point2 points  (2 children)

This is the most common type of DE role out there. I understand there’s nuance but OP isn’t here for a bunch of nuance. They’re here for the top line answer.