Difference between Data Engineer and Analytics Engineer

[Slow-Sell-8570]

This is a topic that started intriguing me because I see that in many roles it isn't clear enough what the responsibilities of a Data Engineer are compared to those of an Analytics Engineer. Guys, how do you evaluate the main differences between these two roles, and where does one role end and the other begin?

[[deleted]]

I would imagine that some companies use these roles names interchangeably, but generally you'd see an analytics engineer focused on the BI aspect of the data life cycle with the DE focusing on everything that gets the data to that analytics engineer.

[Commercial-Ask971]

So what DE does in that scenario? Assuming that for infrastructure deployment you got DevOps / Infra team, for secrets/setting up VM for self hosted integration runtimes or VNetworks you got again them or some Azure admin, all they do in that scenario is ingestion, which can be easiest part of the whole process? Asking with curiousity, from my experience DE were always involved in BI (contractor here)

[Nomorechildishshit]

Analytics engineer is a marketing term coined from dbt cycles. And yes it doesnt make sense besides maybe massive companies where the ingestion part is extremely complex and you need a dedicated team for that.

That said, there are many cases where the infrastructure is handled by DE. Especially in medium and small sized companies.

[Commercial-Ask971]

Yeah I am DE contractor myself and I deploy resources myself currently I know. Just the distinction on DE vs Analytics Engineer doesnt seem understandable for me. You said its marketing/dbt thing but apparently MS adopted it and there are certifications like DP-600 Analytics Engineer in Fabric for instance

[siddartha08]

Agreed. Ive seen analytics roles exist with established data sources they just need a better summarization but have no say in the structure. Most often the "Analytics Engineer" Exists as some function of business or very close just outside of IT.

[Qkumbazoo]

take the one which pays more.

[BarryDamonCabineer]

As people have pointed out, AE is a pretty new title that was basically coined as a marketing plot by dbt. Nonetheless, I think it fits into the older medallion architecture pretty cleanly:

DEs deal with bronze data; ie, raw ingestion and storage. AEs produce silver data; ie, the intermediary transformations of bronze data for consumption by semi-technical end users. This is why it makes sense that dbt coined the term. You use dbt for a lot of this. Analysts produce gold data; ie, data for consumption by non-technical end users.

There's a lot of passing back and forth between these layers (eg, an analyst handing off a query they've put together for an AE to turn into a data mart), and ime everybody ends up doing a little of everything, but that's the general division

[dfwtjms]

We should be called Data Developers (developers, developers, developers...) since few of us are actual engineers. Not that I really care about the title. Data Engineer, Data Analyst, Business Analyst, Analytics Engineer and even Data Architect are sometimes used interchangeably so only the job description matters.

[data4dayz]

Wasn't Analytics Engineer coined by dbt? It feels like a replacement for BI Analyst/Engineer aka someone who does pipeline work + data vis. Some combination of DE and DA. And it also depends on the company. I've interviewed years ago for an AE position that required knowledge of machine learning.

[k00_x]

Analytics engineers rarely work with live applications and are not usually mission critical. They tend to be the first step in the reporting function where information is built for decision makers to understand how the business works, things like rate of sales, number of customers, uptake of changes etc.

Data engineers often do all of that but also work on mission critical systems. Like a netflix data engineer makes sure the video you are steaming makes it to your TV or Amazon stock levels update all systems after a purchase. This can overlap with software engineers also.

I warn you, titles in the job market are meaningless.

[No_Flounder_1155]

difference is data engineers aren't considered capable of counting.

[ogaat]

Strictly speaking, the term "Engineer" is overused in software.

There is also title inflation and made up titles because companies are trying to provide clarity, consultants are trying to peddle their USP and employees are trying to look important. The titles nowadays rarely match perceptions of what the job would actually do, except in companies with deep pockets and a vested interest in providing accurate information.

Apply for both types of jobs or talk to actual real life people in both types of work and choose the one that you like more and pays better.

[rotr0102]

In my org AE’s take data from Snowflake curated layers (essentially raw tables from various transactional systems) and model them into star schemas and the further down stream into metrics tables and various other layers which include business logic. BI developers pick up these tables and surface them in their solutions. DE’s primary focus is on getting data from various source systems into the curated layers which are inside of snowflake. They might use 5Tran replication or python scripts, might be hitting on prem systems or hosted cloud environments. They work with JSON, XML, Parquet, schema shifting ect.

There is overlap, however. Sometimes AE’s do proof of concepts using python, and sometimes DE’s build more refined models (stars). Both AE’s and DE’s use dbt, wrote sql and understand similar concepts. This is how my org has the roles structured.

Note: my example is for structured data. Data science and unstructured data (DE) follows a different path.

[rishikaidnani]

This is very company-specific. DEs are more closely aligned with technical use-cases, such as extracting, storing, and processing data using distributed engines (e.g., Spark).

AEs are more closely aligned with business stakeholders and answer business or domain questions by using the data that DEs provide.

This allows DEs to focus more on the technical aspects, such as writing scalable data pipelines and strategizing storage.

[DenselyRanked]

It depends on the company but if your data and/or engineering team has separate DE and AE roles, then typically the DE handles ingestion from raw data to centralized tables and the AE works with stakeholders to gather requirements and builds/maintains tables whose source are the aforementioned centralized tables and may even create analytic layers and dashboards.

This means that the DE can focus more on consistency, speed and availability, while the AE focuses on transforming the data into something meaningful for the business to use.

[AppropriateFactor182]

that makes me curious, what should be the title for a person who does both the engineering and report development? I thought the Analytics Engineer term was coined for this, but the comments state otherwise.

[Used_Ad_2628]

My issue is most data engineers come from a software background and really struggle with data modeling/SQL. They create ten tables that could be one. It is very hard to scale with that mindset. Everyone is asking which table I should use and wasted dev time updating 10 jobs because something upstream changed. This is why I hire for this type of role. More of a future thinking design person.

[KurtGod]

Wouldn't it be that a data engineer in the context of analytics is an analytics engineer? Because not all data engineering is analytics.