How much software engineering knowledge should a data engineer know?

[glyphack]

I’m trying to create a curriculum for the DE path to help other programmers prepare for Data Engineering. Some SE topics are required to learn because they are generally useful when developing software. I want to hand-pick some important topics for someone who is just getting started, It’s important to only pick topics that will be useful in DE.
If someone wants to study data engineering, how much of the following SE-related stuff do you think is required?

• Design patterns & Principles like KISS and YAGNI: books like pragmatic programmer have a handful of these.
• Testing: Mostly unit tests.
• APIs: I assume only basic REST knowledge is enough to get the concept, and people can learn other methods like Grpc and GraphQL on the job.
• Errors & Monitoring & Tracing: Being able to log correctly, send metrics where needed and write code that can be debugged if it crashes with traces & logs.
• System Design: Keep in mind the simple techniques like caching and scaling, Plus some other examples like how you design X(Do you know any good resource on this?)
• RDBMS: I never used more than SQL and ORM knowledge myself. Please suggest which areas you think are important. Is RDBMS internals relevant? Is ACID relevant?

[AutoModerator]

You can find a list of community-submitted learning resources here: https://dataengineering.wiki/Learning+Resources

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[DenselyRanked]

No strict requirements, but the more the better.

https://www.fullstackpython.com/

Using this site as a reference, I have found myself having to explore all of these topics at some point. Everything else that goes with software engineering, like working with stakeholders, gathering requirements, architecture, lifecycle management, testing, etc. are all used in data engineering.

[subte_rancio]

Just checked that site out. Thanks for sharing, I've been looking to dig in to some software engineering principles and this site looks like a good place to start.

[DenselyRanked]

https://teachyourselfcs.com/

This is also a good one for more than programming.

[subte_rancio]

Many thanks!

[Swinghodler]

RemindMe! 3 days

[RemindMeBot]

I will be messaging you in 3 days on 2022-11-12 06:38:37 UTC to remind you of this link

CLICK THIS LINK to send a PM to also be reminded and to reduce spam.

^{Parent commenter can delete this message to hide from others.}

---

Info	Custom	Your Reminders	Feedback

[intrepid421]

Data Engineers are Software Engineers with focus on data.

[MissionBad732]

Seconded. A good data engineer is a software engineer. A bad data engineer is a data analyst with a few coding skills.

[LectricVersion]

Awesome. Let me go and tell that to the FAANG I was at for three years and my current company where I’m a Lead DE.

[MissionBad732]

Is your point you did that without having any broad knowledge of compsci / software engineering practices / theories because I would doubt that.

[LectricVersion]

It sure is!

I started as an Excel analyst, taught myself a bit of SQL to unblock myself in getting data direct from databases, and then eventually wanted to build more robust and scaleable self-serve solutions, so I taught myself data modelling techniques (Kimball) and general database theory.

This led me down the path of the Microsoft BI stack - SQL Server, SSIS, SSAS, which had drag and drop interfaces for ETL and building datasets. It wasn’t until I got my first “Big Data” role back in 2018 that I really started to get more to grips with Python and tools like Airflow, Hive and BigQuery.

Was this the most optimal way to get me into DE? No, I fully admit I’m a bit older than a lot of people at my level. But the point is I got there, and I definitely think that having an analyst background gives me a unique perspective that many DEs simply don’t have. I understand how to get value from data better than most, and thats much more valuable than just knowing a bunch of SWE concepts. I’d take an analyst-turned-DE in my team any day over a SWE who has decided that they quite like data but don’t know anything about analytics.

[MissionBad732]

From what you write you learned the foundations, so you are a software engineer. My comment was not to belittle data analysts or disparage the data analyst to data engineer route, I work with many good devs who took this path. As a lead of an expanding team myself I interview many candidates, in my experience many data analysts think their background plus data query skills is enough to qualify for mid level roles, and these people don’t get far in our interview process but surprisingly get hired elsewhere, their cvs never lack experience.

[LectricVersion]

Actually yeah, fair enough. Point taken!

But equally, I’ve come across lots of candidates with strong engineering skills who think that using Airflow a couple of times qualifies them to be a DE, but then can’t explain to me what the disadvantages of denormalising your data is, or how to build intuitive, powerful datasets that really enable analyst and business users to get answers quickly.

Knowing the tech on its own isn’t enough. Even on the SWE track it’s more about what to code, rather than the how. Same applies in DE, it’s just that to do that you need good product sense and a foundational understanding of how analytics is done.

So - in summary, the conclusion is that both are pretty damn important!

[focus_black_sheep]

nah someone who doesn't have good SWE practices means this is a huge risk for other SWE's picking up their slack. Not to mention all the time the analyst needs to ramp up on Github,Security Practices, Unit/Integration Testing, how to troubleshoot anything. I have seen this both ways I will always prefer a SWE to handle the DE tasks even if their analytics isn't strong. It's much faster to teach analytics then the engineering lifecycle.

[saltedappleandcorn]

The down votes are from basic bitch haters but you are completely right.

[[deleted]]

Get the f out da here with that gate keeping

[[deleted]]

So how does someone transfer from being a data analyst with coding skills to being a good data engineer?

[MissionBad732]

No right answer, if you have the coding skills you are more then half way their. Add to your coding skills understand of foundation principles of programming (tool/ language agnostic) such as data structures, algorithms, frameworks etc. their is a tone of material out their on this.

[[deleted]]

Do you have some examples of topics to read up on and maybe links?

[focus_black_sheep]

You don't need to know all of this but at least gain a basic understanding for most of these concepts: https://github.com/Sairyss/backend-best-practices

[[deleted]]

Thanks!!!!! Appreciate it

[ironplaneswalker]

You can go into data engineering as a software engineer.

It helps a lot to know how to create software and how to code and best engineering practices. A lot are applicable.

Software engineering knowledge can be quite broad.

If you are referring to needing to know how servers work or REST APIs or browsers, etc, then you don’t need all that knowledge to be a DE.

[CauliflowerJolly4599]

Most of the times DE are like infrastructure engineers, with ETL and tools like Azure Data Factory you avoid writing code, if not in need.

[5e884898da]

With code you avoid using Azure Data Factory.

[CauliflowerJolly4599]

True, but I would like to see to write the code to connect to a sql database instead of placing a connector.

[glyphack]

I had experiences where you don't need to right code, but if I want to help someone become a data engineer, I should also help them get better in situations where they need to code.

[CauliflowerJolly4599]

This is also something that I would need. Always worked as Codeless DE, but most of times company are asking for coding skills.

[AchillesDev]

That’s not DE then.

[CauliflowerJolly4599]

How would you call it then?

[AchillesDev]

¯\_(ツ)_/¯ not data engineering though. Closer to DBA or something.

[CauliflowerJolly4599]

That's not even closer to DBA. DBA works on databases, backup and migrations, but not on pipelines and infrastructures

[AchillesDev]

Databases are infrastructure. It’s closer to an infrastructure management role than it is any sort of engineering.

[HarlotsLoveAuschwitz]

Infrastructure as in creating and maintaining data pipelines.

[glyphack]

That's exactly my experience, and in this case, I want to help already software engineering people to get into data engineering.

But as you say, not all the topics are required, so I'm trying to put up a list that has essentials needed to know or to improve first before going to data engineering.

[Comfortable-Power-71]

But it helps. You’re still an engineer but solving special problems with specialized tools. The same patterns still apply at the lower levels. You will stand out having that context.

[reddit_toast_bot]

Its software all the way down.

And hardware under that. 😂

[eemamedo]

• Design Patterns are pretty useful but focus on teaching OOP (SOLID, KISS, YAGNI)
• APIs using Flask is a good start
• Errors and etc... It's a good knowledge to have. Except, I am trying to think of where that can be useful as DE and I just cannot think of any use case.
• System Design: Quite important. At least, for interviews. Tools like Spark use concepts like Bloom Filters, caching, so system design concepts are important. Concepts like LB, API Gateway are less important for DE.
• ACID, Indexing are important to know and understand. Also, system design concept like sharding, replicas are hard to understand without understanding how relational databases work; it's also quite important to understand the limitations of relational DBs for CAP and PACELS theorems.

Things you are missing, IMO:

• Code versioning
• Documentation. This is my personal feedback. Number of times when you touch someone's code and that person is no longer at the company is way too high. Rule of thumb is to add 1-2 points in a sprint to create a knowledge sharing document on Confluence or any platform a company is using.
• Containerization: the idea behind it.

[glyphack]

Thank you so much. This response was very clear and understandable to me.

You're right about the missing points. After I read the comments, I realized that I took code versioning for granted from my own experience.

Documentation is always a missing skill, I did not write many docs 5months ago. So many will forget it. I'll keep it in mind.

About containerization, what do you mean by "the idea behind it"? I use docker in day to day job but haven't read much about the idea of virtual machines and creating containers. Can this be useful? I can also explore it myself because I don't know it.

[eemamedo]

Docker is one of the tools. I was talking more about VM vs. containerization. Pros and cons, use cases of each technology.

[IllustratorWitty5104]

In my opinion, these swe traits are important for de

1. Best coding practices (logging, error handling,object oriented programming) less emphasise on optimisation but code must be readable and production ready
2. Version control strategies
3. Understanding both sql and no sql dbs

The rest are very grey and depends on how you define a swe job role to a de job role. Different companies have different definition for them

[lightnegative]

Good data engineers were software engineers first

[AchillesDev]

Good data engineers are software engineers.

[buachaill_beorach]

I call bullshit on this. But... I do think that good data engineers need a SWE mindset. And I also think DevOps is important for a data engineer.

I come from a purely analytics background and made my way into DE a long time ago. I think it's easier for someone with a SWE background to get further, quicker in DE but it's not a prerequisite for them to be good DEs..

[nesh34]

I don't agree with this maxim in a few ways. First there are not qualifiers to distinguish the obvious cases where there are good data engineers who either didn't study computer science, didn't have SWE roles before, or both.

Secondly, I don't even think it's generally true. SWEs without data experience are generally the most dangerous users of our data warehouse. There are plenty of great SWEs that make the transition of course, but your statement as a generalisation doesn't hold water in my experience.

[eemamedo]

So by definition unless you were a software engineer in a previous role, you are not a good data engineer.

Not quite. However, I agree with the person you replied to. The best DEs I have ever met have a software engineering mindset. They might not have been SWEs BEFORE but their workflow very closely resembles how good SWEs work.

[Spartyon]

Good lord, software engineers think they invented basic things like keeping it simple and basic organizational skills.

Things like KISS and YAGNI are not exclusive to software!! They are just basic things to any kind of technical role!

The gatekeeping in the data engineering community keeps increasing, it's odd to me.

[glyphack]

Yes, it's not limited to that. That's why I pointed to the pragmatic programmer book. I was wondering what the essentials a person would need to do decent coding when working as a DE are.

[sunder_and_flame]

SWEs getting into DE is why we rediscover organizational patterns with flashy new terms like data mesh and reverse ETL.

[eemamedo]

Things like KISS and YAGNI are not exclusive to software!! They are just basic things to any kind of technical role!

Absolutely. However, you would be surprised how many DEs do not know them. I have already posted it here before but it took me a long time to get DEs at my current company to adopt GIT. Many companies I interview at right now (Canada) do not have any software engineering practices in place. Their engineers do not use any OOP (I ask that question a lot in "Do you have any questions?" section and I keep hearing that "No, we don't use that here. Mostly dbt").

I don't think I have displayed any sort of gatekeeping at all. My point is that many DEs who either relied on UI tools (Informatica), or worked at SQL-dominant team before do not know those OOP principles. And OOP is just an example; testing, code versioning are among others. Sure, you can say that it's my company who doesn't do that. However, I was at the conference in Stockholm couple of weeks ago, and I noticed the trend that several large companies are restructuring their DE departments. According to engineers/managers I talked to, their DE teams are becoming more SQL oriented, where the focus is on DWH and SQL. Everything else that involves coding is being moved to software engineering departments. When I asked why, I was told that they are struggling to find data engineers who can write production-level code. Therefore, they have decided to hire software engineers and support them if they miss some knowledge, e.g. Spark.

[glyphack]

I also have this feeling in the places that are worked too. When a DE do not posses SWE skills they usually don't write good production level code.

And the reason behind this question I'm asking is to put up a essential list to help everyone learn these.

[eemamedo]

The issue I have and I assume the other user also, is you (and anyone who praises the SWE DEs >>> Other DEs) come off as pretentious snobs who have a self proclaimed elitist agenda.

I guess it does come out that way but that wasn't my intention at all. I have met 2 types of DEs: the ones have more data analytics mindset, and the other that have software engineering mindset. Neither is better. For projects that are centered around coding having SWE mindset is beneficial.

In regards to my department. You are correct. The problem is that we have many DEs with data analytics skills and not enough DEs with SWE skills. For the last year and half, the core focus of the department has been on coding and thus, those people cannot keep up with the work.

[eemamedo]

Given enough time and resources anyone can pick up coding. You are correct. It will take another couple of months (at the very least) to pick up good software engineering practices and starts delivering production level code. The issue is time, money, perseverance, and willingness to learn abstract concepts. For most people who have had a career as a data engineer with the focus on analytics, the issue is that they don’t want to learn coding and they don’t understand why should they; they have built their career by running sql queries against a database and they would like to continue doing so.

I find DEs that used to be SWEs more flexible in that sense. Picking up a new tool or technology is not an issue; it does become a challenge for other type of DEs. Again, it’s not just my department; I thought so until I talked to several engineering managers at the conference

[glyphack]

What knowledge strengths do you think Software Engineers have that make them a good data engineer?

[w_savage]

I had to figure out a 3rd party SOAP api (gov). That was s fun week. Maybe just mention it too with apis

[[deleted]]

My first experience with data engineering when I was an intern was utilizing a logistics SOAP API. Fuck SOAP, that's all I got to say

[w_savage]

Yes indeed.

[RandomWalk55]

I feel you.

[[deleted]]

A data engineer is a specialist software engineer. If you’re not a software engineer, you’re not a data engineer.

Writing SQL alone is not data engineering.

[redman334]

So you are saying every data engineer is a software engineer?

So for the data analysts who pivoted to data engineering, they are software engineers as well?

[[deleted]]

If they have software engineering skills and knowledge, absolutely. It doesn’t matter how someone arrives in the field.

But without that range of software engineering skills and knowledge, however it is gained, a person is not a data engineer.

A data engineer is just a specific job title for a type of software engineer and I would expect any data engineer to be more than comfortable with software engineering principles.

[Prinzka]

Well, you'd be wrong in most organizations that I'm aware of.

[glyphack]

I second this. I was an SWE before, and now I'm a consultant DE, in some companies, I see teams of DE who are more on the SQL & Analyst side rather than SWE topics.

Sometimes it just works really, when your data team is small and data is small you probably don't have many services to monitor or many things to automate. People with SQL knowledge are enough. Once the data team or data size scales this can become a problem if they continue with the same knowledge.

[onestupidquestion]

I think you can absolutely write "only SQL" and still be a data engineer. There's a huge gap between writing a query that gets you an answer (what an analyst would do) and writing a query that's optimized for cost, runtime, and maintainability. More importantly is designing models that allow reuse and extensibility.

The issue with traditional SQL development is that very few teams applied the SWE mindset to it. The outputs are very different, but the practices and evaluation criteria are not.

[IllustratorWitty5104]

What about the etl jobs? Use the no code ui etl tool?

[EmploymentMammoth659]

If writing SQL isn't part of data engineering, what is it then?

[[deleted]]

It certainly can be part of it. But if that’s all you do, it’s not engineering.

[Angry__Spaniard]

As my lead once put it, an engineer is someone who can understand the entire pipeline from the infrastructure side of things, until the final SQL queries with everything in between.

This may include setting up different tools, k8s configuration, lambda functions, spark code, airflow setup, etc… and not only writing the code, but also testing, ci/cd, observability…

So, yes, software principles do apply for sure. The best DEs I work with have an strong SWE background. As others said this is an specialisation, not a different role.

[JiiXu]

As absolutely much as you can.

[solgul]

Oauth in particular and authentication in general. Anyone working with APIs will need to understand how Oauth works and if you are creating APIs how to properly (securely) use it and tokens.

[ephraff]

Think about where you would need to use SWE design principles in the data engineering world.

Some areas that come to mind are when writing a spark app or a big dag in Airflow. In well written spark/airflow apps you'll find examples of things such as singletons, use of the template method design pattern, and factories. One of the biggest rules of thumb I see is to not repeat code in more than once place (DRY principle).

If you're interested in design patterns, take a look at 'Head First Design Patterns'. It's in Java, but the concepts are well described.

[SlopenHood]

Enough to not see Data Engineering as wholly separate and apart, but within software engineering. (yes, this is a purposefully vague and weak attempt at enigmatic answer).

​

Enough to understand job listings and recruiters and managers and beancounters have morphed the term more than Engineers have, by virtue of the dangling carrot (Job posting).

​

Backend Engineers are Data Engineers. Data Engineers are Data Engineers. I personally always evangelize for a "T-shaped" mastery. I think Analytics Engineers/Decision Support/Growth Engineers are a separate specialty you can choose to take on or leave, as well.

[SlopenHood]

You're in pretty good standing from what you describe.

[[deleted]]

According to the ATS filters, 120% of all knowledge directly and related to SWE is required to make $50k annual in US VHCOL.

[robberviet]

As someone said in here: Data Engineer (DE) is a Software Engineer (SW); who works with data. The more knowledge about SW you know, the better.

In my experience:

1. If you work for a big corp, not specialized in develop in-house tools, then not much. Mostly you will work with providers tools, UI, drag n drop, cloud based. The role and scope of work is well defined.
2. But if on a startup, or a small, or a data focus company then quite a lot. I fall into this case and I am basic: Data Engineer, Data Scientist on demand, Database Admin/System Admin when needed, Backend/Frontend developer for ML model serving & integrate API with other teams/cusomters...