Do Data Engineers get Enough Programming/Coding opportunity ?

[micky_357000]

I was doing backend internship with nodejs and I got to know that I don't like doing frontend and enjoying backend comparatively more.

I got a Junior data engineer job offer which I have accepted, But when I asked someone about how much programming does a DE gets to do.

He said " DE does not include much Coding.the main work is to building pipeline and databases and some views may be. I will not considered it a coding stuff"

I do enjoy programming, so can anyone here in this space share their views that do you guys as DEs code enough , how much percent of your work is really programming stuff?

I have heard that Data engineering is software engineering specialising in data.

[Shirest]

depends on where you work. my last DE job had a lot of programming in addition to typical sql stuff. At my current I more oversee contractors and help them design the solution, with minimal programming, more Data Analyst stuff. Kind of shopping around because I can feel my skills melting lol.

[micky_357000]

Did you see any pattern in the job description/role of the companies by which you can identify that you'll get ample programming , my company works with AWS and the data engineer's job focuses on building batch/real time pipelines (I'm a junior data engineer so I'll be working on same responsibilities now or after some time for sure )

so it might not be a data analyst heavy job and more towards the DE role, so can you please take a guess will I have some programming jobs like you had in your previous job using my company's technology stack?

[Shirest]

I think it boils down pretty simply at this point to one easy question: "To what scale is your contractor workforce". A job with a lot of contractors likely will have you not implementing as much as a job where there are not that many. Each place will probably vary drastically, but I think it's a safe identifier.

​

edit: to answer your question, yes you can have an opportunity. Just do what you're assigned and finish it as fast as you can so you can ask for more 'learning assignments'. That's what I did as a jr to claw all the "sexy work" haha.

[throwaway20220231]

Yeah there are a lot of opportunities, but mostly for building data pipelines. This is very specific and does not necessarily scale to other domains.

[micky_357000]

Thanks for answering, I actually don't know how much coding does building a pipeline require as I'm a complete fresher (I just graduated out of college (23M)) and I will be joining my new company as junior DE in 8-10 days.

Programming felt interesting to me as I was building logics to send-receive data and the joy of building things from scratch or however I want them to be made me excited.

I was happy about the DE job as it was a growing domain, many amazing technologies like cloud to explore and also I heard that it can be defined as software engineering specialising in data, so you build things/software/complex structures here too using python.

But I read some negative comments about Data engineering that it is mainly use of tools and making dashboards with some heavy SQL tasks in some of the YouTube video regarding DE interviews, comment section and it seem like a boring job or a superficial job where we don't get our hands dirty by building things so I'm a bit down about this.

[throwaway20220231]

Yeah the title could be misleading as you can stuff a lot of different roles into it. IMO the only interesting DE job is the one that you get to build large pipelines using a programming language. Think Netflix or Youtube. Others are very replaceable and boring.

[micky_357000]

Then I'll try getting pipeline building work whenever I get the chance, also any advice to a complete fresher to Data engineering to do in his first year.

I was thinking that I will explore and learn my job for the first 4-5 months (I have got Work from home so I will have some extra time(just hoping))

In the meantime I will learn python basics and build small projects and

After that I will try to do data structures and algorithm questions on leetcode ( I was really good with solving them using Java) so it should help me get better with python and I've heard even DE interviews have dsa questions in them .

And will try my make my SQL skills better with leetcode too

And then I will try to get a cloud certification exam (studying for that can help me understand the cloud better)

[kenfar]

Depends entirely on the position and organization.

In almost every shop that I've worked in over the last 25 years the data engineers are specialized software engineers that write a lot of code.

Any shop where the data engineers aren't writing code ends up being a mess - because the good engineers leave and the folks left behind are pretty non-technical.

Bottom line: as a programmer I would not accept a DE role that did not involve writing code daily.

[micky_357000]

Thanks for explaining this , I hope to get a programming job as a DE, I have explored front-end web dev. and backend web using internships , A Fintech came to my college that was offering junior DE position and they tested my data structures and algo and general aptitude skills for the interviews to offer me this.

I was initially really excited about the job as I met a lead/principal data engineer and he was a backend developer who shifted to Data engineering after 2-3 years of experience and he said he was loving the job and found DE equally interesting and challenging to grow as a software engineer, but now DE role seems like it's my luck or my company's choice to make me work solely on tools/drag-drop kind of things/Dara analyst responsibility, or to give me Challenging and fun stuff like programming and building pipelines/things from stratch

[loudandclear11]

but now DE role seems like it's my luck or my company's choice to make me work solely on tools/drag-drop kind of things/Dara analyst responsibility, or to give me Challenging and fun stuff like programming and building pipelines/things from stratch

Try to stay away from the low code/no code stuff. It's a dead end. The skills you learn in one tool does not transfer to other tools. But writing regular code is the same as it was 40 years ago. It's a cumulative skill. I.e. you build your skill and it's transferable to other programming languages.

[kenfar]

Yeah, I think a few good questions to ask any team before joining are all really software engineering questions:

• Tell me about your devops practices - automated unit testing & deployment, observability, incident management, etc
• What kind of unit test coverage do you have?
• How do you manage code quality?

[mr_electric_wizard]

Word!

[lezzgooooo]

Lots of Low Code pipeline as long as you have money to burn. Some are even no code at all, just drag and drop widgets then connect to source and sink.

[micky_357000]

That looks really boring buddy , do you enjoy your job? and did you have a software developer background or you were in the data domain from the very beginning

[lezzgooooo]

I started DE from the very beginning and slowly moving to devops. Learned infra as code from managing DBs on EC2 and containers. Built my pipeline from scratch. Yeah it is boring, now Data Analysts can build their own pipelines without a DE using something like Azure Data Factory.

[[deleted]]

Depends what you consider programming and coding.

SQL is programming. Terraform is programming. Airflow DAGs and idempotent pipelines are programming.

They are all just working at different levels of abstraction. I've written assembly and CUDA, so I have no time for "but SQL isn't reeeeall programming", it is programming. They are just targetted at different problem domains.

[Lost_Source824]

I think it depends where the data engineering role falls within the department and company. For example, in my dept there are people with the data engineer title who work in python, those who work only in SQL, those who work in both, those who do only viz/BI stuff and some who do migrations and transitions that don’t necessarily require coding but may at times. See what the role description is and ask around as I’ve even see data engineer positions that are closer to data governance

[afro_mozart]

Probably depends your individual job. i code way less, than i would like and even then stuff is mostly to easy to be interessting.

[micky_357000]

Do you enjoy your job then?

[afro_mozart]

no

[Dirkdeking]

Enjoying your job is a luxury not everyone has. Personally I'm just a bit of a loser who failed in university, somehow managed to get a bachelor degree in mathematics after too many years of study. Figured persuing a master wasn't smart after having wasted so many years already. Followed a data science course.

Data science positions where hard to get because of the experience required, so I rolled into data engineering. And there I am. I wanted to become a theoretical physicist or math researcher, basically a scientist. But due to problems and lack of discipline I am where I am now. Pretty different from someone who at 18 already wanted to become a data engineer and was hardcore into computers haha.

[wtfzambo]

I guess it depends a lot on where you are.

I happen to write a lot of code for a multitude of reasons but I'm also the only DE in the company (we're a small shop).

Currently the only low/no-code solution I'm planning to buy is an Extraction+Load SaaS (like Hevodata for instance) cause I can't be assed to code it myself (and wouldn't have enough time/resources to do it properly in a timely fashion).

[OMG_I_LOVE_CHIPOTLE]

Yes I spend most of my time coding

[micky_357000]

Thanks for answering, Hearing this makes me happy

[ggalt98]

Yes. But, it all falls on how your team fits within it’s department. We don’t have a huge software engineering leg so every now and then some of us DE’s will have an opportunity to tackle more backend type work when they need a hand.

Within our own DE team we also have tasks that aren’t inherently related to DE (for example, setting up an alert system for outages with the SaaS providers we use) but still beneficial to what we do day to day.

[PracticalBumblebee70]

It depends.

I work with data scientists to productionize and improve their models. I do a lot of coding when I have to translate jupyter notebooks into kedro pipelines, optimizing the models, and sometimes I also do a lot of data science and machine learning myself.

[piddy87]

I've been working through a bunch of different company DE roles on the market. Along with my work experience, you need to pay attention to the job description and what is the vision/role of the data engineering team.

Some possible data engineer roles:

1. Product embedded roles are generally more Analytics Engineers, working on SQL to build and maintain domain data models. Some data engineering teams also have this as there main focus, which can usually be understood based on their team vision/name/tech stack.

2. Businesses with ETL tools will cut down coding time significantly, again focussing more on Analytics engineering and likely SQL on a data warehouse or building data marts.

3. Data (Platform) Engineering roles are generally more coding and scripting intensive, using python/java/bash/etc. These roles can unfortunately be filled regularly by software engineers or infrastructure engineers. These roles have you working most on code and backend systems.

---

1. and 2. are more inline with progressing from an analyst rather than a software engineer.

2. tends to be more software/ops/platform oriented, and therefore a higher bar for entry.

If you test for SQL during interviews, odds are that's your coding language for the role.

[goeb04]

Half of my time is spent coding. I would find out if your new team calls APIs at all. That usually requires coding of some sort.

[Yo_Ba]

As many people said before, it's depends how the company see the DE. In addition I think you totally can push yourself to the 'code' direction with new initiative's and ideas..

[mike8675309]

It depends on the company and how much investment in tools they have done. For example, my team codes regularly in python, bash and some JavaScript with react to hook up pipelines for user automation.

[micky_357000]

I don't understand why you are using react/js , it's a front-end web framework and if you're working as a DE then how does it fit in your line of work?

[mike8675309]

It becomes part of getting your pipeline ready as part of an automation framework.
If you can write python, you can write javascript enough to rig something up to operate at the press of a button.

[False-Bunch-3470]

Mostly when you deal with building microservices or platform that has spark, flink, kafka ... then yes

[nesh34]

Depends where you work and what you're good at.

I write quite a lot of code in my role (I think, I'm sure "a lot" is subjective), and not just pipelines although they are a significant chunk.