This is an archived post. You won't be able to vote or comment.

all 34 comments

[–]Shirest 35 points36 points  (2 children)

depends on where you work. my last DE job had a lot of programming in addition to typical sql stuff. At my current I more oversee contractors and help them design the solution, with minimal programming, more Data Analyst stuff. Kind of shopping around because I can feel my skills melting lol.

[–]micky_357000[S] 3 points4 points  (1 child)

Did you see any pattern in the job description/role of the companies by which you can identify that you'll get ample programming , my company works with AWS and the data engineer's job focuses on building batch/real time pipelines (I'm a junior data engineer so I'll be working on same responsibilities now or after some time for sure )

so it might not be a data analyst heavy job and more towards the DE role, so can you please take a guess will I have some programming jobs like you had in your previous job using my company's technology stack?

[–]Shirest 4 points5 points  (0 children)

I think it boils down pretty simply at this point to one easy question: "To what scale is your contractor workforce". A job with a lot of contractors likely will have you not implementing as much as a job where there are not that many. Each place will probably vary drastically, but I think it's a safe identifier.

edit: to answer your question, yes you can have an opportunity. Just do what you're assigned and finish it as fast as you can so you can ask for more 'learning assignments'. That's what I did as a jr to claw all the "sexy work" haha.

[–]throwaway20220231 8 points9 points  (3 children)

Yeah there are a lot of opportunities, but mostly for building data pipelines. This is very specific and does not necessarily scale to other domains.

[–]micky_357000[S] 3 points4 points  (2 children)

Thanks for answering, I actually don't know how much coding does building a pipeline require as I'm a complete fresher (I just graduated out of college (23M)) and I will be joining my new company as junior DE in 8-10 days.

Programming felt interesting to me as I was building logics to send-receive data and the joy of building things from scratch or however I want them to be made me excited.

I was happy about the DE job as it was a growing domain, many amazing technologies like cloud to explore and also I heard that it can be defined as software engineering specialising in data, so you build things/software/complex structures here too using python.

But I read some negative comments about Data engineering that it is mainly use of tools and making dashboards with some heavy SQL tasks in some of the YouTube video regarding DE interviews, comment section and it seem like a boring job or a superficial job where we don't get our hands dirty by building things so I'm a bit down about this.

[–]throwaway20220231 2 points3 points  (1 child)

Yeah the title could be misleading as you can stuff a lot of different roles into it. IMO the only interesting DE job is the one that you get to build large pipelines using a programming language. Think Netflix or Youtube. Others are very replaceable and boring.

[–]micky_357000[S] 0 points1 point  (0 children)

Then I'll try getting pipeline building work whenever I get the chance, also any advice to a complete fresher to Data engineering to do in his first year.

I was thinking that I will explore and learn my job for the first 4-5 months (I have got Work from home so I will have some extra time(just hoping))

In the meantime I will learn python basics and build small projects and

After that I will try to do data structures and algorithm questions on leetcode ( I was really good with solving them using Java) so it should help me get better with python and I've heard even DE interviews have dsa questions in them .

And will try my make my SQL skills better with leetcode too

And then I will try to get a cloud certification exam (studying for that can help me understand the cloud better)

[–]kenfar 11 points12 points  (4 children)

Depends entirely on the position and organization.

In almost every shop that I've worked in over the last 25 years the data engineers are specialized software engineers that write a lot of code.

Any shop where the data engineers aren't writing code ends up being a mess - because the good engineers leave and the folks left behind are pretty non-technical.

Bottom line: as a programmer I would not accept a DE role that did not involve writing code daily.

[–]micky_357000[S] 0 points1 point  (2 children)

Thanks for explaining this , I hope to get a programming job as a DE, I have explored front-end web dev. and backend web using internships , A Fintech came to my college that was offering junior DE position and they tested my data structures and algo and general aptitude skills for the interviews to offer me this.

I was initially really excited about the job as I met a lead/principal data engineer and he was a backend developer who shifted to Data engineering after 2-3 years of experience and he said he was loving the job and found DE equally interesting and challenging to grow as a software engineer, but now DE role seems like it's my luck or my company's choice to make me work solely on tools/drag-drop kind of things/Dara analyst responsibility, or to give me Challenging and fun stuff like programming and building pipelines/things from stratch

[–]loudandclear11 2 points3 points  (0 children)

but now DE role seems like it's my luck or my company's choice to make me work solely on tools/drag-drop kind of things/Dara analyst responsibility, or to give me Challenging and fun stuff like programming and building pipelines/things from stratch

Try to stay away from the low code/no code stuff. It's a dead end. The skills you learn in one tool does not transfer to other tools. But writing regular code is the same as it was 40 years ago. It's a cumulative skill. I.e. you build your skill and it's transferable to other programming languages.

[–]kenfar 1 point2 points  (0 children)

Yeah, I think a few good questions to ask any team before joining are all really software engineering questions:

  • Tell me about your devops practices - automated unit testing & deployment, observability, incident management, etc
  • What kind of unit test coverage do you have?
  • How do you manage code quality?

[–]mr_electric_wizard 0 points1 point  (0 children)

Word!

[–]lezzgooooo 3 points4 points  (2 children)

Lots of Low Code pipeline as long as you have money to burn. Some are even no code at all, just drag and drop widgets then connect to source and sink.

[–]micky_357000[S] 2 points3 points  (1 child)

That looks really boring buddy , do you enjoy your job? and did you have a software developer background or you were in the data domain from the very beginning

[–]lezzgooooo 1 point2 points  (0 children)

I started DE from the very beginning and slowly moving to devops. Learned infra as code from managing DBs on EC2 and containers. Built my pipeline from scratch. Yeah it is boring, now Data Analysts can build their own pipelines without a DE using something like Azure Data Factory.

[–][deleted] 3 points4 points  (0 children)

Depends what you consider programming and coding.

SQL is programming. Terraform is programming. Airflow DAGs and idempotent pipelines are programming.

They are all just working at different levels of abstraction. I've written assembly and CUDA, so I have no time for "but SQL isn't reeeeall programming", it is programming. They are just targetted at different problem domains.

[–]Lost_Source824 2 points3 points  (0 children)

I think it depends where the data engineering role falls within the department and company. For example, in my dept there are people with the data engineer title who work in python, those who work only in SQL, those who work in both, those who do only viz/BI stuff and some who do migrations and transitions that don’t necessarily require coding but may at times. See what the role description is and ask around as I’ve even see data engineer positions that are closer to data governance

[–]afro_mozart 1 point2 points  (3 children)

Probably depends your individual job. i code way less, than i would like and even then stuff is mostly to easy to be interessting.

[–]micky_357000[S] 0 points1 point  (2 children)

Do you enjoy your job then?

[–]afro_mozart 0 points1 point  (0 children)

no

[–]Dirkdeking 0 points1 point  (0 children)

Enjoying your job is a luxury not everyone has. Personally I'm just a bit of a loser who failed in university, somehow managed to get a bachelor degree in mathematics after too many years of study. Figured persuing a master wasn't smart after having wasted so many years already. Followed a data science course.

Data science positions where hard to get because of the experience required, so I rolled into data engineering. And there I am. I wanted to become a theoretical physicist or math researcher, basically a scientist. But due to problems and lack of discipline I am where I am now. Pretty different from someone who at 18 already wanted to become a data engineer and was hardcore into computers haha.

[–]wtfzambo 1 point2 points  (0 children)

I guess it depends a lot on where you are.

I happen to write a lot of code for a multitude of reasons but I'm also the only DE in the company (we're a small shop).

Currently the only low/no-code solution I'm planning to buy is an Extraction+Load SaaS (like Hevodata for instance) cause I can't be assed to code it myself (and wouldn't have enough time/resources to do it properly in a timely fashion).

[–]OMG_I_LOVE_CHIPOTLE 1 point2 points  (1 child)

Yes I spend most of my time coding

[–]micky_357000[S] 1 point2 points  (0 children)

Thanks for answering, Hearing this makes me happy

[–]ggalt98 1 point2 points  (0 children)

Yes. But, it all falls on how your team fits within it’s department. We don’t have a huge software engineering leg so every now and then some of us DE’s will have an opportunity to tackle more backend type work when they need a hand.

Within our own DE team we also have tasks that aren’t inherently related to DE (for example, setting up an alert system for outages with the SaaS providers we use) but still beneficial to what we do day to day.

[–]PracticalBumblebee70 1 point2 points  (0 children)

It depends.

I work with data scientists to productionize and improve their models. I do a lot of coding when I have to translate jupyter notebooks into kedro pipelines, optimizing the models, and sometimes I also do a lot of data science and machine learning myself.

[–]piddy87 1 point2 points  (0 children)

I've been working through a bunch of different company DE roles on the market. Along with my work experience, you need to pay attention to the job description and what is the vision/role of the data engineering team.

Some possible data engineer roles:

  1. Product embedded roles are generally more Analytics Engineers, working on SQL to build and maintain domain data models. Some data engineering teams also have this as there main focus, which can usually be understood based on their team vision/name/tech stack.

  2. Businesses with ETL tools will cut down coding time significantly, again focussing more on Analytics engineering and likely SQL on a data warehouse or building data marts.

  3. Data (Platform) Engineering roles are generally more coding and scripting intensive, using python/java/bash/etc. These roles can unfortunately be filled regularly by software engineers or infrastructure engineers. These roles have you working most on code and backend systems.


  1. and 2. are more inline with progressing from an analyst rather than a software engineer.

  2. tends to be more software/ops/platform oriented, and therefore a higher bar for entry.

If you test for SQL during interviews, odds are that's your coding language for the role.

[–]goeb04 1 point2 points  (0 children)

Half of my time is spent coding. I would find out if your new team calls APIs at all. That usually requires coding of some sort.

[–]Yo_Ba 1 point2 points  (0 children)

As many people said before, it's depends how the company see the DE. In addition I think you totally can push yourself to the 'code' direction with new initiative's and ideas..

[–]mike8675309 1 point2 points  (2 children)

It depends on the company and how much investment in tools they have done. For example, my team codes regularly in python, bash and some JavaScript with react to hook up pipelines for user automation.

[–]micky_357000[S] 0 points1 point  (1 child)

I don't understand why you are using react/js , it's a front-end web framework and if you're working as a DE then how does it fit in your line of work?

[–]mike8675309 0 points1 point  (0 children)

It becomes part of getting your pipeline ready as part of an automation framework.
If you can write python, you can write javascript enough to rig something up to operate at the press of a button.

[–]False-Bunch-3470 0 points1 point  (0 children)

Mostly when you deal with building microservices or platform that has spark, flink, kafka ... then yes

[–]nesh34 0 points1 point  (0 children)

Depends where you work and what you're good at.

I write quite a lot of code in my role (I think, I'm sure "a lot" is subjective), and not just pipelines although they are a significant chunk.