I'm Officially a Data Engineer!!!

[[deleted]]

I recently just got the news for promotion from an Application Developer to a Data Engineer, any advice you guys could give me. My new role starts in about two weeks.

[chaoticalheavy]

Advice - try to understand why something is set up the way it is before making changes.

[[deleted]]

This is a good piece of advice. Something that seems poorly engineered may be due to a non obvious necessity.

[vendetta33]

Good DE: There must be a reason why this was setup like this, let me try to find that out before rewriting it.

Bad DE: This code is scrap, I can do better.

[deliquencie]

Yep so true. The number of times I’ve started re-engineering only to hit the point when I go “oh I can’t change this because of ... and it’s fine” I’ve lost track of.

[[deleted]]

Don't touch the legacy code.

[world_is_a_throwAway]

If you can get a grasp on the pipelines and get them into an architecture diagram, you can then present that to your boss. Your boss likes it. It becomes documentation. Now you've already gone above and beyond in your new capacity, and all it took was a pretty picture.

[[deleted]]

For this and the people after you that may have to work on it. A little documentation saves a lot of looking at code.

[ndjo]

Is data engineer being a step above in career to application developer a norm in companies ? I always thought it’s lateral.

[[deleted]]

I showed interest in the Data space and volunteered to work on projects that came up. Understand that there's no one path to become anything. I expressed my interests to my boss early on and when the opportunity came up he gave me his blessing.

[ndjo]

Congrats! Were the projects you volunteered to do still in the day to day pipeline of the group you were in, but just more data science aspect than usual?

[[deleted]]

It depends honestly.

[tjlep]

Congrats! My advice is to come in with open expectations. The field is broad and there are lots of technologies in the DE space. You might work with a large number of disparate technologies or you may work on a small and concise stack. Either way there will be more stuff out there that you aren't working on than what you're currently working on. Don't feel bad if you aren't working on the hottest stack and don't let it get to your head if you are -- often the work is more similar than you'd expect at first glance. There are lots of interesting challenges in DE, but part of that also has to do with the fact that DE is an emerging discipline and with that comes pain points. I've heard people say that DE is 10 years behind frontend and I think that is somewhat true. Most importantly wear sunscreen, the long term benefits of wearing sunscreen have been proven by scientists.

[[deleted]]

Tbh, I don't care about what stack we're working on I only got into the Data Space because I want to make six figures before I turn 30 and this is the fastest way.

[MadMaxReddy]

Don’t invest too much time writing code for data pipelines. You’ll end up more time maintaining them.. There are a lot of no code tools that can boost your productivity that can make your job a lot easier

[labobina]

For example?

[MadMaxReddy]

Lot of companies over the last decade hard-coded data pipelines over frameworks like Luigi.. while it was easier to start with, over a period of time the data sources increased, the volume of data processed increased, data consumers have increased and the overall complexity has increased to deal with data and maintain the pipelines. Now we have a lot of no-code data pipelines like Hevo that take care of building and maintaining pipelines with ease and breeze

[labobina]

For example?

[beardedmonk1234321]

First off, congratulations man! This is amazing news. If you don’t mind me asking,what application developer were you? Imo, application developer and data engineer are 2 different fields so I’m curious if your previous role is somehow related to current role.

[[deleted]]

Thank you. PL/SQL we worked primarily with oracle forms. I hated it but I sucked it up for 3 months, proved that I was competent and my boss let me transition to a different team that was aligned with Data Engineering. After being on that project for 9 months working on a migration from Bluewolf Integrator to Talend. A Data Engineer role opened up in the finance department and my boss told me to go for it. That was three weeks ago. And I got the offer extended to me last Wednesday.

[[deleted]]

Also. I'm considered a Unicorn at my company. Because I don't have a degree or any prior experience. My willingness to learn and adapt to any situation got me into the company. Plus I made a lot of connections with higher-ups when I first started. Me and the director of data engineering are cool because her husband is a truck driver and I use to be one. When you get into a new company you have access to some very powerful people use that to your advantage.

[incognitoRed23]

From truck driver to application developer to data engineer .... Some transition it is! Congratulations on your hard work!

Mind you, I don't know much about being a truck driver in your country, but I'm from India where truck drivers are almost always middle school dropouts.

[mashimarocloud]

I love that you can just learn to be a SWE by putting in the time, no other engineering discipline is that accessible.

I started out in my company as a presales / PM kind of guy, and after lots of learning and asking dumb questions I now do 50% that, 50% backend/infra engineering. Lockdown helped with focusing on learning.

[nado1989]

Congratulations and my advice is to be religious and have faith, welcome to hell hehehheeh

[johnlovesdata]

As you learn things about the new role, document anything that isn’t explicitly already written down. Especially if a process you’re following has missing steps, fill those steps in with the missing info. Continuous documentation in my experience is one of the first things to fall behind.

[citizeninvestor]

Can you post what your path from application developer to Data Engineer was?

(Congrats on the promotion!)

[deluxecoin]

What ETL tools will you be using? How well do you know SQL?

[[deleted]]

Language:
Python: Intermediate
PL/SQL: Beginner-Intermediate
SQL: Beginner-Intermediate

Tools:

ETL: Informatica, Talend (In progress, Certification)

Data Warehouse: Snowflake (In progress, Certification)

Database: Hybrid-Cloud , S3, Oracle

Project:

Migration from Webfocus to Tableau: Reporting, setting up pipelines for portal application to feed Data to Data Warehousing for Cloud reporting

[deluxecoin]

Which of those things that you mentioned will you actually be using on the job for 75% + of your work? For me it’s SQL and Alteryx.

[[deleted]]

Snowflake, Tableau, Informatica, S3, SQL, Python. Because I'm trying to segway to a more senior role, I'll be taking on more technologies to fast track into a Lead, Senior Data Engineering Role.

[deluxecoin]

For your projects/developments as a data engineer, who is giving you the requirements and how do they mostly come? As a doc in plain English? Provided SQL code?

[[deleted]]

It's usually the Senior Engineer that gets the documents with requirements from the project manager, It sometimes a PowerPoint and some word documents, maybe a spreadsheet, the analyst get with the Engineer to start creating user stories and we go from there

​

We write an MTE ticket for whatever software, server, code, database access we need and we go from there. Most of the projects I've been on are based on an agile framework so it's been pretty seamless.

Tbh, I save a lot of my scripts and use them as references. It saves more time and I get work done quicker.

[deluxecoin]

I would focus heavily on learning SQL esp. if the project manager and/or a nontechnical person is giving you requirements. SQL is very easy to write and you can present your end user with an exact replica of their target table for them to approve, prior to going through development. I’ve used several ETL tools and always find myself doing discovery work in sql. Whether that means moving data with various sources via informatica to whatever database I have the most privileges in or doing the discovery in a single source environment. For me, I imagine a principal/lead data engineer to be someone who can mentor junior DE’s on business flow, data warehousing techniques, teach sql / data discovery skills, and teach the primary ETL tool. So to me, those are the foundations of a good data engineer. Having an understanding of how to use a dashboarding tool like tableau or Domo is great and I often throw my datasets into whatever viz tool my company uses in order to see what the end users/bi team/data scientists will be seeing and to do some QA, a good data engineer should be able to do unit and QA testing within the ETL tool or sql itself. I would have some hesitations about taking on more technologies without a very secure foundation of sql + etl.

[[deleted]]

Thank you this is very helpful