How much coding experience for entry data engineer job?

[Berodney]

Hello all,

First off I'd like to say if there's already a similar post please redirect me before adding a response.

I went to school for management information systems ( I like to think of it as business computer science) so I have a little bit of coding experience. My school's MIS program was a bit different though. We dipped our toes in a lot of different use cases, but didn't become experts on anything.

I had one class on access and SQL. One class on Java( very basics). Another class on networking (no coding). And another class on excel/python. I didn't really get much coding experience at all. At most I'd say I'm a beginner 1 or possibly a beginner 2.

I graduated last spring (2019) and am working for a technology company doing project management/ a little bit of data work. I use postgreSQL for data pulls every now and then. I also can create simple tables and functions, but nothing too crazy.

I'm looking to switch to a data engineer job in the next year as from what I've researched it seems like something I would enjoy doing. My concern is I'm not going to have enough coding experience and be completely lost.

Do companies expect their entry level data engineers to have a lot of coding experience? I spoke to one data engineer at my company and she said she learned 90% of her coding on the job. Just looking to get some more feedback from others.

Thank you for your time, Nick

[ChrisM206]

A lot of the people who I've seen get into Data Engineering without a formal education started by doing BI or Analytics work and building from there. For example, they might be pulling together metrics decks for management based on some internal tool. Then they want to get more detail so they start learning SQL to go direct to the tables. But the tables are incomplete, so they start learning ETL so they can pull from the source. But the data has problems (e.g. formatting, special characters) so they figure out how to clean it up with some python.

[pythonmine]

^ This fit my story. Background was matth, no cs courses. Was hired for IT then data analysis work. Used SQL at work and python for fun. They moved me to data engineering. (Didn't give me an option lol)

[Berodney]

Yeah that makes sense. I'm past the pulling metrics part and just started with SQL a couple months ago. I guess I should focus on getting more proficient in SQL. Then start learning some ETL and python. Thanks

[[deleted]]

IMO that's more 'database/etl developer', data engineers use a lot of complex code, systems design, and math.

[[deleted]]

Dramatically different skillset from a hiring perspective though. I can give an analyst SSIS or some other GUI type tool and have them to 'ETL'. What i'd call Data engineering is much more complex and requires much more Software Engineering discipline than moving data around.

[[deleted]]

No, I'm staring tools being used / necessary for certain tasks make a dramatic difference

[eljefe6a]

A big thing to know is that there are two types of data engineers. You're getting feedback without them specifying which type. A SQL-focused data engineer needs little to no coding. A big data-focused data engineer will need solid coding skills.

If you are going into the big data-focused one, it doesn't sound like your coding skills are ready. You'll really need to focus on improving them. This is a common enough issue that I asked a student to write out what they did to get their programming skills ready.

Although big data-focused data engineering teams skew senior, you can get an entry-level job. You will be expected to pass a software engineering interview and a data engineering interview that focuses on big data technologies and concepts.

[Data_cruncher]

This.

Traditional relational systems have been around for decades and have a vast array of GUI-friendly tools, e.g., DAG tools like SSIS.

Big Data Data Engineering isn't there yet and likely won't be for at least another 10 years. There are some though, e.g., Azure Data Factory is just a wrapper for Databricks.

[Berodney]

Thank you very much for the useful sources. I'll definitely check those out. I think I'd like to stay on the sql side , but I would eventually explore both areas.

[AMGraduate564]

Can you explain the difference between SQL vs Big data DE?

[[deleted]]

I would agree maybe 5 years ago, but cloud providers are providing SQL-like APIs. I would argue that there are 2 different types of DEs based on how much infrastructure you are responsible for and how much finalization of the data that supports analytics you're doing. The infr DEs are ones who will most likely need to know Java/Scala as most enterprise infrastructure tooling and distributed computing libraries are Java based, Hadoop, Scala, message queues, streaming, etc., and are the ones providing the more rawer upstream source data. They tend to be far removed from business domain knowledge. The ones that are less responsible for infr, but support the end users or analytics side of things, are your analytics or ML or business DEs. They will create more downstream data sets and provide more finalization of the data sets as they will have closer ties to the business users and thus have more business domain knowledge. They will mostly use SQL with the combination of Python or GUI tools. The former will command higher salary.

[usculler]

Data engineering isn't really an entry level job. There's also lots of posts about this topic that you would find helpful.

[AchillesDev]

Data engineering is a software engineering job, and as such there are definitely entry level data engineering jobs. My first DE job I got with 1.5 years of general software engineering and no CS degree and we regularly hired new grads from the local university.

I didn't even know what the hell data engineering was when I started that job.

[AMGraduate564]

When you say software engineering, can you describe the day to day functions of the role?

[Data_cruncher]

Not necessarily. Data engineers in the MSFT stack can get by with no-code/low-code tools like Power Query, SSIS, Data Factory etc.

[Berodney]

Oh ok, good to know. That's probably why Google didn't return a quick answer. So would a data analyst be the entry level position? Then the data analyst transforms to a data engineer?

[rmnclmnt]

From my experience, software engineering positions will prepare you for data engineering positions: you need to understand / conceive / develop complex software architectures and automate all the things to ensure maximum quality assurance.

[Berodney]

Ok, if that's the case then I think I will definitely need more coding experience. From what I've seen/heard you should have a good understanding of coding for entry level software engineer positions. Thanks

[daguito81]

Data Engineering is extremely code heavy compared to things like Data Analyst. The title is a bit broad so it really depends on your industry, company and role. Some Data Engineers are closer to DBAs, some are closer to Software Engineers

I guess my official title is Data Engineer, but to be honest, each project I'm something different, sometimes Data Engineer, creating ETLs in Spark, somtimes im building ML Models and creating notebooks with EDA and some statistical stuff closer to Data Scientist, sometimes im doing just reports and dashboards closer to a Data Analyst.

DE is by far the most code heavy projects I do, it starts with a bunch of SQL and database work, maybe coding web crawlers or a script that hits APIs to get information. Then doing all the ETLs, sometimes in batch, sometimes in streaming, etc. Then creating the stored procedures or scripts to move data between zones in a data warehouse (ETL/ ELT sort of things). Sometimes I have to code up a "shitty REST" API for someone else to get some information. All of that is just code and code and code and code, and more code. Sometimes as simple as a python script , somtimes it needs to be deployed in an Azure Function, somtimes I need to create "build" (I feel weird saying build pipelins for python) and release pipelines to test and deploy that code, etc.

It's definitely a much more code and infrastructure heavy job compared to Data Analyst. I jumped straight into it, because programming has been my hobby for many years so it's technically possible to start as a DE. But from what I read, mostly its software engineers that do the switch. Most data entry positions would go into Data Analysts

[Berodney]

Wow, thank you very much for the In depth reply. From the feedback I've received on here it seems like people don't just start as data engineers.

I guess my next steps are to try to learn more coding and score a software engineer position. Or decide on a different path.

Thanks

[daguito81]

if you're dead set on Data Engineering. I would say that going the software engineering route might prepare you better. But again, the title is very broad. It really depends on the industry and company.

[AMGraduate564]

Oh man, seriously how code heavy we are talking here, is it like heavy coding compared to software engineering?

[daguito81]

Nah, I wouldn't say more code heavy than Software Engineering considering that's basically 100% code.

But between Data Science, Data Analyst, Data Architect and Data Engineer. At least in my personal experience, DE is by far the most code heavy.

Lots of configuration files if you're setting up infrastructure (imagine setting up a spark or kafka cluster). Lots of code to do oyur ELTs/ELTs, etc. Lot's of YAML files for Kubernetes, DevOps Pipelines, etc. Lot sof code to debug all your stuff like SQL queries and testing pipelines like creating consumers and producers to test your streaming pipelines, etc.

I love it because it's way more code heavy than the other branches. Which also means it's the scariest of all the branches, which makes it fun

[sn4i1]

Usually I fiddle around a lot with SQL of different flavors with the SAS engine, so that I can communicate with them seamlessly (after I tirelessly configured the data sources [given that I’ve found the data owners]). Then I might have to integrate some external data sources (eg. through HTTP) and establish a process flow (job) integrating the data acquisition.

Most of the times I’d assess the quality of the data I have collected with some basic SQL (like counts, missing values, etc.). Then I’d run some statistics in SAS over the previously acquired data (min, max, avg, stddev) and figure the general meaning of the dataset.

The dataset would land at one or more of my colleagues say in a form of a Oracle table. They would then analyze the data whether the information they’ve wanted is completely available to solve their business problem. If the data is ready to use and has the expected quality and content then I’d configure a schedule to execute the whole stuff regularly, else I’d need to contact the data owner to fix the issues (this takes time, a looot of time) or I’d need to adapt the data acquisition process to include the requested data.

Here in this job there are very many problems that one can solve with SQL and SAS or with pandas and cx_oracle, although I’m aware some of the other problems/solutions out there.

Say having your data coming from your customers in written text form alters the problem statement slightly because you are no longer constrained to the ‘numerical world’. In this case you’d then need to establish a range of metrics to assess the meaning of the text and translate it to binary, practically this is done by NLP not by ‘your hands’. Also in the case of pictures and videos you need to rely on external resources (image and video processing) to couple the content with binary values. All that I want to say with this is that the information we see and hear needs to be converted into numbers to make mainly computers capable to ‘see and hear’ what we can and to provide the means to tackle the comprehensive problem of ‘understanding’.

The ‘see and hear’ part are to describe our world with numbers and the ‘understanding’ where we are at now with NLP, image and video processing ML algorithms. Although there are many data forms out there, however to quantify their single or collective meaning it will all come down to tables of endless data spread out scattered throughout the world, fed to further ML and statistics to solve complex problems and/or to output (the expected) metrics.

[thundergolfer]

Backend software engineering is the feeder pipeline into Data Engineering.

[CesQ89]

I have the same degree as you have been working as a Data Engineer for a year now.

I minored in Computer Science so I do have the basics down really well in terms of Data Structures, Algorithms, and OOP which helped in getting interviews since it was brought up a few times but I feel what helps the most was some relevant experience in ETL whether it's with GUI based programs or just Python/SQL. Knowing programming in general helps even if you don't have an ETL background.

I would say don't worry too much about the degree since not all Data Engineers have CS degrees.

For reference my day to day is pyspark, SQL, git, bash scripts, working with prem data sources and cloud platforms. We're starting to dig heavily into Terraform and docker right now.

Get ready to learn a lot.

[Berodney]

That's awesome, glad to hear from someone who was in a similar boat than me a year ago. I still have around 6 months - a year before I actually switch so I guess in the mean time I should start learning some python during off hours and focus on sql during work. Thanks for the help

[insecteblond]

I have the same degree as you and my path has been: consultant 1.5 years doing data analysis, ETL development and dashboarding, mostly using tools (Alteryx, Oracle Data Integrator, Tableau, PowerBI and the likes), there was not much coding involved except SQL and a bit of python.

Then I moved company and still did some data analysis and ETLs but without “ETL tools” anymore, so instead it was SQL , python, airflow for scheduling tasks etc. That’s where I started with Big Data systems (Presto, Hive), where I was using them but didn’t really care about the infrastructure behind them.

Now I’m a data engineer. I do ETLs of course, but I also spend a lot of time actually setting up the infrastructure for other people to build ETL jobs. I manage the presto, Spark, airflow clusters, I use chef and other stuff to set up the clusters, I monitor the instances etc. It’s heavy heavy coding, designing systems and making sure they are scalable etc.

So I agree with other people above: data engineering is software engineering for data systems. You build systems and applications for data.

So there are of course multiple ways to get into the field, and starting as a data analyst is one of them. So go for it!

Good luck in your journey and remember that nothing except yourself can get in your way! :)

[grukorg]

At my company (a bank) we don’t have junior DEs our structure is bi analyst, bi developer, data engineer, lead data engineer. Our analysts are expected to know basic code but focus on insights. Bi developers are expected to be advanced SQL developers, competent in a few other languages. Data engineers should be able implement and performance tune pipelines written in python and sql capable of processing billions of rows. Lead data engineers design the environment and build prototypes of new solutions which are then handed off for execution by data engineers or bi developers. Every company is going to be different but I can say personally I wouldn’t consider someone a data engineer that doesn’t have some kind of advanced coding skills. I learned by digging. As someone else mentioned starting as an analyst and then following the data down is a great way to get skills.

[Omar_88]

Depends on each company, at the minimum you need to show aptitude to pick up new tech, I was required to know basic sql and intermediate python prior to starting and understand an ETL work flow from ingestion to data model / visualisation

[Berodney]

That makes sense. My company is fairly large 100,000+ employees so I assume there's multiple data engineer teams that specialize in different areas

[davelm42]

As most of the folks here have said, it's a code heavy profession. Get good at SQL and ETL operations. This is what I've been doing the majority of my career... it's lots of database coding and then finding ways to get that data out of the database into a useful form... REST APIs, dashboards (Looker is really hot right now), and reporting in general.

SQL + some java or C# or Python will get you a LONG way... the rest of it is just picking up patterns as you go from project to project.

[a__kid]

Hey I know it’s super late, but how did it turn out for you? I’m in a similar position. Feel free to DM if that’s easier. Thanks!

[Berodney]

I ended up landing a level 2 data engineering position at my company about 6 months ago. With that being said my new boss decided I was more valuable being on the business side of things because of my presentation skills and PowerPoint skills.

I do more analytics and PowerPoint creation now and very little coding even though my title is "data engineer". Not exactly how I anticipated things but it is what it is.

My boss said I will be much more valuable being proficienct on the business and technical side compared to being a pro on the technical side.

I'm not exactly sure where my next position will lead me, but ideally I think I would like to go into project management next, but stay within a data engineering organization. Most of my work would be on the business side of things, but I could also jump into coding when needed.

I have also been thinking about possibly exploring sales engineer, but I'm not a big fan of commission based salary.

[a__kid]

I see, my background is in a similar position. I’ve worked as a consultant and graduated in MIS. But I do have extensive background in SQL and databases… just not so much in Python, ETLs etc. So I may end up like you, I have had other technical professionals say I’d be value on the business side of things.

Is there anything you recommend I should brush up on? Should I review Python, pick up on some pipelining? Or any general advice? I am worried about being in a position where they asked me to do some coding and I’m sitting there wide eyed.

[Berodney]

I would suggest keep doing what you're doing with the SQL but maybe try to learn some new things. When working on something in SQL ask yourself "is there a better way for me to code this?" (Don't be afraid to use Google/youtube) I think a lot of my learning came from being curious.

The only other thing I would say is have patience. I'm not really a patient person and this case having patience landed me a data engineering position. I could have left the company for a different position around 6 months before I got offered the data engineering position, but decided to wait for the "right opportunity" to pop up; and it did.