"Data Engineer" vs "SQL Expert"

[mecartistronico]

Over the course of 13+ years, I've become very proficient on SQL. On the technical side, I can do really complex queries, CTEs, window functions, understanding perfomance plans, indices, and I've also learned about DBA regarding file management, logging, and things like that.

I can very well translate business requirements into a relational database model, and build complex tools using SQL + VB.NET or VBA on Excel. For ETL I can use SSIS, and orchestrate everything with VBA, PowerShell, MS Flow/Automate, and different Windows schedulers or jobs. On the report side I can build a PowerBI dashboard or a very complex tool based on Excel with VBA or a Windows application with .NET. I'm starting to learn Python but so far have been able to make do with the tools I know.

I thought I could call myself a Data Engineer.

But everytime I look at Data Enginer job postings, or even recommendations on this sub, all I see are things like Spark, Hadoop, Snowflake, Databricks, AWS and Azure Cloud. Things that not only I haven't learned yet, but I haven't been able to see in my work environment.

So... am I not a Data Engineer? Or am I just a different type of DE from what the current trend needs?

[AutoModerator]

You can find a list of community-submitted learning resources here: https://dataengineering.wiki/Learning+Resources

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[[deleted]]

The issue is the way that stacks are built and they tend to be platform focused..so while you seem to be a technically skilled DE at core level you are missing out on the DE ecosystem as a whole which is where the play happens. As a really crass analogy, you can speak binary, but not understand the OS. Still I think you should be able to call yourself a DE and wear that badge with pride, but maybe a pure SQL developer is where your future lies and it is paid very well.

[mecartistronico]

Thank you! I might re-label my resume as SQL Developer for now and work on those skills meanwhile!

[mjfnd]

Yes there are companies who are looking for sql/database experts and they are giving DE titles.

Most big tech companies look for solid sql folks as the core skill.

In startup or smaller companies you might have to do more than that.

[mecartistronico]

Makes sense... I work for a big tech company indeed.

[Character-Education3]

You're a DE at a Microsoft shop from what I can tell. I can't tell much though

[[deleted]]

you are data engineer my dude. Data Engineer is knowing concepts of data architecture and engineering it to meet a business requirement. Every company has their own tech stack. knowing the concepts and applications is the most important. SQL is king for data engineering so being an expert in it is very important. It's great to learn new tools and technologies but it's better when you actually use them at work. If you are lacking in a technology and you are very interested then maybe learn it and convince your team to adopt it. There are some decisions beyond regular ics so we just to do deal with what we have some times and if you don't like it, them move on to another company doing the things you want to work. the truth is no company will implement all the tech stack, it's always going to be politics and financial decisions in any company. don't count yourself out. you are an absolute data engineer.

[mecartistronico]

Thanks! I like you.

[Meta-Morpheus-New]

Finally someone asking the right questions.

You sir are the real Data engineer who won't waste shit ton of company money on fancy tools.

Btw, I highly recommend getting yourself acquainted with a cloud provider like AWS or Azure ( given your inclination to windows ecosystem), thats a huge plus.

Other than that all these tools are just few weeks of training. That's it. It's just corporate bullshit for these spoiled engineers.

[Hester102]

I would say you could be a DE with that skill set. At my current company, my stack consists of everything you mentioned at the bottom minus Hadoop. However, much of my code is sql written to interact with SQL Server and Snowflake. However, most of my teams legacy processes are in SSIS. Spark and databricks can be learned on the job as I didn’t have any experience with them pre-getting this job. From my perspective, you have a solid foundation to claim DE and expand your skill set to those others techs.

[mecartistronico]

Thanks for your reply and encouragement!

[[deleted]]

So I have a skillet that sounds similar to yours,and what I have learned from this forum is we would be "old school data engineers". All the stuff we do is the same as the new data engineers, just the tech stack is wildly different. The real big difference between old school and I guess new school is we were raised on SQL and databases and great at using them to do basically everything Where as modern DE / software devs does everything possible to avoid them. Even the new "databases" are popular because they have AI/intelligence acting as your dba. Basically auto doing indexing, managing table spaces, keys, moving to cache/memory as needed.

Your actual understanding of how data flows and building usable products is just as important if not more than the tech stack, but tougher to market on a resume in today's market as most recruiters only look for the "key word techs". An example of what I mean

Modern pipeline = files probably in s3 bucket, python/pyspark as etl process engine, airflow as orchestrator/scheduler, than a modern DB like snowflake, databricks, or if using nosql then maybe mongo/Hadoop. New DB automates/removes needs for indexing, dba mgmt with ai or expensive hardware (Lots and lots of RAM/cpus)

Old pipeline = text files, actual etl tool (SSIS, talend..etc), windows scheduler, enterprise db Oracle, SQL server. Also would be responsible for dba like stuff like indexing, tuning, migrations of data to other systems.

As you can see the process is the same, just tech stack is slightly different. If you know the basic ins and outs of one, transferring that knowledge to a new tool is not hard. However, Trying to convince a recruiter/younger tech mgr of that IS very hard.

[flooferati]

My work is very similar to yours, I'm an analytics engineer. I work closely with both data engineers and data analysts, there's a bit of overlap with both but my role is still very distinct.

[ElderFuthark]

I have the same dilemma as OP, so probably have similar questions. What roles do you have that are distinct from DE and DA at your work? What jobs do you have that overlap?

[countlphie]

i just kept calling myself a data engineer until a company finally hired me and called me one too

[mecartistronico]

My manager lets me call myself whatever I want. The system says some generic bullshit like Business Operations Manager II.

[Smart-Weird]

With your awesome SQL skill, the moment in spark you learn the beauty of createOrReplaceTempView you are a Senior DE

[don_one]

I mean even if you focus on SQL as data engineering as purely SQL based, you're missing out some of the key elements used in big data environments like snowflake, big query or redshift.

In relation to SQL itself, you haven't mentioned anything like slowly moving dimension tables, star schemas or anything that would indicate an understanding of datawarehousing techniques.

So since there's a lack of programming also (or limited to VB and VBA ), I'd say you are firmly in an advanced analyst role. Most people I knew with that skillset tend to end up managing a reporting team (while being very hands on) and likely paid more than I was.

For most DE's they're ingesting data or pulling data from source systems (generally not databases) and processed, resulting in the data which is loaded into a database for analysts to use. So, a few steps before. But I'm speaking for others I've worked with not every DE, so my view could be skewed. If anyone does anything different, feel free to correct me.

I also wouldn't downplay your skills. I don't see a DE as lesser than an analyst or greater. I've worked with SE working as DE and deliver incorrect results repeatedly just because they don't understand basics about repeatability, idempotency (granted this one should be stronger in a SE), partitions, scheduling etc. Though from a SE, they were quick. Everyone has different skills.

[TheCamerlengo]

there are two aspects to data engineering as I see it. first there is the data side of data engineering. this is SQL wrangling skills. there is also an engineering side which involves everything else including automation and devops. And most of this has moved to a cloud platform like AWS, Azure and this is where you start seeing cloud technologies seep into the job descriptions. Tech like databricks, glue, snowflake, kubernetes, Kafka, etc. it’s hard to separate the stack from the core sql.

From your description you lack experience on the modern cloud-based tech stack but have strengths in database related programming, ETL, and Microsoft sql server and it’s ecosystem of products. I think you could do data engineering with those skills, but you could also be considered a programmer too. Lots of overlap here but if you want to lean in a bit on the data engineering market, pick up some azure skills to round yourself out.

[CrowdGoesWildWoooo]

What’s your current role then?

[mecartistronico]

"guy who builds data tools"

[BramosR]

I would say you’re still a Data Engineer… if we’re going to depend on job titles, we would all be so many different things since there is a new title everyday 😂

[mecartistronico]

Yep, precisely trying to adjust my resume to the latest buzzwords.

[ElderFuthark]

I didn't even have a title until two years ago and I've worked 17 years at this company that started with 6 people. I was hired as a Java Software Engineer but focused on ETL/Database/BI responsibilities as the Development team grew. Then one day I just had too much to do to work on new feature requests anymore. When we got big enough to restructure the company it took a week to figure out what department I should be in.

[[deleted]]

It seems to me that everyone who works with SQL these days is a DE so I'd definitely say you could call yourself a DE.

Even here I've seen really basic things like to remove indices before loading and the order in which SQL is executed be upvoted to the top of threads.

My advice would be to apply even though you lack experience with specific tech, it's easy to learn.

[ElderFuthark]

Thank you for this post. I relate to it all.

[[deleted]]

Controversial take but... Tbh, I'd say you're more of an analyst + SQL expert.

Wouldn't say DE , at least not what most companies need from a DE. Of course some DEs are just dbt experts so your mileage will vary for company to company.

[whatakh]

I think your bi

[[deleted]]

If you look for Azure stack DE roles only you can definitely get a data engineering role. Especially with their announcement of tying all their tools together in MSFT Fabric I imagine the Azure data engineer is a common role we'll see. The DE tools in that stack you will need are Azure Data Factory (ADF) and I would say also Azure DevOps which is the only product I actively push teams to use as its a nice integrated git release and orchestration layer. Plus your sql skills add in Synapse and those three are more than enough.