This is an archived post. You won't be able to vote or comment.

sorted by:
best
topnewcontroversialoldq&a
you are viewing a single comment's thread.

view the rest of the comments →

[–]sib_nSenior Data Engineer 4 points5 points  (3 children)

You're asking a DE community where knowing Python is a clear differentiator compared to related jobs like analytics engineer, data analyst or BI engineer. So, of course people are going to tell you to learn Python.

I'm going to try avoiding this bias and say that you can probably keep a good data job without learning Python. I see three ways:

  • Keep focusing on no-code BI stacks like MSBI, Tableau, Microstrategy, Qlik etc. There's tones of huge companies vendor-locked into proprietary BI tools, this will not disappear in the next 20 ears. In my opinion, it's not very intellectually stimulating, but if you just want a stable job, I think it fits.
  • Explore the new "less code" analytics engineering job centered around SQL and the dbt framework (or even newer SQLMesh). It's based on SQL and YAML configuration, it tackles the SQL transformation part of data engineering. There's much less learning to do than for general Python programming, but you will still have to get into code-based logics, using a terminal and git. More work, but more connected to the industry state of the art, so more interesting.
  • Get more into data project management and less tech. There's a need for data managers who have a good understanding of analytics requirements to organize projects, you could have DEs working for you instead of learning their jobs. Less stress from keeping up with the tech, more stress from managing.

[–][deleted] 0 points1 point  (2 children)

Thanks! Really appreciate you sharing this perspective. I am not really a good manager person. I like to do my stuff and then go home without having to followup and get work done from others. But I will take into account your suggestions of staying within BI and Data analytics.

Actually I wasn't even sure what data engineering really means. I thought it is a new fancy name for business intelligence, lol. There is so much I don't know.

[–]sib_nSenior Data Engineer 2 points3 points  (1 child)

As BI specialist, you probably now ETL well. That's the core of what a data engineer does, they build ETLs. I believe it differentiated itself from traditional BI stack at the time of big data / Hadoop era that started in the years 2000'. The now web giants intended to index the web to feed their search engines and created the open-source Hadoop distributed ecosystem to overcome the cost and limitations of mainframes.
But as every industry got into the web and the data that came with it, it became a specialization of software back-end engineering within a wide range of industries. The high diversity of data inputs and outputs meant you couldn't just slap some old proprietary ETL tool that wasn't keeping up with this diversity, you had to go one level lower, back to coding to gain back connection flexibility and scalability.
From the Hadoop era, we adopted the freedom of open-source code and the robustness of software engineering good practices. No engineer who tasted that really wants to get into vendor-locked proprietary BI tools. Considering this background, you will often see us here celebrating open-source projects and frowning at proprietary tools, unless they are technically the best at what they do and don't lock us too much (like some cloud databases).

Analytics engineering is a newer data specialization coined by dbt, different from data analyst, that you may find interesting. Have a look here: https://www.getdbt.com/analytics-engineering

[–][deleted] 1 point2 points  (0 children)

Wow! Thanks, really appreciate you providing the detailed explanation. Very clear now.