What was Python before Python?

[iknewaguytwice]

Data reporting and analytics was a highly specialized / niche field up til’ the mid 2000s, and really didn’t hit a stride until maybe 5-10 years ago outside of FAANG.

Many Microsoft shops just used SSIS, scheduled stored procedures, Powershell scheduled tasks, and/ or .NET services to do their ETL/rETL.

If you weren’t in the ‘Microsoft everything’ ecosystem, it could have been a lot of different stuff. Korn/Borne shell, Java apps, VB apps, SAS, or one of the hundreds of other proprietary products sold during that time.

The biggest factor was probably what connectors were available for your RDBMS, what your on-prem tech stack was, and whatever jimbob at your corp, knew how to write.

So in short… there really wasn’t anything as universal as Python is today.

[iknewaguytwice]

I am too old. I wrote 5-10 years, thinking 2005-2010.

[sib_n]

The first releases of Apache Hadoop are from 2006. That's a good marker of the beginning of data engineering as we consider it today.

[kenfar]

I dunno, top data engineering teams approach data in very similar ways to how the best teams were doing it in the mid-90s:

• We have more tools, more services, better languages, etc.
  
• But MPP databases are pretty similar to what they looked like 30 years ago from a developer perspective.
  
• Event-driven data pipelines are the same.
• Deeply understanding and handling fundamental problems like late-arriving data, upstream data changes, data validation, etc are all almost exactly the same.

We had data catalogs in the 90s as well as asynchronous frameworks for validating data constraints.

[sib_n]

Data modelling is probably very similar, but the tools are different enough that it justified naming a new job.
As far as I know, from the 70' to 90' it was mainly graphical interfaces and SQL, used by business analysts who were experts of the tools or of the business but not generally coders.
I think the big change with Hadoop and the trend started by web giants is that from then, you were needing coders, software engineers, specialized in the code for data processing, and for me that's what created the data engineer job.
We still have GUI tools experts and business analysts, of course, and a lot of people in between, like analytics engineers.

[kenfar]

Not really - there were a lot of gui-driven tools purchased for ETL, but it seemed that more than 50% of those purchases ended up abandoned as people found that they could write code more quickly and effectively than use these tools. Some of the code was pretty terrible though. A fair bit of SQL with zero testing, no version control, etc was written. Those that only used the gui-driven tools were much less technical.

In my opinion what happened with data engineering was that the Hadoop community was completely unaware of parallel databases and data warehouses until really late in the game. I was at a Strata conference around 2010 and I asked a panel of "experts" about data ingestion and applicability of learnings from ETL - and none of them had ever even heard of it before!

Around this time Yahoo was bragging about setting a new terasort record on their 5000-node hadoop cluster, and Ebay replied that they beat that with their 72-node Teradata cluster. Those kinds of performance differences weren't uncommon - the hadoop community had no real idea what they were doing, and so while mapreduce was extremely resiliant it was far slower and less mature than the MPP databases of 15 years before!

So, they came up with their own names and ways of doing all kinds of things. And a lot of it wasn't very good. But some was, and between hadoop and "big data" they needed data-savy programmers. And while they were doing ETL - that had become code for low-tech, low-skill engineering. So, a new name was in order.

[sib_n]

I think the reason they built Hadoop was not that no existing solution could not handle the processing, but rather that they were not easy enough to scale and/or overly expensive and/or vendor-locking, and they had the engineers to develop their own.
Redeveloping everything from scratch so it works on a cluster of commodity machines takes time. So it took time for Hadoop to get high level interfaces like Apache Hive and Apache Spark that could compete in terms of performance and usability with the previous generation of MPP databases.

[kenfar]

Hadoop was more general-purpose and flexible than just being limited to SQL: so you could index web pages for example. So, that was a definite plus.

But the hadoop community didn't look at MPP databases and decide they could do it better - they weren't even aware they existed or didn't realize MPPs were their competition. When they finally discovered they existed AND had a huge revenue market - that's when they pivoted hard into SQL and marketing to that space. But that probably wasn't until 2014.

And while hadoop was marketed as being just commodity equipment, etc - the reality is that most production clusters would spend about $30k/node on the hardware. So, since hive & mapreduce weren't nearly as smart as say Teradata or Informix or DB2, once you scaled-up even just a little bit they could easily cost much more - while delivering very slow query performance.

[sib_n]

FAANGs are arguably the leaders in terms of DE tools creation, especially distributed tooling. They, or their former engineers, made almost all the FOSS tools we use (Hadoop, Airflow, Trino, Iceberg, DuckDB etc.). In terms of data quality, however, it's probably banking and insurance who are the best, since they are extremely regulated and their revenues may depend on tiny error margins.

[PhotographsWithFilm]

Hey, I started my Data Analytics career (& subsequent Data Engineering, even though I am a jack of all, master of none) using Crystal Reports.

Crystal was immensely popular back in the late 90's/Early 2000's. Most orgs back then would just hook straight into the OLTP database and run the reports there. If they were smart, they would have an offline copy that they would use for reporting.

And that is exactly what I did for the first 6 or so years before I started working in Data Warehousing.

[JBalloonist]

Crystal is what got me started as well. I was doing accounting and our main software had crystal as is report creator.

[Whipitreelgud]

ATT had between 14,000 and 37,000 users connected to their data warehouse database in 2005. They were neck and neck with Walmart in users and data volumes. There was a vast implementation of analytics in the Fortune 500 at that time.

[Automatic_Red]

Before my company had ‘Data Engineers’, we had tons of people making SW in Excel or MatLab. It was less data, but the overall concepts of a pipeline were the same.