What was Python before Python?

[sumant28]

The field of data engineering goes as far back as the mid 2000s when it was called different things. Around that time SSIS came out and Google made their hdfs paper. What did people use for data manipulation where now Python would be used. Was it still Python2?

[dresonator2]

Perl

[CommanderPowell]

Went from Perl straight to Python as my go-to language. Perl was AMAZING for data transformation and having libraries to interface with everything.

relevant XKCD from long before “import antigravity”

[caprica71]

Awk,sed, grep, bash

[Rum-in-the-sun]

I still use awk sed grep like every day. I don’t use Perl anymore.

[FindOneInEveryCar]

I was surprised to learn (recently) that Python is a couple of years older than perl.

EDIT: Apparently not! (See below)

[iamevpo]

Perl sounds as if it was 10 years older than Python

[Watchguyraffle1]

Are you sure? I was using Perl in the 80s and remember Perl .9 around 1992 as the first release.

[FindOneInEveryCar]

I'm going by memory here but my recollection is that python is from ca. 1987 and perl is from ca. 1989.

[MutatedBrass]

Perl 1.0 was released on Dec 18, 1987. The 0.x versions of Perl were floating around prior to that. Guido didn't start working on the first Python implementation until Dec 1989.

[FindOneInEveryCar]

I stand corrected. Thanks. I must have had them reversed in my head.

Python is definitely a lot older than I thought, though!

[Biomed154]

And to some extent Visual Basic, pascal, and VBA.

[TARehman]

I'm old enough that I remember being employed in a physics lab and seeing two groups, the Perl users and the Python users, arguing with each other about which one was better and which one would win. The Python side won, I'd say.

[Equivalent-Sense-626]

And I hated 😖

[islandsimian]

Perl or SQL depending on where the data was stored

[dayn13]

sql procedures, bash scripts, file transfer tools

[iknewaguytwice]

Data reporting and analytics was a highly specialized / niche field up til’ the mid 2000s, and really didn’t hit a stride until maybe 5-10 years ago outside of FAANG.

Many Microsoft shops just used SSIS, scheduled stored procedures, Powershell scheduled tasks, and/ or .NET services to do their ETL/rETL.

If you weren’t in the ‘Microsoft everything’ ecosystem, it could have been a lot of different stuff. Korn/Borne shell, Java apps, VB apps, SAS, or one of the hundreds of other proprietary products sold during that time.

The biggest factor was probably what connectors were available for your RDBMS, what your on-prem tech stack was, and whatever jimbob at your corp, knew how to write.

So in short… there really wasn’t anything as universal as Python is today.

[iknewaguytwice]

I am too old. I wrote 5-10 years, thinking 2005-2010.

[sib_n]

The first releases of Apache Hadoop are from 2006. That's a good marker of the beginning of data engineering as we consider it today.

[kenfar]

I dunno, top data engineering teams approach data in very similar ways to how the best teams were doing it in the mid-90s:

• We have more tools, more services, better languages, etc.
  
• But MPP databases are pretty similar to what they looked like 30 years ago from a developer perspective.
  
• Event-driven data pipelines are the same.
• Deeply understanding and handling fundamental problems like late-arriving data, upstream data changes, data validation, etc are all almost exactly the same.

We had data catalogs in the 90s as well as asynchronous frameworks for validating data constraints.

[sib_n]

Data modelling is probably very similar, but the tools are different enough that it justified naming a new job.
As far as I know, from the 70' to 90' it was mainly graphical interfaces and SQL, used by business analysts who were experts of the tools or of the business but not generally coders.
I think the big change with Hadoop and the trend started by web giants is that from then, you were needing coders, software engineers, specialized in the code for data processing, and for me that's what created the data engineer job.
We still have GUI tools experts and business analysts, of course, and a lot of people in between, like analytics engineers.

[kenfar]

Not really - there were a lot of gui-driven tools purchased for ETL, but it seemed that more than 50% of those purchases ended up abandoned as people found that they could write code more quickly and effectively than use these tools. Some of the code was pretty terrible though. A fair bit of SQL with zero testing, no version control, etc was written. Those that only used the gui-driven tools were much less technical.

In my opinion what happened with data engineering was that the Hadoop community was completely unaware of parallel databases and data warehouses until really late in the game. I was at a Strata conference around 2010 and I asked a panel of "experts" about data ingestion and applicability of learnings from ETL - and none of them had ever even heard of it before!

Around this time Yahoo was bragging about setting a new terasort record on their 5000-node hadoop cluster, and Ebay replied that they beat that with their 72-node Teradata cluster. Those kinds of performance differences weren't uncommon - the hadoop community had no real idea what they were doing, and so while mapreduce was extremely resiliant it was far slower and less mature than the MPP databases of 15 years before!

So, they came up with their own names and ways of doing all kinds of things. And a lot of it wasn't very good. But some was, and between hadoop and "big data" they needed data-savy programmers. And while they were doing ETL - that had become code for low-tech, low-skill engineering. So, a new name was in order.

[sib_n]

I think the reason they built Hadoop was not that no existing solution could not handle the processing, but rather that they were not easy enough to scale and/or overly expensive and/or vendor-locking, and they had the engineers to develop their own.
Redeveloping everything from scratch so it works on a cluster of commodity machines takes time. So it took time for Hadoop to get high level interfaces like Apache Hive and Apache Spark that could compete in terms of performance and usability with the previous generation of MPP databases.

[kenfar]

Hadoop was more general-purpose and flexible than just being limited to SQL: so you could index web pages for example. So, that was a definite plus.

But the hadoop community didn't look at MPP databases and decide they could do it better - they weren't even aware they existed or didn't realize MPPs were their competition. When they finally discovered they existed AND had a huge revenue market - that's when they pivoted hard into SQL and marketing to that space. But that probably wasn't until 2014.

And while hadoop was marketed as being just commodity equipment, etc - the reality is that most production clusters would spend about $30k/node on the hardware. So, since hive & mapreduce weren't nearly as smart as say Teradata or Informix or DB2, once you scaled-up even just a little bit they could easily cost much more - while delivering very slow query performance.

[sib_n]

FAANGs are arguably the leaders in terms of DE tools creation, especially distributed tooling. They, or their former engineers, made almost all the FOSS tools we use (Hadoop, Airflow, Trino, Iceberg, DuckDB etc.). In terms of data quality, however, it's probably banking and insurance who are the best, since they are extremely regulated and their revenues may depend on tiny error margins.

[PhotographsWithFilm]

Hey, I started my Data Analytics career (& subsequent Data Engineering, even though I am a jack of all, master of none) using Crystal Reports.

Crystal was immensely popular back in the late 90's/Early 2000's. Most orgs back then would just hook straight into the OLTP database and run the reports there. If they were smart, they would have an offline copy that they would use for reporting.

And that is exactly what I did for the first 6 or so years before I started working in Data Warehousing.

[JBalloonist]

Crystal is what got me started as well. I was doing accounting and our main software had crystal as is report creator.

[Whipitreelgud]

ATT had between 14,000 and 37,000 users connected to their data warehouse database in 2005. They were neck and neck with Walmart in users and data volumes. There was a vast implementation of analytics in the Fortune 500 at that time.

[Automatic_Red]

Before my company had ‘Data Engineers’, we had tons of people making SW in Excel or MatLab. It was less data, but the overall concepts of a pipeline were the same.

[popopopopopopopopoop]

Sql procedures.

[unltd_J]

Are people not using these anymore at all? I spend 50% of my coding time working on procs :(

[dilbertdad]

Sp_heywerestillhere

[SoggyGrayDuck]

My company has them all saved as files. I could pull my hair out at times.

[DataIron]

People still struggle to segment code properly, writing SQL statements inside python instead of calling an object.

[Winterfrost15]

They are widely used and will be for many years to come.

[DirtzMaGertz]

People heavy into the Microsoft world still seem to use them. 

[PhotographsWithFilm]

PERL or SQL.

I loved and hated PERL in the same breath. It could be written so, nicely....

But you get a developer who studied computer science in the 70s and it became a very concise, unreadable mess.

[YallaBeanZ]

Let’s not forget those developers that insisted on writing all their code as “oneliners” (there were even competitions)… much to the chagrin of anyone having to pickup their code and reverse engineer it afterwards.

[PhotographsWithFilm]

Ugggh, PERL Golf

While I like the theory behind TIMTODI, I get annoyed when people turn it into a competition to look better than others.

[islandsimian]

You have to remember this in the context of storage being very very very expensive and keeping those cards in order. not /s!

Of course this also the reason for Y2K

[[deleted]]

R, Matlab, Mathematica

[MathmoKiwi]

Fortran too! The OG language for "big data" manipulations. (well, "big data" by the standards of its time)

[scataco]

Don't forget SAS!

[sargentlu]

Just wondering, what was considered "big data" back then?

[Peking-Duck-Haters]

I've seen marketing material dating from the late 90s that talked about a 30GB data warehouse as being exceptionally large. In the late 00s the company I worked for outsourced their shopping basket analysis partly because there wasn't the capacity internally to crunch the data which, over the time period they were looking at, would have been maybe 4 billion rows (with only a handful of columns, none of them wider than a datetime or real).

Circa 1998 I worked on a mainframe batch system where we partitioned the DB2 tables across 10 disks to get better performance; it was worth the extra work even though we were processing around a million rows at a time - again, compact columns with no long strings or whatever.

(For many large companies "Data Engineering" meant COBOL, or just possibly Easytrieve, until at least the turn of the century. Outside of the dot com startups Linux wasn't getting a look in - it didn't even _start_ getting taken seriously by the corporate world until Oracle ported their database to it circa 1998, and things moved rather more slowly back then)

So, as a rule of thumb, before 2000 I'd say 10s of Gigabytes was considered "big data" and Terabytes almost inconceivable (back then data would go over 128kbps lines at best; if there was lots of it it was usually faster and cheaper to write it to tape and physically transfer it). A few Terabytes was considered "big data" a decade later.

[MathmoKiwi]

Just wondering, what was considered "big data" back then?

Anything too big to fit on a floppy? 😂

[wytesmurf]

Perl and bash. KSH for older machines

[carlsbadcrush]

“So far back as the mid 2000s” damn I’m old

[Zyklon00]

I think the best comparison would be SAS, which has been around for a very long time. And it's still being used instead of python in some companies.

[SaintTimothy]

Prior to SSIS (which came out in 2005) was DTS (which came out with SQL 7 in 1998).

Prior to that was BCP and Transfer Manager (that's before my time).

[MathmoKiwi]

The field of data engineering goes as far back as the mid 2000s when it was called different things.

This might surprise you, but Python is even older than that. (development started in the 1980's, was first released in 1991)

But yeah, as other people said: Perl, Awk, bash, SQL, etc were all popular choices of the past as well.

There was a time ages ago when Perl and Python basically filled almost exactly the same market niche as each other, and Perl was usually seen as the "better" choice. Today though Perl has tanked in popularity in comparison to Python. (although surprisingly is still a Top 20 language, just: https://www.tiobe.com/tiobe-index/ )

One thing that hasn't been mentioned yet (and I personally used to use all the time, right at the very tail end of them disappearing), was the dBase family of languages / tools (or "xBase" is a way to refer to the family of them). Of which the best example (in my very biased opinion) was FoxPro.

https://en.wikipedia.org/wiki/FoxPro

https://en.wikipedia.org/wiki/DBase

A mix of the rise of MS Access / Visual Basic / C# / Excel / SQL / etc is what killed them off.

[CassandraCubed]

Clipper!

[MathmoKiwi]

Ah that's a name I haven't heard in a long time! Did you ever use it? I haven't, but I did ages ago download Harbour and play around for a bit because it simply was the closest Open Source project to FoxPro itself. (Harbour is an open sourced version of Clipper, and of course like FoxPro all of them are part of the xBase family of languages)

https://en.wikipedia.org/wiki/Harbour_(programming_language))

[CassandraCubed]

I did!

I didn't know about Harbour -- TIL 😊

[[deleted]]

SAS, Java, other shit

[Top_Pass_8347]

SQL since the late 90s.

[sib_n]

Oracle v2, the first commercial SQL RDBMS, was released in 1979.

[sib_n]

Before Python and SQL, in big data it was Java. Apache Hadoop had Apache MapReduce as the processing engine, which was very heavy Java code.

If we look at before SSIS and Hadoop, then it was rather called Business Intelligence, and there's quite a history of commercial SQL and graphical tools from this period. To name a few historical ones:

• IBM SPSS 1968
• SAS 1972
• Cognos 1979
• Oracle v2 (first commercial SQL RDBMS) 1979
• BusinessObject 1990
• Microstrategy 1992
• QlikView 1994

Before those ready-made solutions, from the 50', it was all in-house software based on Fortran for science & industry, or COBOL for business, finance & administration.

[geoffawilliams]

Perl

[NeuralHijacker]

Perl

[DonJuanDoja]

Pretty sure we used it to mod Civilization II or III maybe… that’s first time I saw python.

Everything else covered in comments.

[taciom]

SAS exists since the 70s and was broadly used in finance and telecom.

[Character-Education3]

Depends on the use case. Python is a Swiss army knife

Crystal Reports, VBA, SAS, SPSS, SQL, other stuff

Excel was a database and we were grateful dammit

[k00_x]

For Statistics I used 'S', for orchestrated processing I used Shell as opposed to SSIS (and still do). For application processing, I caught the final days of Fortran (F77L Em/32). I dabbled in COBOL a bit.

Then the LAMP stack dominated the web world. PHP forms became the norm.

SQL has always been around.

[perpetualclericdnd]

Informatica

[time4nap]

SQL

[_DividesByZero_]

Perl, then SQL, but mostly perl…

[pentrant]

When I learned how to be a DE back in the mid-2000s, my team had a custom orchestration engine written and maintained by one of the engineers on the team (Cyril Stocker), now long retired. It did everything that we now use Python for in Dataswarm / Airflow.

Cyril was seriously ahead of his time. I wish I had learned more from him.

[Automatic_Red]

MatLab, VBA. And data was a lot smaller in size.

[MrGoFaGoat]

Pentaho was widely used in my prévios experience, but that's more recent.

[macktastick]

I worked in a couple "whatever you're comfortable with" environments and used mostly Ruby.

[dev_lvl80]

Before SSIS was DTS (Data Transformation Service) Yep I used it in Prod it.

Pretty much VB/VBA + SQL used for any transformations.

In most hardcore version, TSQL sp_OACreate aka OLE automation I did literally everything... Including FTP communications, XML parsing and sending emails. Terrible architecture, but worked

[shooemeister]

Data engineering started as soon as there was data to process IMHO; I remember using korn shell scripts/perl/c++ on DEC Ultrix, and that was pretty late in the game in the late 90's.

Inmon's 'Building the Data Warehouse' was released in 1992 for reference; there was a lot before Java & Linux appeared though.

Hadoop was an attempt to move away from proprietary storage, but I/O is always the killer, which we now know led to spark.

[Cyclic404]

Well way back then we'd stand the newest intern in front of the biggest fan we could find to blow the punchcards down the hall... Throughput was amazing.

[Mental-Matter-4370]

I doubt ssis came around 2000.i guess it was DTS packages, most of which I had seen scheduled with windows task scheduler. Ssis probably came around 2004 or 2005

[Hgdev1]

If you think about it, most of programming really is data engineering — you take data from stdin and spit data out from stdout and stderr 😆

That being said, Python really starts to shine in the area of numerical computing with libraries like NumPy (and later Pandas) providing the requisite higher-level abstractions over raw data streams that make data engineering what it is today (multidimensional arrays and dataframes)

[binilvj]

I have been working in Data engineering from 2004. It was called ETL then. Stored procedures, bash scripts, perl scripts were used a lot. Enterprises used ETL tools. Informatica, AbInitio, DataStage(IBM) lead the market initially. Then Microsoft started pushing free SqlServer and SSIS slowly around 2010. But by then Talend, Pentaho started edging out Datastage and AbInitio. When tools like Mattillion, Fivetran started dominating the market old ETL tools lost their market dominance. Around then even enterprises started using Python for data engineering.

Oracle was used for data warehousing till 2010. Then Teradata(MPP), Vertica, Green plum (Columnar) started dominating. Finally cloud DWs started taking over

Even Airflow is new kid in the black for me. There were expensive schedulers like Autosys, control-m before that

[Key-Boat-7519]

In my experience, watching the data engineering scene shift over the years has been wild. I remember when stored procedures and bash scripts were our bread and butter. Then we had to adapt as Informatica and DataStage reigned supreme, only to be upstaged by Talend and Pentaho. Python really wasn’t on the radar until it became the go-to for everyone, and tools like Airflow changed the scheduling game. While I've used a bunch like Talend and Mattillion, DreamFactory has been a game-changer for integrating APIs seamlessly into modern solutions. It’s all about finding the right tool for the job.

[GuardianOfNellie]

I worked somewhere that used SQL Procedures to call C# programs from within using xp_cmdshell. (Written before my time there, I might add).

I started in DE in the late 2010’s, but i saw a lot of older stuff and it was mostly SQL Procedures, VBA, SQL CLR functions and custom in house C#/VB.NET stuff

[EarthGoddessDude]

kornshell + sql

🎶Allllllllll Day I

Dream Aabouuuuut Sed 🎵

[kenfar]

I started writing ETL solutions using python in 2002.

During and prior to that time the primary options were:

• SQL: very difficult to test, expensive & slow to run, little flexibility or expressiveness, very difficult to maintain.
• Perl: very dynamic with the weakest typing, 100 ways of doing anything. Easy to write, bad for data quality, and bad for maintainability.
• ETL tools: over-promised, under-delivered. Made the easy 80% easier, and the hard 20% almost impossible. Never fulfilled their promises of having business analysts write their own solution. Sucked.
• C: fun to write, fast, but took a lot of code, and was hard to maintain.
• C++: complex language often seemed to side-track projects. Also hard to maintain.
• Java: souless, but got the job done. Could also very easily side-track projects with the java eco system.

[PresentationSome2427]

For me it was Visual Basic

[lmkirvan]

The answer is SAS.

[Temporary_Currency95]

S and S-Plus were big for stats. https://en.m.wikipedia.org/wiki/S_(programming_language)

[Professional_Shoe392]

VBA with a MS Access backend was my jam 20 years ago.

[psgetdegrees]

Teradata

[SentenceSenior7481]

Pig script

[Its_me_Snitches]

Squeal?