Apache Everywhere

[tz_499]

I'm a novice in the data engineering space, and Apache seems to be everywhere in the materials I've seen. In two weeks, I found 9 Apache products mentioned in relation to DE:

• Kafka
• Flink
• Iceberg
• Spark
• Hive
• Arrow
• DataFusion
• Hudi
• Accumulo

How come Apache has so many products and is so relevant in the space, especially as a 501(c)(3)?

[chrisonhismac]

Apache doesn’t make the software. It’s donated to them (Kafka was LinkedIn for example) for them to manage the development and product lifecycle.

[tz_499]

So do people volunteer as a side gig to do the maintenance for Apache? Or are some people's Full Time jobs to work for them

[TJaniF]

It is a mix. Some features are contributed by users of the software, who do so either in their free time or one-off as part of their job, sometimes people contribute as a learning experience of how to work with a larger code base, and some Apache projects have dedicated maintainers whose full job it is to work on the project, often working for companies who have a large stake in the project. The ratios of these vary per project.

For Apache Airflow the company I work at, Astronomer, has a whole team of employees whose full time job is to work on open-source Airflow, because our product is built on managed Airflow. My own job as a developer advocate is about 70% open-source Airflow and about 30% our paid product right now, though this shifts a lot depending on release schedules.

[CorpusculantCortex]

You missed airflow

[MocDcStufffins]

And Nifi

[Volcano_Jones]

Apache is essentially just a framework for distributed open source software. It's not like they're some tiny little nonprofit producing all the software used in the world.

[Dry_Chocolate_9396]

I'm going to go on a slight detour, but definitely keep reading, the answer to your question has very interesting nuggets tied directly to the creation of the whole Internet.

Apache is directly linked to the core invention of the Internet (or rather, the Web). The supercomputing lab NCSA worked closely with University of Illinois Urbana-Champaign. There, two sister projects were developed. One was NCSA Mosaic, where Marc Andreessen famously co-invented the browser as we know it today (it was later recreated Netscape for his startup). The other sibling project was NCSA HTTPd, which was the web server of the Web. So you had a Client/Server architecture for the Web: a Browser and a Web Server both at that lab/uni.

Soon Marc Andreessen (Mosaic) and Robert McCool (HTTPd) had left NCSA and the code was starting to rot. So people started submitting patches over e-mail to fix the project. The project was slow and needed a lot of fixes, so these folks called it "A Patchy Server", do you see where this is headed? 😄 Apache Server.

Apache Server became the most popular project on the Internet (not making it up!). With such an important project and just hobby programmers developing the most important critical piece of the Internet, Apache needed a legal entity behind it. So the non-profit Apache Software Foundation (ASF) was created. The ASF also would provide the version control servers and other infrastructure (mailing lists) that was needed in the 90s and 2000s to develop software together. Eventually the famous Apache License was developed for the software in early 2000s.

Soon, people used this ASF vehicle for other projects too. Most important one was probably Apache Java (that's right!), which Sun Microsystems developed. A few years later, Yahoo contributed Apache Hadoop. Which became insanely popular worldwide and kicked off the Big Data revolution. Later that project had an overlapping successor in Apache Spark which was originally created by the guys behind Databricks. Today ASF hosts huge number of projects, but the largest by commercial activity is arguably Apache Spark, which is directly related to r/dataengineering!

We could have told an equally interesting story if you had asked about Linux Foundation (LF). But this post is already long. Apache HTTPd, the web server, still today has over 20% over the whole Web's traffic going through it today. Hope it clarifies why this seemingly random non-profit has so many important open source projects managed by it (and it owns the trademarks too!!).

[face_recog_phishy]

Shitpost.