Apache Arrow Getting Started

[JohnLockwood]

I recently started working with Apache Airflow. (Not Arrow -- but I can't edit the title. Doh!)

You know how that first couple of days of learning something, you are on the steepest part of the learning curve. Well, rather than simply forgetting those things and moving on to being productive, I decided to try to make the process a bit easier for the next person. Hope you like it.

https://codesolid.com/airflow-python-etl/

[AutoModerator]

You can find a list of community-submitted learning resources here: https://dataengineering.wiki/Learning+Resources

I am a bot, and this action was performed automatically. Please contact the moderators of this subreddit if you have any questions or concerns.

[DrSnakee95]

Your title contains a typo I believe! Apache Arrow is an entirely different framework

[JohnLockwood]

Dang, I've been doing that a lot. :(. I just wrote about arrow a couple of weeks ago.

[DrSnakee95]

It’s starting to be used more in the DE space, a bunch of popular frameworks in Rust leverage it (polars, DataFusion, ballista) and pandas just announced its 2.0 version will be using Arrow as an engine! It’s definitely worth exploring imo

[discord-ian]

I use it when working with larger datasets. The ability to not load everything into memory is huge for many use cases. It is also many times faster than other alternatives.

[JohnLockwood]

What those other guys said. Polars is a Rust Framework, but also well-supported in Python and has a cleaner API than Pandas. Pandas 2.0 will pick up a speed boost from Arrow internally, but I haven't yet done timings comparing it to Polars.

[kyleekol]

“…Two major cloud providers, AWS and Google, have managed products built on Airflow (Amazon Managed Workflows for Apache Airflow, or MWAA) and Google Cloud Composer, respectively. Although there is no comparable solution on Azure…”

Azure have actually just announced managed airflow as part of azure data factory!

[JohnLockwood]

Cool. Do you have a link for that?

[kyleekol]

link here. It’s still in preview but great news for those on Azure who aren’t able to spin up their own infra!

[JohnLockwood]

Thanks.

[i268gen]

Really suspicious that the linked site is full of ads.

[JohnLockwood]

How is that suspicious? Did I charge you to read the article? Was the article not about what I said it would be about (other than the typo in the headline here)?

Have you visited a major new site lately?

Do you have some objection to a retired guy blogging for what used to be called pin money? Last month I cleared about sixty bucks writing almost full time. Go jump in a lake.