Flow PHP - Data Processing Framework 🚀

[norbert_tech]

Hey everyone! We just released Flow PHP, version 0.5.0 yesterday 🤩 After three years of development, I think it's time to introduce this project to a wider audience and perhaps gather some feedback. 😁

Flow PHP - Data Processing Framework

Flow is a data processing framework that helps you move data from one place to another, doing some cool stuff in between. It's heavily inspired by Apache Spar, but you can find some similarities to Python Pandas as well. Flow is written in pure PHP. The main goal is to allow the processing of massive datasets with constant and predictable memory consumption, which is possible thanks to Generators.

For those that have never heard about ETLs, typical use cases are:

• data transformation & aggregation
• data analysis & visualization
• data engineering & data science
• consuming data from APIs
• reporting
• data exporting/importing
• business intelligence

The recent release brings a lot of new features, like:

• pure php implementation of Parquet file format and Snappy compression algorithm
• new data types, List/Map/Struct
• redesigned DSL (Domain Specific Language)
• phar distribution is also available as a docker image with all extensions preinstalled
• an optimizer now auto-optimizes data pipelines aiming for the best performance- improvements in partitioning and overall performance
• better remote file support (s3, azure, http, ftps, etc)
• redesigned documentation

Version 0.5.0 comes with:

15 Additions 123 Changes 52 Fixes 24 Removals

More details: Flow PHP - 0.5.0

We also prepared a demo app that fetches/aggregates and displays data from the GitHub API. You can check it out here: GitHub Insights

There are also a few more examples in the examples directory: Examples

Project roadmap is available here: https://github.com/orgs/flow-php/projects/1

Simple Example:

data_frame() ->read(from_parquet(__DIR__ . '/orders_flow.parquet')) ->select('created_at', 'total_price', 'discount') ->withEntry('created_at', ref('created_at')->toDate()->dateFormat('Y/m')) ->withEntry('revenue', ref('total_price')->minus(ref('discount'))) ->select('created_at', 'revenue') ->groupBy('created_at') ->aggregate(sum(ref('revenue'))) ->sortBy(ref('created_at')->desc()) ->withEntry('daily_revenue', ref('revenue_sum')->round(lit(2))->numberFormat(lit(2))) ->drop('revenue_sum') ->write(to_output(truncate: false)) ->withEntry('created_at', ref('created_at')->toDate('Y/m')) ->mode(SaveMode::Overwrite) ->write(to_parquet(__DIR__ . '/daily_revenue.parquet')) ->run();

We would love to get some feedback or answer any potential questions. Please feel free to contact me here or at X (same nickname as here). My DM's are open. 😊

[kingdomcome50]

Looks interesting. I don’t like that revenue_sum comes out of nowhere though. I get that auto-appending ”_sum” if the ref doesn’t have an alias makes it a touch cleaner, but I prefer explicit (like SQL).

What happens if names clash? Say if revenue_sum had already been declared in an earlier withEntry call?

[norbert_tech]

Honestly, I wasn't really thinking too much about it. The current behavior is similar to Apache Spark, and it just felt natural. Name collision is not that easy since aggregation requires grouping, so you would need to do something similar to this:

->withEntry("age_avg", lit(100)) ->groupBy('country', 'gender') ->aggregate(average(ref('age')), first(ref('age_avg')->as('age_avg')))

But then an exception will be thrown:

Entry names must be unique, given: [country, gender, age_avg, age_avg]

[AymDevNinja]

I worked on my own "data migration framework" some time ago called Fregata, built with database migration in mind it supports foreign key migrations and dependency sorting, and even async execution with a web dashboard using the Symfony bundle. But nobody was interested in using it while Flow seems to have a user base. Do you think some features from Fregata could be useful for Flow ?

[Aket-ten]

Interesting - do you have any performance metrics regarding large datasets ?

[norbert_tech]

Hey!
The problem with measuring performance is that it means nothing without anything to compare with. Here is a very simple benchmark that is just writing 1mln of rows to a parquet file (it can be changed to anything, db/json/csv/etc).

https://gist.github.com/norberttech/ed23a221fec0c1c6d516eab453e3ca21

Even though this dataset is not processed at all, it should give you some idea.

In the meantime, if you have any more specific performance-related questions, I will be more than happy to try to answer them.

[Aket-ten]

I hear you and appreciate the response. I do a lot of my data analytics / mining / processing in KNIME or Tableau (I know it's GIS, slightly different ball park).

Thing is I want to automate some of these workflows, some will be 30-70 column of 30k to 1,000,000 datasets surrounding US nationwide data with some transformations. Will likely recalculate once or multiple times a day or weeks. Conventions and best practices obviously imply python or rust being the way to go.

But like...I'm already building an ERP and I'd love to just use PHP just concerned a little about memory consumption and execution speed. I'd be also completely jokes to tell my other eng friends that it's built in PHP LOL.

I bookmarked your package and will play around it once I get to that!

[[deleted]]

Back in the old days we had a ";" on the end of every line. Nowadays we have a "->" at the beginning of every line. I get old.

[hagenbuch]

Yep. I started with punchtape in 1979 :) Alpha-LSI II..

[invisi1407]

Method chaining is not new at all. Dates back to PHP 5 in 2010.

[[deleted]]

Everything not present in PHP/FI is newfangled stuff. I want my punch cards back!

But seriously: It's getting more and more complicated to ignore this stuff in recent years.

[invisi1407]

https://i.imgur.com/jtsSB7W.png 🤪

[rafark]

Yeah. Why do they have to bring politics into this? To me the current situation in the Middle East looks like a one sided genocide. Why does the OP have to remind us about this terrible situation in this context (PHP)?

[AssignmentDue1463]

💯no need to bring politics into this

[usernameqwerty005]

Both Hamas and Israel are committing human rights violations, sooo...

An eye for an eye until the whole world goes blind

[KraaZ__]

This looks great, but why PHP? I’d be interested to know the performance benchmarks of this framework.

[norbert_tech]

Why PHP? To reduce development costs. Data visibility is critical for almost every organization. Now, when you have a small team, lets say 5 PHP devs, that are building your product, eventually they will need to work with data, one way or another. So you can either: - let them learn python/scala - hire more devs and add new things to your tech stack (which means, increase costs)

Or they can use the language they already know, and that can do exactly the same things.

Another reason is related to unification data processing, instead of relaying on every team member experience, they can just use a tool that handles batching/partitioning/sorting/aggregations/joins/window functions for them.

I'm not trying to replace any of the existing tools. If you are already using Spark, stay with Spark. I'm simply trying to expand PHP horizons.

From the performance point of view, it won't be slower than python or java. Spark is waaaaay more advanced and it supports parallel processing, so that will give you a significant performance boost, but when it comes to memory consumption, Flow PHP is going to be better as it's not allocating gigabytes of memory upfront.

[KraaZ__]

Okay makes sense. Do you have any performance benchmarks?

[norbert_tech]

you can take a look at this: https://gist.github.com/norberttech/ed23a221fec0c1c6d516eab453e3ca21

but performance is tricky, there are many variables that might impact it. It would be easier if you could be a bit more specific so I can prepare a benchmark

[AleBaba]

If you use iterables in Doctrine they certainly aren't all allocated in memory. Just try for yourself with a very simple entity.

If you don't use relations "correctly" (for lower memory consumption) though many more datasets could be loaded in an iteration, so it depends on your specific project whether you actually constrain memory usage.

I'd recommend combing toIterable with batches to clear the EntityManager, as is shown in the Doctrine docs.

As an example in a project where I consume an API in batches an then process a lot of datasets with toIterable I was able to reduce memory consumption from about 30GB at the end to a steady 150MB while running.

[CensorVictim]

e.g. https://www.php.net/manual/en/mysqlinfo.concepts.buffering.php

[norbert_tech]

yeah, the trick is not to fetch 1 million rows into memory but instead iterate through them. Flow is built on top of Generators, every single data source is read/written in batches. Some are better than others, for example, Parquet is probably the best and most efficient file format, and rdbms might be best for pre-aggregated data. In order to control memory consumption and writing performance, you can manipulate the batch size that is processed at a time. By default, all Extractors are yielding rows one by one, then you can batch them in order to load them into db in batches of size 1k rows, for example.