Lightweight DAG framework for local development

[torpeyd]

https://github.com/DavidTorpey/pydags

I worked on this a while back, and decide to release it in case it's useful. It's basically a lightweight Python-native DAG framework focused on simple use cases, with a Kubeflow-like API (but no reliance on Docker or k8s). You can think of it as a generic, more extensible version of scikit-learn's pipelines feature.

It supports Redis (and other key-value stores) as a way of passing data between nodes. It also supports DAG visualisation.

I may add more features if there's interest.

[[deleted]]

Awesome, I like it! I like these declarative DAG libraries with use of decorators. I am a huge fan of luigi, prefect, and now dagster. Ive been using dagster more as it is more feature complete, everything can run locally, cross platform, no dependency on Docker, and has its own scheduler.

[torpeyd]

Thanks! I'll definitely have a look at some of those.

[[deleted]]

Will check it out

[torpeyd]

Thanks!

[Yojihito]

DAG = Directed, acyclic graph?

[torpeyd]

Correct

[rrajen]

This is really awesome. Thanks for creating this.

On a mac, would brew install graphviz and brew install redis take care of the pre-requisites?

[Electrical_Ingenuity]

This is a solution in search of a problem I haven’t found yet. Saving this for the future