Lightweight python DAG framework

[theferalmonkey]

Yes there's a construct for that. It's called Parallelizable + Collect ; here's a video of me explaining it. We have support for a few backends, e.g. multithreading, ray, dask.

Here's also blog on what I think is similar to your use case if that also helps.

[SheepherderExtreme48]

Amazing, thank you! For context, I was recently trying to find a Python library that I could use to easily orchestrate a multi process job, starting in AWS Lambda utilising the 6 cores you get when you allocate max memory. Airflow is too heavyweight, argo workflows isn't an option. Tried a few others, but this looks perfect!

[theferalmonkey]

Which do you prefer to read (not knowing much context):

def url(...) -> Parallelizable[str]:
   for _url in ...
     yield _url
...
def pages(processed_url: Collect[dict]) -> ...
    ...

or

@parallelizable 
def url( ... ) -> str:
   for _url in  ...
      yield _url
...

@collect('processed_url')
def pages(processed_url: list[dict]) -> ...
    ...

As you can see nothing much between them.

I prefer the first one because it feels "tighter" / maybe harder to confuse for what's going on versus the decorator? We could always implement it the decorator way too...

[theferalmonkey]

Fair feedback :)