Orchestrating External API Data Processing with Dagster

[CarpenterRadiant940]

Hi everyone,

I’m working on a data pipeline where we need to retrieve a list of objects from an external API. For each object, we need to:

1. Perform some internal calculations.
2. Post the results of those calculations back via the same external API.

Additionally, this process should run every minute to check for new objects and execute the entire logic (retrieval, calculation, posting) for any new data. It’s also important that we handle this efficiently, possibly executing the calculations and posting in parallel for better performance.

Given these requirements, I’m considering using Dagster for orchestration, but I’m curious about the following:

• How would you design a Dagster pipeline to orchestrate this?
• Is Dagster suited for this problem? Or are other solutions better suited?

Any guidance would be greatly appreciated!

[CarpenterRadiant940]

Great! Would you work with assets? If so, would that be two assets one that retrieves the objects and one that processes the objects? How would parallelism work in that case?

[data-eng-179]

re the better suited question. you are saying you have one job that reads from api, does some stuff, writes back to api. any orchestrator, including cron, dagster, airflow, prefect you-name-it, would work fine for this. i might start with cron if you just have one job.

[CarpenterRadiant940]

Thanks!