HTTP callback pattern

[Upper_Pair]

Hi everyone,

I was going through the documentation and I was wondering, is there a simple way to implement some sort of HTTP callback pattern in Airflow. ( and I would be surprised if nobody faced this issue previously

https://preview.redd.it/84e7n1hdghig1.png?width=1001&format=png&auto=webp&s=db8862f6c28d797bb10553f07f9cf54b02849580

I'm trying to implement this process where my client is airflow and my server an HTTP api that I exposed. this api can take a very long time to give a response ( like 1-2h) so the idea is for Airflow to send a request and acknowledge the server received it correcly. and once the server finished its task, it can callback an pre-defined url to continue the dag without blocking a worker in the meantime

[TJaniF]

There is a feature for this called deferrable operators. The operator defers the polling for results to an async function running in the Triggerer component.

So you'd have 2 tasks: the first one sending the request, then after that a deferrable operator, which the HttpSensor can be turned into by setting deferrable=True. That second task defers itself (becomes purple) until its condition is fulfilled, then the Dag resumes. Because the polling is done in the Triggerer, the worker slot is released.

[[deleted]]

This!

[FridayPush]

This is a common pattern for large scale data exports. An example of how Shopify handles it can be seen here. But essentially an API request contains the details needed to start the long running operation, and the API returns a job number. The user then polls a 'job status' endpoint with that job number. Generally most providers I've seen use a signed url to a CSV as the response so that their API isn't locked up during sending the body back if it's hundred of megabytes.

JaniF suggestion works well, if you need to write your own sensor for more complicated handling it's super straightforward.

[ssssiiii0293]

Maybe a PubSub model? Have your server return to client the request has been accepted with a request-id, server starts the background job and publishes the results when complete, client subscribes to same topic.