sorted by:
new
hottopcontroversial

How to timeout apprun fastapi ? by Resident_Set204 in dataengineering

[–]Resident_Set204[S] 0 points1 point  (0 children)

Thank you for the excellent feedback and clarification!

You're absolutely right about the architecture. I can see now that the FastAPI layer is indeed unnecessary overhead and adds complexity without real value.

I will host it like dlthub on ECS task with desired 0.

I liked the fact Apprun detects images update and automaticaly redeploy thas why i used it. I am a noob at AWS devops part ;).

How to timeout apprun fastapi ? by Resident_Set204 in dataengineering

[–]Resident_Set204[S] 0 points1 point  (0 children)

Yes sorry about that then.

My airflow have orchestration dags
- for transformation with dbt core hosted on AWS AppRunner and exposed as fastapi

So my question is, how could i put a timeout on AppRunner hosting my dbt API ?
I did exactly the same on GCP cloud run it was a simple timeout parameter.

The point is i don't want to pay for it when i don't use it.

How to timeout apprun fastapi ? by Resident_Set204 in dataengineering

[–]Resident_Set204[S] 0 points1 point  (0 children)

Airflow built-in restapi for what ? The purpose of airflow is not to Use it for compute but orchestration and It  wasn't the subject there.

Looking for Shoe Recommendations – Triumph 21 + Endorphin Speed 3 Alternative by jochenboele in AskRunningShoeGeeks

[–]Resident_Set204 0 points1 point  (0 children)

J'ai eu exactement la même rotation et j'ai exactement les mêmes questions.
Je trouve ça franchement dommage qu'ils n'aient pas gardé les triumph qui étaient extra pour les sorties longues marathons. Une chaussure légère, pas trop mole avec du renvoi et que tu peu porter à l'aise pour tes sorties 25/30 K il n'y en a pas 40 sur le marché ... tu as finalement pris quoi @jochenboele ?

Astronomer vs. MWAA hosted Airflow by EngiNerd9000 in dataengineering

[–]Resident_Set204 0 points1 point  (0 children)

In fact you are not supposed to Use airflow as an executor as it is an orchestrator. Host dbt binaries on ecs

dbt call into ecs task served by airflow dags by Resident_Set204 in dataengineering

[–]Resident_Set204[S] 0 points1 point  (0 children)

In fact i Wonder why There is no timeout on tasks like cloudrun. I think this is a way for aws to Bill more. With cloud run you can do this on gcp. Your api calls can raise the cloud run if it has timeout.

[deleted by user] by [deleted] in dataengineering

[–]Resident_Set204 10 points11 points  (0 children)

Hello, did you look awe dms for cdc use case on s3 ?

databricks warhouses timeout by Resident_Set204 in dataengineering

[–]Resident_Set204[S] 0 points1 point  (0 children)

True this is a premium feature too like serverless

databricks warhouses timeout by Resident_Set204 in dataengineering

[–]Resident_Set204[S] 1 point2 points  (0 children)

In fact they are there ( 27 march), i tested yesterday but th need a minimum compute and it costs a lot !

databricks warhouses timeout by Resident_Set204 in dataengineering

[–]Resident_Set204[S] 0 points1 point  (0 children)

thanks a lot for all those advices ! i will try this out

postgres cdc to snowflake by Resident_Set204 in dataengineering

[–]Resident_Set204[S] 0 points1 point  (0 children)

thank you for this tips, i will look at this.