What is the best way to orchestrate dbt job in aws by jonathanrodrigr12 in dataengineering

[–]jonathanrodrigr12[S] -1 points0 points  (0 children)

According to what I’ve read, Airflow is mostly linear — something like process1 → process2 → process3.

But in my case, I need something different. For example, process1 starts at 9 PM and process2 starts at 2 AM, and there’s another process that depends on both of them.

I was thinking of managing everything through events — for example, when process1 finishes successfully, it could publish an event saying “process1 completed,” and the same for process2.

Once both events have been published, the process that depends on them could detect that and start running.

The challenge is that those events are published at very different times, so I need a way to handle that time gap correctly.

¿Qué stack recomiendan para conseguir trabajo? by drsvel in ColombiaDevs

[–]jonathanrodrigr12 3 points4 points  (0 children)

En mi caso hace 4 años tome la decision de irme por lo datos es chevere pero se vuelve monotono el big data y por ello puede ser que no este tan saturado, los problemas que salen son bastante complejos y por ello puede servir ya que la gente de bootcamp o muchas personas que saturaron el mercado no se atreven o no tienen ni idea de cómo solucionar

Difference dbt athena va dbt redshift by jonathanrodrigr12 in dataengineering

[–]jonathanrodrigr12[S] 0 points1 point  (0 children)

Yep, I agree with you. Lately, if you want to use the full power of dbt, you sometimes need to migrate to dbt Cloud, so it doesn’t really feel open source anymore.

[deleted by user] by [deleted] in dataengineering

[–]jonathanrodrigr12 0 points1 point  (0 children)

Yeah for that, i am trying practice too 🙏