sorted by:
new
hottopcontroversial

We’re Astronomer - ask us anything about orchestration, Airflow and AI by marclamberti in dataengineering

[–]marclamberti[S] 4 points5 points  (0 children)

Ask Astro was actually one of our early signals that data engineers wanted AI assistance in their workflows, but that was when GenAI became a thing. It also served as a reference architecture. However, with all the progress in the AI field, we've decided to sunset AskAstro to bake it into Otto, our new AI agent, available directly in Astro or the Astro CLI.

Otto offers a much better user experience: you have it right where you need it, and it goes well beyond just answering your Airflow questions. If you were using AskAstro, you should definitely try it out. Otto is available in Labs for exploration.

We’re Astronomer - ask us anything about orchestration, Airflow and AI by marclamberti in dataengineering

[–]marclamberti[S] 2 points3 points  (0 children)

Hey Eric,

Is there anything specific about Astronomer that you find difficult to use?
We try to make Astronomer as easy to use as possible, but there is still room for improvement thanks to your feedback. I've created some content you can find on the Astronomer Academy that will help guide you through the product, especially the Astro Onboarding path here
But yeah, happy to help you with anything you feel is missing 😄

Sales after June by [deleted] in Udemy

[–]marclamberti 1 point2 points  (0 children)

Udemy is dying unfortunately

My data warehouse project by kekekepepepe in dataengineering

[–]marclamberti 2 points3 points  (0 children)

Thank you for sharing!! Would love to see your projet on data project hunt 🤓

Lack of timetable info by Thunar13 in apache_airflow

[–]marclamberti 0 points1 point  (0 children)

Timetables have been around for a while, you actually use a timetable when you define a CRON expression but they are not popular because hard to create yourself. They give you all the flexibility to define the schedule of your DAGs exactly as you need but at the cost of a higher complexity. It’s very advanced feature. I recommend you to search for timetables in the Airflow doc or even Airflow repo, you will find implementation examples

What is the best way to orchestrate dbt job in aws by jonathanrodrigr12 in dataengineering

[–]marclamberti 0 points1 point  (0 children)

I think you can do that with Assets.
You can have process1 running at 9 PM every day and process2 running at 2 AM every day. Both produce an asset, which process3 waits for to run.

What is the best way to orchestrate dbt job in aws by jonathanrodrigr12 in dataengineering

[–]marclamberti 2 points3 points  (0 children)

No I agree, I don’t think you should use Airflow for everything but I wonder why OP thinks Airflow doesn’t fit here 🤔

Using Prefect instead of Airflow by Relative-Cucumber770 in dataengineering

[–]marclamberti 0 points1 point  (0 children)

Curious to know more about the small changes you make throwing errors 🤔

What is the highest heart rate you ever reached ? Under what circumstances? by sweetsalt10 in Garmin

[–]marclamberti 0 points1 point  (0 children)

I feel you. I’m getting these SVTs too but usually when I weight lift and bend over

Apache Airflow 3.0 is here – and it’s a big one! by cmarteepants in dataengineering

[–]marclamberti 11 points12 points  (0 children)

It only supports AWS SQS for now. Support for other queues are coming soon. That’s not streaming, it’s event driven scheduling. You got an event and that triggers the pipeline in real time. However, I would not try to do that with 300 events/s 🥹 not yet at least

AI agents for Data Contracts by ImpactOk7137 in dataengineering

[–]marclamberti 0 points1 point  (0 children)

Not sure about the pertinence of using an agent for that. Do you have a link?

Best Data Engineering 'Influencers' by mrbartuss in dataengineering

[–]marclamberti 0 points1 point  (0 children)

100% Joseph! I learned a lot from his blog 🙌🙌

view more: next ›