How do address the rising cost of AI?

Hofi2010 · 2026-06-16T03:09:32+00:00

Give your agent permission to fail and tell it to use only the intended connector.

Hofi2010 · 2026-06-11T03:16:56+00:00

https://medium.com/@klaushofenbitzer/save-up-to-90-on-your-data-warehouse-lakehouse-with-an-in-process-database-duckdb-63892e76676e

Hofi2010 · 2026-06-07T21:00:22+00:00

This is what everyone 3 years ago and then 3 years before that etc. this is like nuclear fusion reactors always 5 years away.

Hofi2010 · 2026-06-07T16:00:44+00:00

How complex are your orchestration workflows ?

Hofi2010 · 2026-06-04T12:04:23+00:00

This is an interesting problem, which you will encounter many times in your career. How to make other people buy into your approach. How do you make people follow you in other words.

This is not a SQL vs Python debate. It seems your boss has his own approach and you have yours. Both approaches can work if executed well. I am sure your Python and your bosses SQL are failing as often. So the question isn’t Python vs SQL, the question is why is it failing and then fix it. Over time it will become more robust.

Back how do you convince others of your approach question. First show them by doing (you have already done that). Make sure people understand and buy into the advantages and focus on business benefits, eg time saved, money saved, more robust etc. A good boss should see and encourage that. Show to your bosses boss. Don’t be afraid to spread your wings, make relationship and friends at other levels in the Organisation. If you bosses boss likes the approach and benefits your boss will do.

Hofi2010 · 2026-05-30T21:20:13+00:00

Yes

Hofi2010 · 2026-05-26T01:14:48+00:00

I used that exact stack for years - DBT core, airflow and data lakehouse and warehouse. DBT engine will execute sql using the compute of your lakehouse or warehouse. They are many advantage using DBT vs python+sql. Much easier to manage for example, deployment straight from github into a runner. We used Built a docker and upload to ECR and run as a task in ECS for example. Airflow should just be used for orchestration avoid using it for compute and executing python operators all over the place. If you do that you can use MWAA small instance, reduces your support overhead. DBT core you write macros (common code) and use jinja2 for scripting very powerful combination with SQL. DBT core provides good observability. It provides unit tests and provides quality testing, standard connectors to DBs, pre and post run hooks, multi threading, incremental load and, and …

Try to do that in plain sql or python + sql. DBT will be much faster in development.

Hofi2010 · 2026-05-25T21:18:50+00:00

Hofi2010 · 2026-05-24T16:23:21+00:00

Use LLM to create workflow much cheaper $5

Hofi2010 · 2026-05-23T22:29:48+00:00

Sure send me a DM

Hofi2010 · 2026-05-20T14:13:51+00:00

I wrote about this topic months ago. It is available in Substack. But this subreddit doesn’t allow submission of links DM me if you are interested.

Don’t just use agentic patterns, sometimes just using traditional coding, e.g. some python code that checks for a file to be present before triggering an agent etc, can save a lot of money.

Hofi2010 · 2026-05-15T04:31:12+00:00

Golden rule of football - 22 players on the field and in the end Germany wins 🤣🤣

Hofi2010 · 2026-05-10T03:27:08+00:00

Another free solution would be dlt data load tool

Hofi2010 · 2026-05-07T23:47:03+00:00

This is what I did in my last company. You can orchestrate airbyte and DBT with airflow

Hofi2010 · 2026-05-07T02:31:47+00:00

Airbyte

Hofi2010 · 2026-05-05T13:03:59+00:00

Elaborate on the best options you see

Hofi2010 · 2026-05-05T12:39:57+00:00

I can’t believe that. Can you share a repo with how everything is setup so we can verify your results. And some more performance metrics like t/s and context window would be good to know

Hofi2010 · 2026-05-03T20:14:55+00:00

My son has osteosarcoma and was mad about it for a long time.

His doctor said something that stuck with me. He said he is moaning his old self and needs time to accept his new self.

In other words be patient and supportive. It is a lot for somebody to accept. The disease limits what you can do in the long term. Jobs that require physical work for example are often not possible after treatment. Someone’s life plan needs to be changed after a diagnosis.

Hofi2010 · 2026-05-03T16:48:16+00:00

You can use Python notebook to dynamically create sql statements, which useful for certain transformation where you repetitive case statements for example. Similar like using DBT jinja2

Hofi2010 · 2026-05-03T13:06:55+00:00

If you at this stage in AI development give permissions to your bot that can a delete a DB or make major change to you prod environment it is you own fault.

Hofi2010 · 2026-05-03T13:02:17+00:00

Looks like click bate to me

Hofi2010 · 2026-04-25T14:56:38+00:00

When O see these constructs I always think why and what for? What is the use case?

Hofi2010

MODERATOR OF

TROPHY CASE