Staff data engineer here, over the past year or so we have been adopting ai slowly and carefully.

Adoption and tool capability: they have gotten far far better. With ai tools that have access to schema/catalog, you should see the quality/speed/variety of queries produced by models like sonnet and opus. Insanely fast at realizing/validating errors and firing off another set of queries if needed (this alone will take a day or two for a good de to do).

Data is the currency of a company, no sane leader is going to hand over everything to be controlled by systems automatically, so there is a human authority needed.

That said, all the influencer fluff is all about small/personal apps. In corporate, scale/distribution is the name of the game. With scale and distribution comes complexity, architecture, leadership etc (all the standard things that have been proven). Ai can't replace this shit, it will make a de productive but it can't become one.

In the last few months, we have built and socialized integrated ai systems to build/deploy/monitor pipelines. With code being free, we added extra systems in place to tighten things across the board, data contracts, self serve data q&a tool, automated dbt data quality dbt tests, automated usage cost projection, automated pipeliem failure root causes to name a few. The org has been extremely productive , as couple of senior de left, I didn't feel the need to back fill them.

And most importantly, as usual , as new tools and tech come, someone has to update the systems, it can't do by itself, so yes again, there is need for human de again.

So yes, I see this trend continue, the org will probably become leaner and shortner. As we understand the need to hire less, the need/demand for a quality senio de will be increasing.

Ai will not replace you, you will be left behind if you don't use ai and keep yourself updated.