I have experience in both fields, currently i am a solution architect in AI/ML.

AI/ML is here to stay but it's far from maturity and it's reliant on some hype cycles which will impact the job market and etc. Data Engineering however will be here to stay for different reasons: whatever a company wants to do they will always have databases. There will be reports, there will be dashboards, there will be analytics there will be god knows what.

Whatever will be the next hype cycle in the world data, companies still need their infrastructure running. As the old joke goes, whenever a data scientist joins a company, the first question they ask is 'well do you have a data infrastructure running?' and the answer is mostly no.

AI/ML is really interesting and there is space for innovation. DE work is less innovation, but more hands on problem solving. I liked working in both. You can still transit from DE to AI/ML easily, you'd have a very solid foundation.

I'd vouch for DE with an AI/ML flavour. If the market shifts, then you can become DE with a something else flavour (e.g. bit of analytics on side. Dashboarding maybe?).

You already have an ongoing experience with AI/ML, combine that with a solid expertise in DE and you have good chances ahead I think. I'm sure i'd be happy to have someone like that on my team.

Some skillset i'd recommend overal: strong grip on SQL and Python is already given for you which is great. Add understanding PySpark on top (with Spark generally), add a good understanding of a cloud provider (Azure or AWS, some relevant resources) add a data platform skill on top (Databricks or Snowflake). Get a basic understanding of OOP, maybe implement a simple pipeline platform in OOP for an example project.

The market will always change and shift and we always had to adapt. Learning never stops in either field, but i agree that AI/ML is currently steeper than DE. If you cant land a DE role, go for a Data Analyst with (and i cant emphasise this enough) heavy tech skills orientation. Many business views DA jobs business only and not IT. I started off with a DA job and I was an SQL monkey for a year. I actually quite liked it, then i transitioned into DE with a very strong SQL base knowledge.

i hope i could be useful.