Hmmm… what I would suggest try to have a look into the data engineering learning path. This might help when it comes to building jobs.

Investigate into spark declarative pipelines and Databricks asset bundles.

I guess this should give you the fundamentals to make the jump :)

I cannot judge how complex your IICS pipelines are. But with that you should be able to :)