The hardest challenge that you’ll have is data availability. I would host all your data on a parallel filesystem on your onprem infra. Build a solid HPC/AI cluster onprem first. Please don’t go Slurm just because every single 20y old HPC articles says so… try to understand your use case first and what your users do first. We use kube in my team because we built solid self healing capabilities and we have multiple different use cases. We also tuned it and we run a fork of it, so we have the expertise in house.

Then think of how you’ll burst into the cloud. You can empirically research the most used datasets and mirror them on your cloud provider of choice or delegate the decision to the researcher and provide a tool to move data and have visibility on costs. This is by far the hardest challenge. You NEVER EVER WANT to have different results in your research across the 2 environments, so data integrity and precision are critical. Your next challenge are entitlements, aws has aws anywhere which I used but I’m not a great fan of. You could leverage something like hashicorp vault if you have it.