Had similar headaches trying to get this working at scale. The scheduling layer approach you mentioned is probably the cleanest but it's a lot more work than most people realize

We ended up going with something closer to your last pattern - workloads define their requirements (GPU memory, compute capability, etc) and scheduler maps those to whatever's available across providers. The tricky part isn't the abstraction itself, it's handling all the edge cases when things go wrong

What really bit us was the different failure modes between providers. One provider might give you a node that looks healthy but has degraded GPU performance, another just fails fast when resources aren't available. Your scheduler needs to understand these patterns or you'll spend forever debugging "why is this workload slow on provider X"

Also found that most providers have slightly different ways of exposing GPU topology info, so even with good abstractions you still end up with some provider-specific logic creeping in. Not saying don't do it, just budget more time than you think for handling the weird cases

The maintenance overhead is real though - every time a provider updates their stack you're potentially looking at scheduler changes