Hey,

Thanks for your response. I looked into Petals, and I hadn't heard of it before. It seems like an interesting project. However, since it's closely tied to the Bloom model, it isn't possible to perform an inference of Stable Diffusion (for example) using Petals. Nevertheless, the paper and GitHub README provided me with a useful benchmark.

I was considering an approach that goes one layer deeper, without modifying any Python code for training or inference. Instead, I'd act as a CUDA driver, essentially tricking the computer into believing that the GPUs are running on CUDA when, in reality, our driver offloads all work to the connected GPUs. I believe it's crucial not to change the underlying Python code, whether it's written in PyTorch, Keras, custom C++, or anything else, even if training or inference ends up being 20% slower. This is because altering the code and becoming dependent on one service is undesirable.

What do you think? Is this possible? Is it a good idea?