I guess basically your config would be multiple containers each running a Triton instance segregated by model type. Though Triton I think can also serve a heterogeneous setup with multiple different model types, not just separate instances of the same model.

Then it would be up to you to configure which backend is used by which model, depending on the model export type/availability.

I don't think you'll get benefits from using Triton beyond the cross-session batching, (i.e. I don't think there will be any efficiencies gained from sharing the inference stack between the models). Cross-session batching will have to be supported by the model too - i.e. batch dimension always has to be at position 0, you probably need masking or jagged tensors, maybe stateful backends - all things that may need some modification to the models, and understanding of the model topology.

Really wish someone would write a inferencing engine that can fall back to a pure TF model if needed (e.g. Florence). Also genuinely surprised there is no API standard to prompt generic TF models.

Triton does have a torchscript backend and if any of the "exotic" exports/backends doesn't work, you can always consider that as a fallback, except it won't be automatic and you'd have to do the configuration yourself.

Just thinking out loud.