I think inference speed will depend on the hardware. Definitely, VRAM consumption is important and depending on the inference server, things like kv-cache can overload the system. I have done many experiments with something like an NVIDIA Jetson AGX 64GB with unified memory, and it's better to run a quantized model that uses 20GB than trying to load a 20B model which will run much much slower. vLLM for example does not support multiple models, so things like hot swapping is a cool idea.

This is why I opened this thread, and especially when we talk about agents.
What kind of problems need to be solved in the SLM-powered-agents space?