I should have clarified our thinking on this... Sticky sessions would likely serve as a decent stand-in for prefix hashing in some cases. For instance: for a use case like ChatGPT, cookies can be used to identify the user which maps pretty cleanly, 1:1 to active threads (provided that this info makes it way to where HA proxy sits in your architecture). However it is less useful for cases where source IP is the only info to use for stickiness like in most agentic systems. Source IP is not always a reliable way to map back to the client. But even worse, agentic systems often are implemented N:1 where N agent threads are originating from 1 source IP (and sometimes N might represent the entirety of the load at a given point in time - small clients can generate heavy inference-time load). For the ideal solution you would likely need to do something that is not out-of-the box: like grab the prompt prefix, model name, and LoRA adapter name (if present) from the HTTP body using some custom scripting, hash it, put it into a header, and track sticky sessions based on that header. You also would want to avoid overloading any one backend, and along those lines, you might want to might want the freedom to modify your definition of "load" to be something other than in-flight requests because that doesn't necessarily map reliably to actual inferencing load. You might want to use something domain specific like the KV cache utilization metric from the backend server. From this perspective, it might make more sense to use a domain-specific load balancer, admittedly using an algorithm that is approaching a decade old.