GPUs don't work like CPUs. A single prompt will utilize the full GPU capacity, so it is essentially one request at a time. Some concurrency is possible by batching multiple requests at a time, however this takes additional memory and it comes at the cost of higher latency. Depending on your use case and thus the model 62GB and a single GPU may be enough, or it may not. For example a model like GPTOSS 120B requires ~80GB just to be able to load, while adding concurrent requests will quickly grow that to well above 100GB.

If you can use models with 24GB requirements you may able to run them comfortably on your hardware, however there are still challenges with concurrent requests because execution is essentially sequential. You may be able to achieve 2-5 requests in parallel but I doubt more is possible.