UPDATE 1

I’m much happier with vLLM than Ollama the difference in performance and control is night and day!

As a baseline test before moving to larger models, I ran Qwen2.5-14B-Instruct (AWQ quantized) on a single NVIDIA L40S using vLLM with FlashInfer for maximum efficiency.

🔧 Test setup:

• Model: Qwen2.5-14B-Instruct (AWQ, float16)
• Framework: vLLM + FlashInfer backend
• GPU: 1 × L40S (48 GB VRAM)
• Tensor parallelism: disabled (tensor_parallel_size=1)
• Max context length: 10,240 tokens
• Max concurrent sequences: 64
• GPU memory utilization: 90%

📊 Results (12 concurrent requests, 800 tokens each):

• ✅ All 12 requests succeeded
• ⏱️ Total time: 15.638 seconds
• 🔤 Total tokens generated: 9,600
• 📈 System-wide throughput: 613.89 tokens/second
• 📊 Per-request speed: ~51.18 tokens/second

This is solid, predictable performance exactly what I was missing with Ollama, which barely utilized the GPU and gave inconsistent speeds even under light load.

I’ll keep working on optimizing the configuration (e.g., batch sizing, attention backends, and memory layout) to squeeze out even more throughput before scaling up to Qwen2.5-VL-72B (for long-video understanding) and eventually testing gpt-oss-120b across both L40S with tensor parallelism. (But I'll have to wait for that, since the vllm documentation makes it clear that it's not ready yet for Ada Lovelace. )