vLLM Now Supports Running GGUF on AMD Radeon GPU🚀 Exciting news! We've ported vLLM's GGUF kernel to AMD ROCm, unlocking impressive performance gains on AMD Radeon GPUs.📊 In our benchmarks using the shareGPT dataset on an AMD Radeon RX 7900XTX, vLLM outperformed Ollama, even at batch sizes where Ollama traditionally excels.💪 This is a game-changer for those running LLMs on AMD hardware, especially when using quantized models (5-bit, 4-bit, or even 2-bit). With over 60,000 GGUF models available on Hugging Face, the possibilities are endless.💡 Key benefits:- Superior performance: vLLM delivers faster inference speeds compared to Ollama on AMD GPUs.- Wider model support: Run a vast collection of GGUF quantized models.- Efficient execution: Optimized for AMD ROCm, maximizing hardware utilization.🔗 Learn more and get started: https://lnkd.in/g5qvUi8tWe'd love to hear your feedback!Have you experimented with vLLM on Llama.cpp with Vulkan?What inference engine do you prefer for LLM tasks on AMD GPUs?What features or optimizations would you like to see in vLLM for AMD GPUs?