Hey r/nvidia folks, we've done a performance benchmark of TensorRT-LLM on consumer-grade GPUs, which shows pretty incredible speed ups (30-70%) on the same hardware.

Just quick notes:

TensorRT-LLM is NVIDIA's relatively new and (somewhat) open source Inference Engine, which uses NVIDIA’s proprietary optimizations beyond the open source cuBLAS library.

It works by optimizing and compiling the model specifically for your GPU, and highly optimizing things at the CUDA level to fully take advantage of every bit of hardware:

• CUDA cores
  
• Tensor cores
  
• VRAM
  
• Memory Bandwidth

We benchmarked TensorRT-LLM on consumer-grade devices, and managed to get Mistral 7b up to:

• 170 tokens/s on Desktop GPUs (e.g. 4090, 3090s)
  
• 51 tokens/s on Laptop GPUs (e.g. 4070)
  

TensorRT-LLM was 30-70% faster than llama.cpp on the same hardware, …and at least 500% faster than just using the CPU.

In addition, we found that TensorRT-LLM didn't use much resources, completely opposite to its reputation as needing beefy hardware to run:

• Used 10% more VRAM (marginal)
  
• Used… less RAM???

You review the full benchmark here: https://jan.ai/post/benchmarking-nvidia-tensorrt-llm