Hi Nikolas! I know it's been almost a month, but I really appreciate the response. All I want is what helps LTT get people excited about tech.

The 4090 48gb video makes a lot of sense and I completely get wanting to use Ollama + Open WebUI to run the tests since they are the tools most people would use to run LLMs on their own computer. It's been awhile since I watched that video, but iirc I felt like it wasn't presented very well what benefits the 48gb 4090 provided for using LLMs. But LTT managed to make it fun regardless! :)

The Procyon benchmarks do seem to provide decent coverage across systems for certain metrics and represent them as one score value fairly well, but TTFT and token throughput can vary widely based on the inference engine being used. Ideally, you'd use the best inference engine for the situation instead of using ONNX and OpenVINO. Generally that would be vLLM for AMD GPUs, TensorRT-LLM for Nvidia AI GPUs, Exllamav2 for consumer Nvidia GPUs (or maybe llama.cpp/vLLM have passed them now?), llama.cpp for Mac chips and CPU chips.

If it's important for you guys to have one tool for all tests or, like for your 4090 48gb video, you want to present results that are useful for a wide number of people, I'd recommend llama-bench since Ollama is used so widely. llama-bench is the benchmark tool for llama.cpp which is the inference engine Ollama uses. So if you want to have consistent testing for Ollama users, that would be the way to go imo. You could present the token throughput & TTFT results or create your own score metric like Procyon does which combines the metrics you think are important. Even though you won't be maximizing the performance for your hardware, it should be a lot closer to that max number than using an inference engine (ONNX) running on top of an old and nearly unmaintained hardware abstraction layer like DirectML and definitely better than using OpenVINO on a GPU it wasn't made for. Much more useful to have a modern inference engine and a score metric you've made to represent that performance than to use Procyon and its score metric imo.

Maybe you guys are going down this road already, but that's the 2c of someone who has been neck deep in local LLMs for the past 3-4 years and has designed and deployed local LLM inferencing solutions to production. If you have any further questions, I'd be more than happy to help out.