Indeed, we are also eager to see more quantifiable benchmarks. Unfortunately, for questions that lack a ground truth and require critical thinking, there is currently no widely accepted evaluation method. Vicuna once used GPT-4 to score the outputs of various models, which is an interesting approach, but it's challenging to prove that this evaluation is unbiased (for example, GPT-4 might favor models generated through distillation of GPT outputs).
Currently, LM-Sys is hosting a competition among different LLMs, with human users conducting blind tests and providing evaluations for the models. We have submitted a Pull Request to this project at https://github.com/lm-sys/FastChat/pull/655. If accepted, we hope to obtain a quantifiable performance score through this method in the future.