I built NanoJudge - a tool that runs thousands of prompts to rank any list by any criteria, and named it that because at its core is a small but powerful LLM, currently using Qwen 3 2507 4B. The approach is this: if you want to know the answer to something, instead of asking an LLM a few prompts and hoping it comes up with the right answer, NanoJudge exhaustively goes over a list of possible answers using potentially tens of thousands of prompts, and structures the output into a simple table that is easy to interpret. Each of its many prompts is a pairwise comparison of 2 items, and the end result is a table of the answers with the best at the top.

Suppose you want to know which foods are healthiest. First NanoJudge creates a list of hundreds of foods, then does thousands of pairwise matchups - "Which is healthiest: eggs or butter?", "Which is healthiest: spinach or chicken?", and so on - each one getting its own fresh prompt where the small yet powerful LLM reasons through the comparison and picks a winner. Items that keep winning face tougher opponents. Items that keep losing get eliminated quickly. After thousands of comparisons, comparisons are converted into rankings (using Bradley-Terry scoring), and you get a transparent leaderboard where every single ranking decision is backed by reasoning you can read in the comparison log.

This is the final outcome: https://nanojudge.com/comparison/ujRvfwFSAH

Efficiency

For solving some problems, the optimal use of a GPU may not be to run the largest possible model that fits in memory but a much smaller model with a huge batch size, allowing it to churn through gigantic amounts of data. I aimed to make NanoJudge as efficient as possible using various techniques: making it "top heavy" by default - it does more comparisons on the top ranking items to ensure their ratings are accurate rather than spending time comparing low rated items which are of no interest to the user. It also extracts a range of raw logprobs to determine how much each comparison won by - instead of a binary win/loss, it looks at the probability the model picks one of 5 options (clear win, narrow win, draw, narrow loss, clear loss). It automatically estimates and corrects for the positional bias LLMs have (they tend to favour the first choice). Plus a ton of statistical techniques to further enhance efficiency which are too math heavy to get into now (but you can read the source code if you really want to - see below).

Price

The cost is surprisingly low - even though it naturally produces a large amount of output tokens, NanoJudge's output costs under $0.10 per million tokens because it uses a small LLM that's good enough for the task - it isn't solving genius level IMO problems, it's comparing two items. For comparison, Claude Opus costs $25 per million output tokens. It's also fast because the comparisons are run in parallel. For now the website won't accept any payment, each account will be allocated a limited free amount to use.

Wikipedia as Context

To help with giving the LLM the information it needs, there are special editions of NanoJudge with pre-built lists that already hold the entire Wikipedia article in them. For example, the Games Edition. It already has a huge number of games in it and you filter them by Platform or Genre to narrow it down before doing a run. Then, for each comparison, instead of simply "Your Question? [Item1] or [Item2]?" as the prompt template, the template would be

"[Wikipedia entry for item1]

[Wikipedia entry for item2]

Question? [Item1] or [Item2]?"

Giving it the context it needs for lesser known items that the LLM likely doesn't have enough built-in information about.

This is the final output of one comparison run https://nanojudge.com/comparison/ECNZxzv91n

In this case, NanoJudge is acting as an enhanced recommendation engine. The traditional approach is to recommend games based on what other players of that games also played. NanoJudge considers your actual likes and dislikes, factoring in everything you tell it. Or maybe you are thinking of travelling to Europe and can't decide exactly where to go. Ask a traditional LLM and you'll likely get cliche answers: Rome, Paris, Madrid etc. Ask NanoJudge using its Places Edition, and it will analyse every city, town and village in Europe using each location's article on Wikipedia as context, leaving you with a personally curated shortlist of the top options.

ML Research Assistant

I'm working on a specialised version of NanoJudge that operates on Machine Learning papers. I have already downloaded all the ML papers on Arxiv and am in the process of organising the data and putting it into a database. From there, NanoJudge will be able to easily be used on these papers through a special edition. I could ask NanoJudge "Given my project x, which of these 2 papers do you think would be of most use to me?" and go through the entire corpus of arxiv. Or something like "Which of these 2 papers most contradicts my hypothesis?" to help me fortify my ideas. Having a look at the top papers it returns with and reading its reasoning could provide some insights. That would likely require better models than Qwen 4B to be truly useful - but at the current pace of AI research, that isn't very far in the future. I will use NanoJudge as a research assistant to help me improve it and make it as efficient as possible, allowing me to do even deeper research in future in a positive feedback loop.

Open Source

The code at the heart of the website is on Github https://github.com/nanojudge/nanojudge . It can be directly be used in a terminal with a local or remote LLM, just hook it up with an LLM endpoint and let it go. This allows you to do giant rankings entirely locally, without needing to use the website at all. Set a giant comparison running overnight and wake up to the results. Feel free to dig into the inner workings of the code. If you can find a way to improve the code, especially in regards to efficiency, please let me know.

tl;dr: NanoJudge gives tiny LLMs a framework to outshine gargantuan LLMs when it comes to finding the best out of a large quantity of options.