assay gives a small model on your own laptop a question and gets back an answer with a probability you can check. Every picture below is replayed from real benchmark runs. Nothing on this page is made up.
Pick a task and an item. Each system's odds for every option are drawn as bars. The right answer is marked with a tick.
When it says "80% sure", is it right about 80% of the time? A perfect system follows the diagonal. Lines below the diagonal are overconfident: they claim more certainty than they have earned.
Keep only the answers the system is most sure about and ignore the rest. A trustworthy confidence number makes accuracy climb as you answer less.
Each item is asked twice: once with the options in one order and once reversed. This chart shows how often the top answer changed. Lower is better.
This part needs the server running on your machine, because the pictures above are replays and this one is live. Start it with python -m assay serve --model gemma3 --cors --port 8787, then press Ask.
Answers appear here.