How many guesses is fair? Measure it, don’t copy it

Six is the number everyone uses, and nobody can tell you why. It is the number the game you played first happened to use, and it propagated. For a puzzle whose difficulty you have not measured, six is a guess about a guess.

This is how the limits here were arrived at instead — a method that works for any deduction game, and which produced one answer that looked so wrong we went back and checked it.

The method, in four steps

1. Write a solver that plays like a strong human, not like a computer. The one that matters may guess any word in the full dictionary, including words that cannot be the answer, and picks whichever splits the remaining field best.
2. Run it against every answer in the pool. Not a sample — all of them, so the distribution is the real one and not a draw from it.
3. Take the ninetieth percentile, not the mean. The mean tells you about a typical day. The limit has to survive a bad one.
4. Add one. A limit exactly at the solver’s p90 makes a human who plays perfectly fail one day in ten. The spare guess is the difference between a hard game and an unfair one.

What came back

Which is how the four-letter board — the one labelled Easy — ended up with 10 guesses while the six-letter board gets 8. That was not a kindness. The solver genuinely needs more room on short words, for reasons that turned out to be structural rather than accidental, and which are their own post.

The measurement that could have killed the game

The reason to do this before building anything is that it is also a kill criterion. The design note that started this project set one: if a naive solver needed more than about twelve guesses on the standard board, the feedback rule was too weak to build a game on and the whole idea was dead.

The naive solver — one that only ever guesses words which could still be the answer, with no cleverness at all — needs a p90 of 5 on the regular board. Not twelve. Six.

That gap — between what the arithmetic predicts and what a solver actually does against a real word list — is the whole argument for running the experiment rather than reasoning about it. It took an afternoon and it was the difference between shipping and abandoning.

Three solvers, because one would have lied

Only the third is the right one to set a limit from, and the other two are there to bracket it. The greedy solver knows the answer pool, which no player does, so it flatters the game. The naive solver never spends a guess on pure reconnaissance, so it is a floor rather than a model. Setting the limit from either would have produced a board that was quietly too tight or quietly too loose, and there would have been no way to tell which.

If you are building one of these

• Measure before you pick a number. An afternoon of solver runs is cheaper than a year of players quietly finding your game unfair.
• Percentiles, not averages. Nobody remembers the median day.
• Model the player you have, not the one you can write most easily. A solver that can see your answer list is not measuring your game.
• Let the measurement overrule the label. Ours said the easy board was the hard one. We kept the label, because it describes word length and that is what people choose on, and we moved the guess limits instead — but we did not pretend the result had not happened.
• Re-run it when the word list changes. The limits are a function of the pool. Change the pool and they are stale, silently.

Reproduce it

npm run calibrate regenerates the whole table from the word lists the game ships, and the limits in the code carry a comment pointing at it. The numbers on this page are read from that output rather than typed, so if the pools ever move, this post moves with them.