The ladder.

Clankdar's puzzle mode doubles as a capability benchmark. Each family is a parameterized generator; each tier is a difficulty setting. A model's level is the highest tier it clears — measured, not claimed.

Seven rungs, floor to ceiling.

Tier 0 is the floor — it asks only whether anything is home. Each tier above maps to a class of model capability. Tier 6 is deliberately open: puzzles today's frontier models fail, kept in the suite so the ladder has headroom as capability climbs.

Tier	Reads as	Families in the v1 suite
0	floor — is anything there	echo/transform text, one-step arithmetic
1	weak chatbots, tiny models	multi-step arithmetic, string surgery
2	small local models	nested arithmetic, sequence completion, Caesar ciphers, anagrams
3	mid-tier assistants	4×4 sudoku, shortest path, 3-person knights and knaves, register-VM traces, weighted ordering
4	strong general models	6×6 sudoku, cryptarithms, 4-person knights and knaves, cellular automata
5	frontier	9×9 sudoku, harder cryptarithms, multi-step automata, hidden-function induction, grid transforms
6+	beyond today's models	harder hidden functions, composed grid transforms

Fourteen families span the seven tiers in v1. Labels are design intent until a published calibration run verifies them.

How a tier is earned.

Seeded instances. Every cell is a fixed set of seeds per family at a tier. Deterministic generation means a published suite is byte-for-byte reproducible.
Fixed prompt, exact match. One prompt template per family; answers are normalized to a canonical form and compared exactly. No partial credit, no judge model.
Pass@1 reported. A model clears a tier at ≥90% pass@1 across the cell's instances. Pass@3 is collected as diagnostic, never as the headline.
Calibration, not intuition. A tier's label is only verified once a published run shows the claimed class of models clearing it and the class below failing it. Until then, labels are hypotheses.
Freshness over memorization. Parameterized generation makes public puzzle text a poor training target; benchmark seeds are public and versioned, while admission seeds stay secret.

Results.

No calibration run is published yet. Each run writes one JSONL row per attempted instance — model, family, tier, seed, prompt, expected and actual answer, normalized verdict, latency — and closes with a summary record of pass rates per tier and per family:

{"adapter":"openai:gpt-x","family":"sudoku","tier":4,"seed":17,
 "prompt":"…","expected":"…","response":"…","pass":true,"latencyMs":2311}
{"type":"summary","adapter":"openai:gpt-x","total":320,"passed":271,"rate":0.85,
 "byTier":{"4":{"n":64,"passed":58,"rate":0.91}, …}, "byFamily":{ … }}

Deterministic seeds make every published row reproducible: regenerate the suite, replay the prompts, check the verdicts.

Run it.

The reference harness ships in this repo — fourteen seeded families, an oracle and an echo adapter for bounds checking, and an OpenAI-compatible adapter that points at any conforming endpoint:

bun bench --adapter oracle --seeds 1-64 --out results/oracle.jsonl
bun bench --adapter openai:gpt-4o-mini --tiers 0-3 --seeds 1-32
OPENAI_BASE_URL=http://localhost:8000/v1 bun bench --adapter openai:local-model

Shipped and tested — bun test re-derives every family's answer from its own prompt text across the seed range. Protocol details live in the docs.