ClankdarModel benchmark.
Five models were given the same 500 puzzles. Here’s how often each was exactly right.
| Requested model | Correct |
|---|---|
| openai/gpt-5-mini | 87.6%437/499 correct |
| google/gemini-2.5-pro | 80.6%403/500 correct |
| openai/gpt-4.1 | 41.0%205/500 correct |
| openai/gpt-4o-mini | 28.0%140/500 correct |
| meta/llama-3.1-8b | 9.4%47/500 correct |
Recorded September 17, 2026. One answer per puzzle, no tools. A pass requires the correct answer in the requested format. One timed-out request is excluded from the response count.
These scores come from an earlier puzzle set, clankdar-suite-v2. Checks use ALGAL program puzzles by default, and no model scores for those puzzles have been published.
Method and downloadable evidence
Each model received the same 25 task settings and 20 seeds per setting: 500 attempts per model, 2,500 in total. Calls used Vercel AI Gateway with a requested temperature of 0 and a 4,096-token completion budget. The table shows the requested model aliases, not independently verified model identities.
The scorer compares each complete response with a deterministic reference answer. Provider errors are separate from wrong answers. The report retains counts, confidence intervals, and results for each task; formatting diagnostics are not counted as passes.
The recorded test is clankdar-suite-v2, with scorer clankdar-score-v2 and seeds 101–120. The manifest records the suite hash and source revision, including uncommitted changes present when the runs were collected. Its limitations section describes the gaps in the experiment record. Every published score can be recomputed from the archived responses.
Download the full report · Inspect the run manifest
- google_gemini-2.5-pro-962e7b87ec69.jsonl.gz
- meta_llama-3.1-8b-9c3e6009a3b5.jsonl.gz
- openai_gpt-4.1-c8893463e7ce.jsonl.gz
- openai_gpt-4o-mini-94d9f0da9727.jsonl.gz
- openai_gpt-5-mini-020751dc92eb.jsonl.gz
The reports and manifests for these runs are also on Hugging Face as Clankdar benchmark reports, licensed CC BY 4.0. The license covers those files, not the recorded model responses.
Twenty seeds per task setting are an exploratory sample. Public puzzles can be memorized, delegated, or solved with code. These scores describe this run only; they don’t measure general intelligence or safe behavior.
Other recorded experiments
These records use different tasks or conditions and are kept separately from the comparison above.
For your own model comparison, see the benchmark runner guide.