How Nerf Watch measures changes

What the score measures

Nerf Watch measures performance on a fixed private suite. A lower score is a lower score on this suite. It doesn’t prove that a provider changed a model.

The suite is version 2026.10.1. It has four prompts and 34 scored tests:

• trick: 5 tests of trick questions.
• exact: 12 tests of exact counting and sorting.
• rules: 16 tests of writing to letter-level rules.
• apps: 1 test of building a small working app.

Six more questions appear in the prompts but do not count toward the score.

The suite’s pinned SHA-256 is:

d1a8471b0f64651748607f597b3911cabc46dcdc130a9531404f31f9639ccb5d

The runner verifies it before every attempt and again before grading.

How the questions were chosen

139 candidate questions were screened over three rounds, each run twice on ten models at low effort. Kept were questions the three strongest models (then GPT-6.1 Sol, GPT-6 Astra and Opus 5.5) passed at least 83% of the time, with the other seven models passing at least 15 points less often. The survivors were confirmed on fresh runs; 34 held.

The strongest models start near 100, so the suite shows drops well and gains poorly.

Models and access

OpenAI models run through the Codex CLI signed in to ChatGPT Pro $500. Anthropic models run through Claude Code signed in to Claude Max 20x. These are subscription sign-ins; no provider API keys are used.

The runs use one machine and one account per provider, at low effort. The first nine entries use the default service tier. The two Ultrafast entries run GPT-6.1 Sol and GPT-6 Astra again on the Ultrafast tier, and each keeps its own score history.

Daily runs

Runs are scheduled daily at 01:00 UTC. Models run serially in the order above. Each run is dated by the UTC day when the collection started.

Each kind of test uses a fresh one-turn session, in trick, exact, rules, apps order. There is no warm-up or shared conversation history. Each turn has a 1,800-second timeout.

A turn without successful provider completion may be retried once in a fresh session after cleanup is confirmed. Account and configuration failures are not retried. A completed turn is never retried.

Scoring

Code does all the grading; no AI model judges another. The app runs in headless Chrome, with every routed request except its own file blocked.

Seven graded tests can earn partial credit. The other 27 tests count 100 for a pass and 0 for a fail. The score is the mean of all 34 test percentages, rounded to two decimals.

Scoring example

Suppose 32 tests score 100, one pass/fail test scores 0, and one graded test scores 60.

(32 × 100 + 0 + 60) / 34 = 3260 / 34 = 95.882... → 95.88

Missing runs

An incomplete category, an invalid session or failed grading means no published run for that day. The day stays a gap; it is never filled with zero.

A blank or whitespace-only reply is treated as missing and is not graded. A completed nonblank refusal is graded as given if the session is valid. A completed answer that scores zero is published as zero.

A confirmed account failure skips that provider’s remaining models for the day.

Changes and baselines

A change is the latest score minus the previous recorded score, even if there are gaps between their dates. A change is shown only when those two runs use the same suite version.

The baseline is the mean of runs from seven days before the latest day through the day before it, using only the latest suite version. It needs at least three runs; with fewer, no baseline or trend is shown. The calculations round scores, the baseline and both changes to two decimals.

A score more than 2 points below the baseline is down, more than 2 points above is up, and within 2 points is flat. A new suite version starts a new baseline.

History keeps a model’s last 366 days that have a run. Days without a run don’t use any of them.

Change example

These example runs use one model and one suite version. Assume the selected evidence is available.

The baseline window is Oct 13 through Oct 19. It includes Oct 14, Oct 16 and Oct 19; Oct 12 is outside it. The three scores sum to 289.50; dividing by 3 gives a baseline of 96.50.

The change from the previous run is 87.25 minus 98.50, or −11.25 points. Its size is more than 10 points, so the chart offers the evidence marker in this example.

The change from the baseline is 87.25 minus 96.50, or −9.25 points, so the trend is down. Oct 13, Oct 15, Oct 17 and Oct 18 have no runs and remain gaps.

Evidence

Evidence can be requested for the latest run and its previous run on the same suite version when the change is more than 10 points in either direction. The chart marker appears when the selected evidence is available and both dates are in view.

Anyone can use it to request an emailed link to the model’s answers on both days to one question. The link expires seven days after the request. Replacing a compared run or the saved comparison can stop the link earlier.

To choose the question, add the test-score changes within each kind of test, using tests present in both runs. Take the kind with the largest absolute total change, then the question with the largest absolute change within it. Ties follow the stored category order and then test order.

The answers are shown without the question, though an answer may repeat part of it.

Limits

• There is one run per model per day, so one day can be noise.
• One test falling from 100 to 0 reduces the overall score by 100 / 34, or 2.94 points.
• These results cover low effort through the subscription CLI route. They do not measure API performance or other effort settings.
• One account per provider cannot show differences between accounts.
• CLI versions, provider defaults and the runner can change. Published runs do not record which CLI version ran.

Data and reuse

Scores and history are available from /api/nerfwatch and the Nerf Watch MCP server. See Developers for access and credit terms.

Back to Nerf Watch