Nerf Watch

Is Claude or Codex getting worse?

Once a day, the same private tests go to Claude models through Claude Code on Claude Max, OpenAI models through Codex on ChatGPT Pro, and Kimi K3.1 Pro, GLM-5.3 and DeepSeek V4 Pro through the OpenCode CLI on OpenCode Go. Effort is low, except DeepSeek V4 Pro uses high, its lowest available variant. Code does the grading. The questions stay private to reduce the risk that models are trained on them. When a score moves more than 10 points from one run to the next, a marker on that model's chart lets you ask to see the answer behind it.

Last run stored

Use these scores in your own app or dashboard: public API and reuse conditions.

Latest scores

Percent of the suite passed. A change is in points, against the model's previous run and against the mean of its runs in the 7 days before the latest, once there are 3 of them. Within 2 points of that baseline reads as steady.

Each model's latest score and how it compares
ModelLatestRun onvs last runvs 7-day baselineTrend
Fable 5.1 · Anthropic96.8Nov 11, 20260.0−1.2 from 98.0 Steady
Haiku 5.5 · Anthropic88.1Nov 11, 2026−2.6+5.7 from 82.4 Higher
Opus 5.5 · Anthropic100.0Nov 11, 20260.0+2.6 from 97.4 Higher
Sonnet 5.5 · Anthropic91.2Nov 11, 2026−5.9−6.9 from 98.1 Lower
DeepSeek V4 Pro · DeepSeek83.4Nov 11, 2026−13.7−9.1 from 92.5 Lower
Kimi K3.1 Pro · Moonshot AI83.4Nov 10, 2026+18.0−1.8 from 85.2 Steady
GPT-5.6 Luna · OpenAI52.6Nov 11, 2026+4.7−0.1 from 52.7 Steady
GPT-5.6 Sol · OpenAI92.5Nov 11, 2026+1.3−1.0 from 93.5 Steady
GPT-6 Astra · OpenAI99.8Nov 11, 2026−0.1−0.1 from 99.9 Steady
GPT-6 Astra Ultrafast · OpenAI99.8Nov 8, 2026−0.1+1.3 from 98.5 Steady
GPT-6 Luna · OpenAI16.5Nov 11, 2026+2.1+2.1 from 14.4 Higher
GPT-6.1 Sol · OpenAI100.0Nov 11, 20260.0+1.3 from 98.7 Steady
GPT-6.1 Sol Ultrafast · OpenAI99.5Nov 8, 2026+2.4+0.5 from 99.0 Steady
GLM-5.3 · Z.ai71.4Nov 11, 2026−14.2−14.0 from 85.4 Lower
Aug 14, 2026 to Nov 11, 2026, UTC, on one scale for every model

Fable 5.1 · Anthropic

Latest 96.8 on Nov 11, 2026 · Steady

020406080100Aug 14Nov 11
As a table
Day, UTCScoreSuite
Nov 11, 202696.82026.10.1
Nov 10, 202696.82026.10.1
Nov 9, 202695.42026.10.1
Nov 8, 2026100.02026.10.1
Nov 7, 202697.12026.10.1
Nov 6, 2026100.02026.10.1
Nov 5, 2026100.02026.10.1
Nov 4, 202697.12026.10.1
Nov 3, 2026100.02026.10.1
Nov 2, 202697.12026.10.1
Nov 1, 202699.82026.10.1
Oct 31, 202699.82026.10.1
Oct 30, 202697.12026.10.1
Oct 29, 2026100.02026.10.1
Oct 28, 202693.92026.10.1
Oct 27, 202699.82026.10.1
Oct 26, 202696.82026.10.1
Oct 25, 202699.82026.10.1
Oct 24, 202699.82026.10.1
Oct 23, 2026100.02026.10.1
Oct 22, 202696.82026.10.1
Oct 21, 202699.82026.10.1
Oct 20, 202697.12026.10.1

Haiku 5.5 · Anthropic

Latest 88.1 on Nov 11, 2026 · Higher

020406080100Aug 14Nov 11
As a table
Day, UTCScoreSuite
Nov 11, 202688.12026.10.1
Nov 10, 202690.72026.10.1
Nov 9, 202682.32026.10.1
Nov 8, 202684.62026.10.1
Nov 7, 202688.22026.10.1
Nov 6, 202694.12026.10.1
Nov 5, 202643.12026.10.1
Nov 4, 202693.82026.10.1
Nov 2, 202691.22026.10.1
Nov 1, 202679.12026.10.1
Oct 31, 202694.12026.10.1
Oct 30, 202682.32026.10.1
Oct 29, 202693.92026.10.1
Oct 28, 202692.52026.10.1
Oct 27, 2026100.02026.10.1
Oct 26, 202693.82026.10.1
Oct 25, 202675.52026.10.1
Oct 24, 202691.92026.10.1
Oct 23, 202696.92026.10.1
Oct 22, 202696.52026.10.1
Oct 21, 202682.02026.10.1
Oct 20, 202679.42026.10.1

Opus 5.5 · Anthropic

Latest 100.0 on Nov 11, 2026 · Higher

020406080100Aug 14Nov 11
As a table
Day, UTCScoreSuite
Nov 11, 2026100.02026.10.1
Nov 10, 2026100.02026.10.1
Nov 9, 202697.12026.10.1
Nov 8, 2026100.02026.10.1
Nov 7, 202690.92026.10.1
Nov 6, 202699.82026.10.1
Nov 5, 202697.12026.10.1
Nov 4, 202697.12026.10.1
Nov 3, 202697.12026.10.1
Nov 2, 2026100.02026.10.1
Nov 1, 2026100.02026.10.1
Oct 31, 2026100.02026.10.1
Oct 30, 2026100.02026.10.1
Oct 29, 202697.12026.10.1
Oct 28, 202699.82026.10.1
Oct 27, 2026100.02026.10.1
Oct 26, 202693.92026.10.1
Oct 25, 202698.32026.10.1
Oct 24, 202697.12026.10.1
Oct 23, 202696.72026.10.1
Oct 22, 202699.82026.10.1
Oct 21, 202697.12026.10.1
Oct 20, 2026100.02026.10.1

Sonnet 5.5 · Anthropic

Latest 91.2 on Nov 11, 2026 · Lower

020406080100Aug 14Nov 11
As a table
Day, UTCScoreSuite
Nov 11, 202691.22026.10.1
Nov 10, 202697.12026.10.1
Nov 9, 2026100.02026.10.1
Nov 8, 2026100.02026.10.1
Nov 7, 202696.62026.10.1
Nov 6, 202696.82026.10.1
Nov 5, 202696.52026.10.1
Nov 4, 2026100.02026.10.1
Nov 2, 202697.12026.10.1
Nov 1, 2026100.02026.10.1
Oct 31, 2026100.02026.10.1
Oct 30, 202699.52026.10.1
Oct 29, 2026100.02026.10.1
Oct 28, 202699.12026.10.1
Oct 27, 202695.02026.10.1
Oct 26, 202699.82026.10.1
Oct 25, 202693.92026.10.1
Oct 24, 202696.82026.10.1
Oct 23, 202695.92026.10.1
Oct 22, 2026100.02026.10.1
Oct 21, 202699.52026.10.1
Oct 20, 202696.92026.10.1

DeepSeek V4 Pro · DeepSeek

Latest 83.4 on Nov 11, 2026 · Lower

020406080100Aug 14Nov 11
As a table
Day, UTCScoreSuite
Nov 11, 202683.42026.10.1
Nov 10, 202697.12026.10.1
Nov 9, 202694.12026.10.1
Nov 8, 202691.22026.10.1
Nov 7, 202698.32026.10.1
Nov 6, 202688.22026.10.1
Nov 5, 202681.72026.10.1
Nov 4, 202696.82026.10.1
Nov 2, 202694.02026.10.1
Nov 1, 202690.42026.10.1
Oct 31, 202687.92026.10.1
Oct 30, 202691.22026.10.1
Oct 28, 202691.22026.10.1
Oct 27, 202697.12026.10.1
Oct 26, 2026100.02026.10.1

Kimi K3.1 Pro · Moonshot AI

Latest 83.4 on Nov 10, 2026 · Steady

020406080100Aug 14Nov 11
As a table
Day, UTCScoreSuite
Nov 10, 202683.42026.10.1
Nov 9, 202665.42026.10.1
Nov 7, 202687.52026.10.1
Nov 6, 202694.12026.10.1
Nov 5, 202685.32026.10.1
Nov 4, 202693.62026.10.1
Nov 2, 202669.72026.10.1
Nov 1, 202691.22026.10.1
Oct 31, 202679.32026.10.1
Oct 30, 202687.82026.10.1
Oct 28, 202692.22026.10.1
Oct 27, 2026100.02026.10.1
Oct 26, 202684.52026.10.1

GPT-5.6 Luna · OpenAI

Latest 52.6 on Nov 11, 2026 · Steady

020406080100Aug 14Nov 11
As a table
Day, UTCScoreSuite
Nov 11, 202652.62026.10.1
Nov 8, 202647.92026.10.1
Nov 7, 202661.72026.10.1
Nov 6, 202648.32026.10.1
Nov 5, 202650.82026.10.1
Nov 4, 202654.72026.10.1
Nov 3, 202662.82026.10.1
Nov 2, 202653.02026.10.1
Nov 1, 202650.82026.10.1
Oct 31, 202658.32026.10.1
Oct 30, 202650.82026.10.1
Oct 29, 202644.22026.10.1
Oct 28, 202647.92026.10.1
Oct 27, 202634.32026.10.1
Oct 26, 202618.62026.10.1
Oct 25, 202646.82026.10.1
Oct 24, 202650.42026.10.1
Oct 23, 202654.82026.10.1
Oct 22, 202647.82026.10.1
Oct 21, 202665.42026.10.1
Oct 20, 202644.92026.10.1

GPT-5.6 Sol · OpenAI

Latest 92.5 on Nov 11, 2026 · Steady

020406080100Aug 14Nov 11
As a table
Day, UTCScoreSuite
Nov 11, 202692.52026.10.1
Nov 8, 202691.22026.10.1
Nov 7, 202693.72026.10.1
Nov 6, 202693.52026.10.1
Nov 5, 202695.22026.10.1
Nov 4, 202693.92026.10.1
Nov 3, 202699.42026.10.1
Nov 2, 202688.52026.10.1
Nov 1, 202698.22026.10.1
Oct 31, 202690.52026.10.1
Oct 30, 202698.02026.10.1
Oct 29, 202693.62026.10.1
Oct 28, 202693.72026.10.1
Oct 27, 202699.62026.10.1
Oct 26, 202695.82026.10.1
Oct 25, 202696.82026.10.1
Oct 24, 202696.92026.10.1
Oct 23, 202694.92026.10.1
Oct 22, 202694.12026.10.1
Oct 21, 202691.02026.10.1
Oct 20, 202688.22026.10.1

GPT-6 Astra · OpenAI

Latest 99.8 on Nov 11, 2026 · Steady

020406080100Aug 14Nov 11
As a table
Day, UTCScoreSuite
Nov 11, 202699.82026.10.1
Nov 8, 2026100.02026.10.1
Nov 7, 2026100.02026.10.1
Nov 6, 202699.82026.10.1
Nov 5, 2026100.02026.10.1
Nov 4, 202699.82026.10.1
Nov 3, 202699.82026.10.1
Nov 2, 202697.12026.10.1
Nov 1, 2026100.02026.10.1
Oct 31, 202699.82026.10.1
Oct 30, 2026100.02026.10.1
Oct 29, 202699.82026.10.1
Oct 28, 202697.12026.10.1
Oct 27, 2026100.02026.10.1
Oct 26, 2026100.02026.10.1
Oct 25, 2026100.02026.10.1
Oct 24, 202697.12026.10.1
Oct 23, 202699.82026.10.1
Oct 22, 2026100.02026.10.1
Oct 21, 2026100.02026.10.1
Oct 20, 2026100.02026.10.1

GPT-6 Astra Ultrafast · OpenAI

Latest 99.8 on Nov 8, 2026 · Steady

020406080100Aug 14Nov 11
As a table
Day, UTCScoreSuite
Nov 8, 202699.82026.10.1
Nov 7, 2026100.02026.10.1
Nov 6, 2026100.02026.10.1
Nov 5, 2026100.02026.10.1
Nov 4, 2026100.02026.10.1
Nov 2, 202697.12026.10.1
Nov 1, 202694.12026.10.1
Oct 31, 2026100.02026.10.1
Oct 30, 202699.82026.10.1
Oct 29, 2026100.02026.10.1
Oct 28, 2026100.02026.10.1
Oct 27, 202699.82026.10.1
Oct 26, 2026100.02026.10.1
Oct 25, 2026100.02026.10.1

GPT-6 Luna · OpenAI

Latest 16.5 on Nov 11, 2026 · Higher

020406080100Aug 14Nov 11
As a table
Day, UTCScoreSuite
Nov 11, 202616.52026.10.1
Nov 8, 202614.42026.10.1
Nov 7, 202618.82026.10.1
Nov 6, 20265.92026.10.1
Nov 5, 202615.32026.10.1
Nov 4, 202617.92026.10.1
Nov 3, 20267.82026.10.1
Nov 2, 202615.32026.10.1
Nov 1, 202618.82026.10.1
Oct 31, 20268.72026.10.1
Oct 30, 202616.32026.10.1
Oct 29, 202610.92026.10.1
Oct 28, 202616.82026.10.1
Oct 27, 202626.02026.10.1
Oct 26, 202616.82026.10.1
Oct 25, 202613.12026.10.1
Oct 24, 202614.02026.10.1
Oct 23, 20264.42026.10.1
Oct 22, 202617.62026.10.1
Oct 21, 202619.92026.10.1
Oct 20, 20269.02026.10.1

GPT-6.1 Sol · OpenAI

Latest 100.0 on Nov 11, 2026 · Steady

020406080100Aug 14Nov 11
As a table
Day, UTCScoreSuite
Nov 11, 2026100.02026.10.1
Nov 8, 2026100.02026.10.1
Nov 7, 202699.82026.10.1
Nov 6, 202694.02026.10.1
Nov 5, 202699.82026.10.1
Nov 4, 2026100.02026.10.1
Nov 3, 202696.52026.10.1
Nov 2, 2026100.02026.10.1
Nov 1, 202688.22026.10.1
Oct 31, 202696.92026.10.1
Oct 30, 202699.82026.10.1
Oct 29, 202696.52026.10.1
Oct 28, 202699.62026.10.1
Oct 27, 202699.82026.10.1
Oct 26, 202699.52026.10.1
Oct 25, 202699.62026.10.1
Oct 24, 202699.82026.10.1
Oct 23, 202699.62026.10.1
Oct 22, 202697.12026.10.1
Oct 21, 202699.62026.10.1
Oct 20, 202699.82026.10.1

GPT-6.1 Sol Ultrafast · OpenAI

Latest 99.5 on Nov 8, 2026 · Steady

020406080100Aug 14Nov 11
As a table
Day, UTCScoreSuite
Nov 8, 202699.52026.10.1
Nov 7, 202697.12026.10.1
Nov 6, 202699.82026.10.1
Nov 5, 2026100.02026.10.1
Nov 4, 202699.82026.10.1
Nov 2, 202697.12026.10.1
Nov 1, 2026100.02026.10.1
Oct 31, 2026100.02026.10.1
Oct 29, 202697.12026.10.1
Oct 28, 2026100.02026.10.1
Oct 27, 2026100.02026.10.1
Oct 26, 202699.82026.10.1
Oct 25, 2026100.02026.10.1

GLM-5.3 · Z.ai

Latest 71.4 on Nov 11, 2026 · Lower

020406080100Aug 14Nov 11
As a table
Day, UTCScoreSuite
Nov 11, 202671.42026.10.1
Nov 10, 202685.72026.10.1
Nov 9, 202695.02026.10.1
Nov 8, 202686.52026.10.1
Nov 7, 202675.02026.10.1
Nov 6, 202691.72026.10.1
Nov 5, 202678.22026.10.1
Nov 4, 202685.82026.10.1
Nov 2, 202685.62026.10.1
Nov 1, 202679.92026.10.1
Oct 31, 202687.82026.10.1
Oct 30, 202680.12026.10.1
Oct 28, 202673.82026.10.1
Oct 27, 202686.42026.10.1
Oct 26, 202691.32026.10.1

Speed

Daily response time by model from our local probe, scheduled at 00:10 UTC. Each point is the median completion time of three successful fresh single-turn responses to one fixed small task, at high reasoning effort and Standard/default service, or the Ultrafast service tier for the two Ultrafast lines. Dates are UTC; a model without three successful responses that day has no point.

  • GPT-6.1 Sol
  • GPT-6 Astra
  • GPT-6 Luna
  • Opus 5.5
  • Fable 5.1
  • Sonnet 5.5
  • GPT-5.6 Sol
  • GPT-6.1 Sol Ultrafast
  • GPT-6 Astra Ultrafast
  • GPT-5.6 Luna
  • Haiku 5.5
  • Kimi K3.1 Pro
  • GLM-5.3
  • DeepSeek V4 Pro
  • Fable 5.2

Drag across the chart to zoom in.Pinch to zoom; drag with two fingers to move.

010s20s30sSep 30Nov 11

Output speed

Tokens per second from the daily Nerf Watch run, scheduled at 01:00 UTC: total output tokens divided by total completion time across its four fixed tasks. Effort is low, except DeepSeek V4 Pro uses high, its lowest available variant. Service is Standard/default, or Ultrafast for the two Ultrafast lines. Output includes reported reasoning or thinking tokens once; OpenCode’s separately reported reasoning is added to output. Timing includes waiting within the measured turn and excludes setup and process exit. Higher is faster; dates are UTC.

Drag across the chart to zoom in.Pinch to zoom; drag with two fingers to move.

0100200300Oct 25Nov 11

How it works

What
Trick questions, exact counting and sorting, writing to letter-level rules, and building a small working app.
How
Each kind of test runs in a fresh session. Code does all the grading, and the app is tested in a real browser. No AI model judges another.
Where
Through Codex, Claude Code and the OpenCode CLI, signed in to paid subscriptions rather than using provider API keys.
What a drop means
A score below the model's own baseline, the mean of its recent runs. One day's dip can be noise, and a lower score is a lower score on this suite, not proof that a provider changed a model.
Evidence
When a model's score moves more than 10 points from its previous run, a marker on its chart lets you ask for its answers, on both days, to one question: the one that changed most within the kind of test that changed most. We email you a link that works for 7 days. You see the answers, never the question. Sorting and letter-rule answers also show the line the grader printed when it checked them.

Read the full method (English)