AI model parameters grew 24,000-fold, then went sparse

Total parameters grew faster than the count active per token

The chart separates two quantities that once moved together. Its step line follows the published dense-model record, ending at PaLM’s 540B in April 2022. Paired dots show total and active parameters for five selected mixtures of experts, from GLaM to Kimi K3.

A sparse model can hold far more parameters than it uses for one token. That makes total size a poor substitute for per-token compute: Kimi K3 holds about 5.2 times PaLM’s parameters while activating about 5.2 times fewer. The comparison concerns parameters, not a measured speed or cost advantage.

PaLM’s 540B remains the disclosed dense record

117M became 540B in less than four years. GPT-2 reached a headline 1.5B in February 2019; Megatron-LM, T5 and Turing-NLG followed at 8.3B, 11B and 17B. GPT-3 reached 175B in May 2020, roughly 117 times GPT-2’s headline count in 15 months. MT-NLG reached 530B in October 2021, and PaLM reached 540B the following April.

Across the GPT-1–PaLM endpoints, that is about 4,615-fold growth, or roughly ninefold a year. The starting count needs a qualification: GPT-1’s announcement gave no parameter total. Its commonly cited 117M comes through the GPT-2 paper’s comparison with the original GPT; the report therefore rates it medium confidence.

Chinchilla showed why more parameters were not the only route forward. Its 70B model trained on 1.4T tokens with Gopher’s training compute and outperformed the 280B Gopher. PaLM arrived six days later, but no larger dense language model was disclosed afterward in the report’s window. PaLM-E’s 562B includes a vision extension and is outside that comparison.

Kimi K3 holds 2.8T parameters and activates 104B

1.2T total and 96.6B active: Google’s GLaM made the distinction explicit in 2021. Each token used about 8% of the model, and the paper reported about half GPT-3’s inference FLOPs per token. DeepSeek-V3 later used 37B of 671B parameters, about 5.5%.

The newer large MoEs use roughly 3–4% of their totals for each token. Kimi K2 uses 32B of 1T, DeepSeek-V4-Pro 49B of 1.6T, Kimi K3 104B of 2.8T, and Qwen3.8 95B of 2.4T. Kimi K3 routes each token to 16 of 896 experts, alongside shared experts.

Active counts also require provenance. Grok 2’s roughly 115B active count is derived from released weights and its configuration, rather than published as a size by xAI. Its always-on dense residual component helps explain why its active share is much higher.

The largest model-card count comes from open weights

2.8T is the largest total written into a model card in the report, for Moonshot AI’s Kimi K3. It was announced on July 16, 2026, with weights public by July 27. DeepSeek-V3 to Kimi K3 is about 4.2-fold growth in 19 months.

The report’s open decoder-model sequence passes through OPT-175B, BLOOM, Grok-1, Snowflake Arctic, DeepSeek-V3, Kimi K2 and DeepSeek-V4-Pro. Switch-C is a separate case: its 1.6T research checkpoint was open, but it is an encoder–decoder trained for span corruption, outside that decoder-model sequence.

Rounded headlines can also create a record that disappears with more precise counts. Yuan3.0 Ultra’s 1.01T exceeds Kimi K2’s 1T headline, while a later paper puts K2’s shared backbone at 1.04T. The table preserves that qualification.

Thinking Machines Lab’s Inkling has a 975B headline, with stored weights summing to about 952.4B; the difference is unexplained. Mistral Large 4’s announced 1T did not yet make it an open-weight record: its weights were still pending.

GPT-4’s 1.8T is a leak, not a lab disclosure

1.8T total and roughly 280B active is the report’s reconciled GPT-4 estimate, based on SemiAnalysis’s leak reporting and corroborating accounts. OpenAI’s technical report withheld model size. The estimate has medium confidence; corroboration does not turn it into an official count.

Serving speed and API prices underpin another class of estimate. Epoch’s Ege Erdil put GPT-4o at roughly 200B and Claude 3.5 Sonnet at roughly 400B, warning that the GPT-4o figure could be wrong by a factor of two. Serving speed bears directly on active compute; an inferred total also depends on assumed sparsity.

Knowledge probes infer size from factual recall, calibrated against 93 open models. The IKP study’s 90% prediction bands span about 3.2 times either way, and heavily safety-tuned models can yield lower bounds. Its revised GPT-5 estimate fell from about 4.1T to 2.0T; GPT-5.5 fell from about 9.7T to 4.7T. These remain low-confidence inferences, not measured weight counts.

Other evidence needs its own label. Musk’s 3T claim for Grok 3 and Grok 4 is a CEO statement, carried by an unverified transcription in the report. The roughly 8T Mythos 5 and 5T Fable 5 figures are Financial Times industry estimates relayed by Reuters, not Anthropic disclosures. None belongs on the chart as a verified size record.

How we compiled this

The source report reconciles 118 models from more than 30 labs. It classifies 78 totals as disclosed, including 15 with medium confidence, and 40 as third-party estimates, including 35 with low confidence. The tables here select models that explain dense growth, sparse activation and the different evidence behind closed-model figures.

Dates normally mark the earliest public announcement naming a model, rather than the day its size or weights became public. Weight status is frozen at the cutoff; open weights include restrictive and non-commercial licences. Announcement dates and later releases therefore remain separate in the notes.

We retain lab headline counts and flag material differences from stored weights. Estimates keep their source, method, confidence and available ranges; an unknown active count stays unknown. The report’s withdrawn, unchecked and unreleased training-target rows are excluded here, as are models with no usable total. A closed model’s undisclosed architecture is not established merely because the source CSV assigns it an MoE category.