The Data Wall: Exhausting human written words.

Epoch AI projects that high-quality human-written text data on the public internet was exhausted before 2026. Frontier models are already overtrained by 5× compared to standard compute-optimal levels. Furthermore, publishers are actively withholding access: the MIT Data Provenance Initiative documented a sharp contraction in crawlable content. Cloudflare data shows AI crawler growth slowed from 32% in April 2025 to just 4% in July 2025 as publisher blocking surged, highlighted by Anthropic's lopsided 38,000:1 crawl-to-refer traffic ratio^[37].

As pre-training data hits physical limits, scaling laws have shifted to inference runtime. Reasoning models spend massive token budgets to search and self-correct. OpenAI's GPT-5.6 Sol features a 'max' reasoning setting requiring over 700,000 GPU-hours of test-time search and evaluation compute to verify complex paths, raising active inference costs significantly.^[?][38]. This has shifted the economic balance: a single query can cost 4–17× more in compute and latency, driving Chinese daily token call volumes to over 140 trillion tokens in Q1 2026, making inference the dominant infrastructure cost^[40].

Hyperscalers are training models on synthetic data. This includes Reinforcement Learning from AI Feedback (RLAIF) under Constitutional AI ^[10], self-play bootstrapping (e.g. DeepSeek-R1-Zero using pure RL to generate reasoning traces, which then initialize SFT and distillation pipelines ^[?]), and sandbox environments that provide ground-truth compilation feedback.

Recursively training models on synthetic data from prior models introduces 'model collapse,' where semantic variance degrades. To mitigate this, labs are combining synthetic generation with ground-truth verification loops: executing generated code in sandboxes to verify syntax, checking mathematical steps with formal proof assistants (Lean), and comparing output trajectories with physical simulators^[37].