AI Model Collapse and What Clean Pre-2022 Text Now Costs

By generation nine the model was writing about jackrabbits. Black-tailed, white-tailed, blue-tailed, red-tailed, yellow-tailed. The prompt had been about the design of English parish church towers.

That output is printed as Example 1 in the paper that made AI model collapse a public argument: Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Nicolas Papernot, Ross Anderson and Yarin Gal, Nature volume 631, pages 755 to 759, published 25 July 2024. Received 20 October 2023. Accepted 14 May 2024. The finding travelled fast and got flattened on the way. What most people took from it is that AI eats itself. What the paper shows is narrower and considerably more useful, and the narrow version is the one that explains why a newspaper archive from 2009 now has a price.

Both things are true. The mechanism is real. The doom reading is wrong. The gap between them is where the money went.

The AI model collapse experiment, exactly as run

OPT-125m. Meta's smallest open causal language model, fine-tuned on wikitext2. Five-way beam search, training sequences blocked at 64 tokens, the model asked to predict the next 64 tokens for every block in the training set, producing an artificial dataset the same size as the original. Five random seeds. That is the whole rig.

The base fine-tune reached 34 mean perplexity against a zero-shot baseline of 115, so it learned the task properly. Then two regimes. In the first, five epochs, with no original data retained for any subsequent generation. Performance drifted from 20 to 28 perplexity points across the generations. Degradation, not annihilation. The models still learned the underlying task, badly.

The qualitative failure is the part that matters. Tails go first. The authors split it into early model collapse, where the model loses information about the low-probability edges of the distribution, and late model collapse, where the modes entangle and the output converges toward something with very small variance that carries little resemblance to the original. Nine rounds of that gets you coloured jackrabbits.

A 2025 Author Correction to the Nature article fixed a subscript in the theoretical intuition section, alpha where beta was meant. The experiments stood.

The asset was never the writing. It is the timestamp, and specifically a timestamp somebody will indemnify.

The ten percent that stops it

The second regime in the same paper is the one almost nobody quoted. Ten epochs, and at every new generation a random 10% of the original data points is sampled back in. The authors report only minor degradation of performance.

Ten percent. That is the entire intervention.

Matthias Gerstgrasser, Rylan Schaeffer and colleagues formalised it in April 2024. Replace real data with each generation's synthetic output and you do trend toward collapse. Accumulate instead, keeping the real data and adding synthetic on top, and the test error carries a finite upper bound independent of the number of iterations. They demonstrate it across language, molecular and image models. Collapse is a property of a replacement loop. It is not a property of synthetic data.

In March 2025 Schaeffer, Joshua Kazdan, Alvan Caleb Arulandu and Sanmi Koyejo hand-annotated 28 papers on the subject and counted eight distinct definitions of model collapse in active use, falling into three incompatible families: the behaviour of test loss over fitting iterations, the deformation of the real data distribution and the scaling behaviour of test loss. People arguing about model collapse are frequently not arguing about the same quantity.

One place the tails argument is not academic: low-resource languages. Devon Jarvis and co-authors argued in May 2026 that collapse lands hardest on communities whose data was thin to begin with, and I think that is the most credible version of the worry. For English, the tail is rare vocabulary. For Igbo or Nigerian Pidgin, the tail is most of the language. Web text in those languages was already scarce and is now heavily machine-translated. The 10% real fraction that rescues English may simply not exist to sample from.

The strongest case against any of this mattering

Microsoft released Phi-4 in December 2024, 14 billion parameters, trained on roughly 400 billion tokens of synthetic data spanning about 50 dataset types, each with its own seed set and multi-stage prompting procedure. The technical report is blunt: synthetic data is not a cheap substitute for organic data, it has direct advantages over it. Phi-4 beat much larger models on mathematical reasoning.

Alibaba's Qwen3, May 2025, was pre-trained on 36 trillion tokens across 119 languages and dialects, with synthetic content generated by domain-specific models including Qwen2.5-Math. And the systematic study by Feiyang Kang and co-authors at EMNLP 2025 found that mixing roughly one third rephrased synthetic data with two thirds natural web text speeds up pre-training by five to ten times at large data budgets, with the useful synthetic share converging near 30%.

So the strongest models use a great deal of it and it works. That is not a weak objection. It is the correct one.

Here is what it does not do. It does not remove the demand for verified human text. It prices it.

Look at what those pipelines are anchored to. Phi-4's synthetic sets are seeded from filtered organic documents, public web material and educational sources. Qwen3's largest single data gain came from fine-tuning Qwen2.5-VL to read PDF-like documents and refining the recognised text with Qwen2.5, adding trillions of tokens. That is transcription of real paper, not generation. And the same EMNLP study reports that textbook-style purely generated data on its own produces notably higher loss across many downstream domains, in patterns consistent with model collapse, while generator models above roughly 8 billion parameters do not produce better pre-training data than smaller ones.

Thirty percent is a ratio. A ratio needs a denominator. The denominator is real text, and the question stopped being whether synthetic data is bad. It became how much verified human text you have to keep in the mixture, which is a procurement question with a line item attached.

Provenance is a data engineering problem before it is a philosophical one

Crawl date is not authorship date. Common Crawl gives you fetch time. HTML gives you whatever the CMS felt like emitting. Republication, syndication, translation, template migration and scraper resurfacing all reset the visible date. If you want a corpus you can honestly certify as pre-2022, you end up building a provenance column, and it tends to look the same everywhere: registrable domain, earliest independent sighting such as a Wayback capture or a DOI or a print ISSN or a git commit hash, licence status and a detector score you do not trust very much.

Detectors are the weak leg. Graphite's October 2025 study classified 65,000 English articles published between January 2020 and May 2025 and reported the AI-generated share rising from 2.2% to 51.7%, crossing human volume around November 2024 and then flattening. It used a commercial detector at a 50% threshold. Take the trend, not the decimal.

China attacked the problem from the regulatory end. The Cyberspace Administration published the Measures for Labeling AI-Generated Synthetic Content on 14 March 2025 alongside the mandatory national standard GB 45438-2025, both in force 1 September 2025. Two obligations: explicit labels a user can see, and implicit labels a machine can read, carrying provider code, a content identifier and a generation timestamp written into file metadata.

I read the standard in Chinese the week it landed, and the visible watermark was never the interesting part. The header field was. Then watch what happens on domestic platforms once labelled content starts circulating: the visible tag survives because it is burned into pixels, and the header metadata does not survive a re-encode, a screenshot or a copy-paste into a text box. A provenance scheme that dies on first re-upload is a labelling scheme.

Which is the honest state of the field. Nobody has a durable, adversarially survivable way to mark text as machine-written. Shumailov's team said so themselves in the Nature discussion: it is unclear how LLM-generated content can be tracked at scale, and the alternative they name is community-wide coordination on provenance or nothing.

What clean text costs, with dates

The market did not wait for the science to settle.

Axel Springer and OpenAI, December 2023, three years, non-exclusive, reported in the tens of millions of euros. Reddit and Google, announced 21 February 2024, reported at roughly $60 million a year. OpenAI licensing the Financial Times archive on 29 April 2024. News Corp and OpenAI in May 2024, reported by the Wall Street Journal at more than $250 million over five years. Taylor & Francis to Microsoft at a reported $10 million in the first year, with academics objecting publicly that authors were given no opt-out. Wiley disclosing $23 million and then a further $21 million in a single year. Amazon and the New York Times in May 2025, reported at $20 million to $25 million a year, the paper's first AI licence.

Then the infrastructure caught up with the lawyers. On 1 July 2025 Cloudflare began blocking AI crawlers by default for new domains and opened a pay-per-crawl beta, flipping roughly a fifth of the web from opt-out to opt-in. On 10 September 2025 the Really Simple Licensing standard launched with Reddit, Yahoo, Quora, Medium, Ziff Davis and O'Reilly behind it, defining machine-readable terms that include pay per crawl and pay per inference.

And the punitive number. Anthropic agreed to pay roughly $1.5 billion, about $3,000 a work, to settle a class action over pirated books, preliminarily approved on 25 September 2025. Judge Alsup had held in June 2025 that training on lawfully acquired books was fair use and that sourcing them from pirate libraries was not. Read together, those two holdings do not say text is off limits. They say provenance is the billable thing.

Notice the shape change. The 2023 and 2024 agreements bought back catalogues, a fixed archive delivered once. The newer ones increasingly rent a live feed. Different products, different logic. The archive is bought because it is old and clean. The feed is rented because it is new.

The archives nobody is selling

EleutherAI released Common Pile v0.1 on 5 June 2025: 8TB of public domain and openly licensed text drawn from 30 sources, about two years of work, leaning on Library of Congress and Internet Archive digitisations. They trained Comma v0.1 at 7B on 1 trillion and 2 trillion tokens and reported parity with LLaMA 1 and 2 class models built on unlicensed data.

Harvard's Institutional Books 1.0 arrived the same month: 983,004 public domain volumes, roughly 242 billion tokens, more than 300 languages, out of Harvard Library's participation in Google Books starting in 2006. Ai2 shipped Olmo 3 on 20 November 2025 on Dolma 3, a mix of about 5.9 trillion tokens, with OlmoTrace for tracing generated text back to training data. Weights and data both.

And the loss column. Robyn Speer stopped updating wordfreq in September 2024, writing that the web is now full of text written by no one to communicate nothing. wordfreq covered more than 40 languages. It was a measuring instrument, retired because what it measured got contaminated. The catalogue at lowbackgroundsteel.ai indexes what is left, including Library of Congress newspapers from 1730 to 1969 and the English Wikipedia dump from August 2022. The name is borrowed from steel smelted before the 1945 atmospheric tests, the only steel clean enough for sensitive radiation instruments.

The asset has a shelf life

Pre-2022 text appreciates as a commodity and depreciates as knowledge at the same time. A corpus certified clean is also certified ignorant of everything since. It has never seen a transformer inference stack, a 2024 election, a 2026 tariff schedule or the word Qwen. You cannot build a current model out of it. You can only use it as ballast, the fraction that keeps the mixture anchored while everything else in the training set gets younger and more synthetic each quarter.

So what is being priced is not the writing. It is the timestamp, and specifically a timestamp somebody will indemnify. Which produces the obvious problem. The moment a provenance claim is worth $250 million over five years it is worth forging, and there is no cryptographic anchor underneath most of the web to forge against. The 2009 archive is trustworthy precisely because nobody at the time was trying to prove anything about it.

Tools referenced

Ai2 Olmo 3, reviewed here: Ai2 Olmo 3 review.

MinerU, reviewed here: MinerU review.

PaddleOCR, reviewed here: PaddleOCR review.

dots.ocr, reviewed here: dots.ocr review.

Label Studio, reviewed here: Label Studio review.

Sources

Shumailov et al., AI models collapse when trained on recursively generated data, Nature 631, 755-759 (25 July 2024): https://www.nature.com/articles/s41586-024-07566-y

Gerstgrasser et al., Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data (April 2024): https://arxiv.org/abs/2404.01413

Schaeffer, Kazdan, Arulandu, Koyejo, Position: Model Collapse Does Not Mean What You Think (March 2025): https://arxiv.org/abs/2503.03150

Kang et al., Demystifying Synthetic Data in LLM Pre-training, EMNLP 2025: https://arxiv.org/abs/2510.01631

Digiday, 2024 in review: a timeline of the major deals between publishers and AI companies: https://digiday.com/media/2024-in-review-a-timeline-of-the-major-deals-between-publishers-and-ai-companies/

Cloudflare, Introducing pay per crawl (1 July 2025): https://blog.cloudflare.com/introducing-pay-per-crawl/

EleutherAI, The Common Pile v0.1 (5 June 2025): https://blog.eleuther.ai/common-pile/

Graphite, More Articles Are Now Created by AI Than Humans (October 2025): https://graphite.io/five-percent/more-articles-are-now-created-by-ai-than-humans