Which benchmarks appear in which training corpora

The pretraining pass — three benchmarks against 21.33 GB of C4

C4 is a public snapshot of web text — Common Crawl from April 2019, cleaned — the kind of corpus models are actually pretrained on. Every benchmark below was assembled after that snapshot, so a match here can never mean the exam leaked into this pile. What a match shows is the reverse: the text posing a test item already existed on the public web before the benchmark did.

Corpus

c4-en-26shards26 shards, Common Crawl, ODC-BY

Size

21.33 GB9,264,249 documents

Tokens

3,418,685,530one pass, streamed

Method

n = 10exact, ingot-0.1.0

Corpus hash

c046ce3c890918b4d68094c438c05321same bytes, same number

“2.4% of MMLU is contaminated in C4” is wrong, and that is the finding. The 25 retained MMLU matches were inspected and none is contamination: they are NATO Article 5, Patrick Henry, the Fifth Amendment — canonical text, the famous passages the whole world quotes. MMLU quotes famous documents; web text contains them. The overlap is exact, reproducible and evidentially worthless — and a scanner that printed the rate without the words would have called it a scandal.

GSM8K's two matches are everyday phrasings each found in a single document, and HumanEval's are digit runs and a textbook prime definition — the number sequences every explanation of numbers contains. Judge all of it yourself:

gsm8k — 2 matches, 2 retained under the evidence cap, 2 shown — the full retained set is in results/pretraining-c4.json

humaneval — 15 matches, 15 retained under the evidence cap, 4 shown — the full retained set is in results/pretraining-c4.json

mmlu — 865 matches, 25 retained under the evidence cap, 6 shown — the full retained set is in results/pretraining-c4.json

Canonical or leaked — the cross-provenance test

A passage that reached two corpora by genuinely different routes is a property of the language, not evidence a benchmark leaked. So the flagged MMLU items were scanned again — against The Pile with every web-crawl subset removed: 251,488 documents of court opinions, patents, paper abstracts and digitised books, routes a crawler was never involved in.

67 of the 342 flagged items are confirmed canonical — their text lives in non-crawl print, most of it in court opinions. The other 275 are undetermined, not leaked: a control set of nine unmistakably canonical items scored 1 of 9 against this reference, so absence from it carries no information. Presence is evidence; absence is a lead.

• freelaw confirmed 46 items
• gutenberg-pg-19 confirmed 3 items
• wikipedia-en confirmed 2 items
• arxiv confirmed 2 items
• stackexchange confirmed 2 items
• pubmed-central confirmed 2 items
• pubmed-abstracts confirmed 1 item

The distillation pass — the first corpus where leakage was possible

SlimOrca is 517,982 GPT-4 completions over FLAN prompts, assembled in 2023 — after every benchmark here was published. The leak window was open, so for the first time a match could be more than canonical text. Every flagged item was therefore measured rather than eyeballed: the longest verbatim run between the item and each document that matched it, published beside the scan.

Corpus

slimorcaMIT · derived — both SHA-256s in the manifest

Size

0.92 GB517,982 documents · 0 lines skipped

Tokens

150,699,944one pass, streamed

Method

n = 10 and 13exact, ingot-0.1.0

Corpus hash

e99263650d2340c89aa50b7367100224same bytes, same number

GSM8K's train split is inside this corpus; its test split is not. 4 of 4 flagged test questions are template siblings of train-split questions that appear verbatim in the very same documents — and 0 test questions appear verbatim themselves. Across all three benchmarks the triage reads 0 leaked · 2 partial · 34 phrase-level, and the longest verbatim run anywhere is 31 tokens of a 227-token item.

A scanner that stopped at the flag would have called this a test-set leak. The words say otherwise — and they also say the fine-tuning set carries the training questions, which is worth knowing before quoting a GSM8K number for any model tuned on it.

The instruction-tuning pass at n=10, the Ingot default

In plain words: n=10 means 10 identical words in a row is what counts as a match, and an unscannable item is one too short to ever produce a match at that length — named, never silently skipped.

3 benchmarks against 2 corpora. 6 distinct benchmark items flagged out of 15,525, appearing as 7 rows below because an item found in two corpora is one finding, not two.

Results at n=13

The field's default since GPT-3, reported so findings stay comparable with prior published work. It flags less and leaves far more of each benchmark unscannable.

Evidence from the instruction-tuning pass

Every finding below is a verbatim match. Judge them yourself — some overlaps are canonical facts with one natural phrasing rather than contamination, and the only way to tell is to read the text.

humaneval in alpaca-52k

humaneval in dolly-15k

mmlu in alpaca-52k

What these findings actually are

All 6 instruction-set items were inspected and none is contamination. Every one is canonical text: a prime or digit sequence inside a HumanEval docstring, the proper noun "I Have a Dream speech", a stock definition of capitalism. They match because that text is everywhere, not because a benchmark leaked.

That is published rather than quietly dropped, because it is the finding. A phrase appearing in exactly one benchmark item still carries no evidential weight if the whole world writes it. A corpus-side document-frequency filter was added and did not fire — these phrases sit in two or three documents of a 26k corpus, below any sane threshold, while still being ubiquitous in English. At C4 scale the same filter discarded 757 matches — it only engages once the corpus is enormous, which is why the pretraining pass exists.

Limitations, stated first

• The pretraining pass covers 26 of 1,024 C4 shards — about 2.5% of one snapshot of one corpus. Every rate is measured on exactly those shards and extrapolates to nothing.
• Every benchmark postdates the April 2019 crawl, so no match here is the exam leaking into this pile. A match means the item's posing text existed on the public web before the benchmark was assembled.
• The pretraining pass has run at n=10 only; the n=13 comparability pass is not yet published.
• SlimOrca is a derived corpus — ShareGPT turns flattened to text by scripts/fetch-sft.ts. The manifest records the source file's SHA-256 beside the derived file's, so the derivation is checkable end to end.
• The train-versus-test sibling check was run for GSM8K only; the other benchmarks' flagged items were triaged by measured run length, not against a train split.
• The instruction-tuning corpora are 2023-era fine-tuning sets of a few million tokens. A null result there says almost nothing about pretraining — which is why the pretraining pass exists.
• Exact matching only. Paraphrased contamination is not counted and is invisible to this pass.
• Unscannable items are benchmark items too short to produce any n-gram. Nothing can ever match them, so they are reported rather than folded into a clean result.
• A vendor can run this on their own corpus and edit until nothing matches. See the threat model.