What's inside AI training data?

One mountain of text, two questions

An AI learns from a mountain of text scraped from the web — its training data. Two questions follow. Did your words end up in the mountain? And were the exam questions the AI is graded on — its benchmarks — in there too, so the score measures memory rather than ability (that is contamination)? Ingot answers both the same way: it finds the exact overlapping words and shows them to you.

• Pick the exam. Three public benchmarks live in the menu — maths, code, general knowledge.
• Drop the study material. Read here, in this tab — nothing is uploaded anywhere.
• Read the overlaps. Every match shown with the words around it. Some overlap is innocent — that is why you see the words, not a score.

The scan that ran when you arrived was real — the built-in sample, read in this tab. Watch it work replays it step by step, or press play for the film.

Your corpus never leaves the machine

Not a policy — an architecture. There is no upload path: your file is read, hashed and scanned inside this tab. Open the network panel and watch it stay empty.

Never dropping a confidential corpus in a browser? Same scanner, same numbers, on your terminal: npx ingot-scan contaminate --index gsm8k --corpus your-corpus.jsonl

Index the benchmark, stream the corpus once

Ingot memorises the exam — every ten-word run it contains — then reads your file once, checking each ten-word run that streams past. One pass, no matter how many questions. lm-evaluation-harness indexes the corpus instead, and reports nine days on the Pile.

  7.0 MB/sec single-threaded, end to end  →  20 GB in about 48 minutes

  21.4 GB of gzipped C4 shards on disk · GSM8K at n=10 · 51 minutes wall clock
  decompression, JSON.parse, hashing and match bookkeeping all included
  reproduce with: node scripts/pretraining-scan.ts

The field's 13-gram default is wrong

How many identical words in a row should count as a match? Everyone uses 13, because GPT-3 did in 2020 — and we could find no re-derivation since. So we ran 8 through 13 over a 1,000-item benchmark with 300 planted copies:

n=13 leaves 6.8% of the benchmark unscannable — items shorter than 13 tokens can never match at all. At n=10 that is 0.3%, edited copies are caught at 81.5% against 69.6%, and the cost is two false positives per thousand that you can read and dismiss in a second.

Three benchmarks against 21 GB of C4

We ran the scanner ourselves: three public exams against C4, a public snapshot of web text. Every shard's URL and SHA-256 is published, so every number reproduces.

Corpus

c4-en26 shards, Common Crawl

Size

21.33 GB9,264,249 documents

Tokens

3,418,685,530one pass, streamed

Method

n = 10exact, ingot-0.1.0

"2.4% of MMLU is contaminated in C4" is wrong, and that is the finding. We inspected 200 retained matches and found no contamination — we found NATO Article 5, Patrick Henry, the Fifth Amendment: canonical text, the famous passages the whole world quotes. MMLU quotes famous documents; web text contains them. The overlap is exact, reproducible and evidentially worthless.

A false positive you can see is a judgment you make in one second; as a rate, it is one you cannot evaluate at all. The full write-up, including the two published numbers this project had to retract, is in three ways contamination scanning silently fails.

What Ingot refuses to do

Each of these was a defect first, caught by reading the output instead of trusting it. All eleven are written up in docs/measurements.md.

• No silent clean result. Every report names the items nothing could ever match.
• No verdict without the text. Ingot reports what overlaps; whether that is leakage is your call.
• No paraphrase in the headline. Weaker evidence sits in its own labelled section, outside the count.
• No unmeasured claims. The false-positive rate prints beside the detection rate, from the same run.