Proof
The scanner is running on this page right now
It read a corpus in this tab, on your CPU, and nothing left the machine. The same engine runs your check. Evaluating a model instead? Drop your own training data below — any size, it streams.
JSONL — one {"text": …} record per line. Plain or gzipped —
any size, it streams.
One mountain of text, two questions
An AI learns from a mountain of text scraped from the web — its training data. Two questions follow. Did your words end up in the mountain? And were the exam questions the AI is graded on — its benchmarks — in there too, so the score measures memory rather than ability (that is contamination)? Ingot answers both the same way: it finds the exact overlapping words and shows them to you.
- Pick the exam. Three public benchmarks live in the menu — maths, code, general knowledge.
- Drop the study material. Read here, in this tab — nothing is uploaded anywhere.
- Read the overlaps. Every match shown with the words around it. Some overlap is innocent — that is why you see the words, not a score.
The scan that ran when you arrived was real — the built-in sample, read in this tab. Watch it work replays it step by step, or press play for the film.
six words this page keeps using
| term | plain meaning |
|---|---|
| benchmark | a standard exam for AI models |
| training data · corpus | the pile of text a model studied |
| token | roughly one word |
| n = 10 | ten identical words in a row is what counts as a match |
| contamination | exam questions found inside the study pile |
| canonical | famous text the whole world quotes — it matches, but it is not leakage |
Your corpus never leaves the machine
Not a policy — an architecture. There is no upload path: your file is read, hashed and scanned inside this tab. Open the network panel and watch it stay empty.
connect-src 'self' — the browser itself refuses to send data anywhere else. Check the response headers. Or pick the exam, go offline, and scan.Never dropping a confidential corpus in a browser? Same
scanner, same numbers, on your terminal:
npx ingot-scan contaminate --index gsm8k --corpus your-corpus.jsonl
Index the benchmark, stream the corpus once
Ingot memorises the exam — every ten-word run it contains — then reads your file once, checking each ten-word run that streams past. One pass, no matter how many questions. lm-evaluation-harness indexes the corpus instead, and reports nine days on the Pile.
7.0 MB/sec single-threaded, end to end → 20 GB in about 48 minutes 21.4 GB of gzipped C4 shards on disk · GSM8K at n=10 · 51 minutes wall clock decompression, JSON.parse, hashing and match bookkeeping all included reproduce with: node scripts/pretraining-scan.ts
Why this number replaced a faster one
This said 37.1 MB/sec, 20 GB in 9.0 minutes until we scanned a real corpus. That figure is the scan kernel — tokenize, roll, look up — over text already read and already parsed, and quoting it as a scanning rate dropped most of the actual work. The kernel number is still the right one for judging optimisation; it was the wrong one to put here.
The field's 13-gram default is wrong
How many identical words in a row should count as a match? Everyone uses 13, because GPT-3 did in 2020 — and we could find no re-derivation since. So we ran 8 through 13 over a 1,000-item benchmark with 300 planted copies:
| n | verbatim | edited copies | false positives | unscannable |
|---|---|---|---|---|
| 8 | 100% | 90.9% | 5 / 1000 | 2 / 1000 |
| 9 | 100% | 85.9% | 3 / 1000 | 2 / 1000 |
| 10 | 100% | 81.5% | 2 / 1000 | 3 / 1000 |
| 11 | 100% | 79.7% | 1 / 1000 | 31 / 1000 |
| 12 | 100% | 75.0% | 0 | 57 / 1000 |
| 13 | 100% | 69.6% | 0 | 68 / 1000 |
n=13 leaves 6.8% of the benchmark unscannable — items shorter than 13 tokens can never match at all. At n=10 that is 0.3%, edited copies are caught at 81.5% against 69.6%, and the cost is two false positives per thousand that you can read and dismiss in a second.
The standard error, a retraction, and the caveats
The 81.5% against 69.6% gap is real at 300 planted items, where the standard
error is about 2.3 points. An earlier version of this table claimed 100% against 3.5% — an
artifact of a test fixture that deleted words on a fixed lattice, which decided in advance
which values of n could survive. It is written up in docs/measurements.md,
because a tool that audits other people's numbers cannot quietly restate its own.
Caveats: one corpus pair, both 2023-era, and edited copies simulated by random word dropping rather than a model rewriting text. Real paraphrase will be harder.
Three benchmarks against 21 GB of C4
We ran the scanner ourselves: three public exams against C4, a public snapshot of web text. Every shard's URL and SHA-256 is published, so every number reproduces.
- Corpus
- c4-en26 shards, Common Crawl
- Size
- 21.33 GB9,264,249 documents
- Tokens
- 3,418,685,530one pass, streamed
- Method
- n = 10exact, ingot-0.1.0
| benchmark | flagged | rate | matches | discarded | unscannable |
|---|---|---|---|---|---|
| gsm8k | 2 / 1319 | 0.152% | 2 | 0 | 0 |
| humaneval | 6 / 164 | 3.659% | 15 | 115 | 0 |
| mmlu | 342 / 14042 | 2.436% | 865 | 642 | 28 |
“2.4% of MMLU is contaminated in C4” is wrong, and that is the finding.
We inspected 200 retained matches and found no contamination — we found NATO Article 5, Patrick Henry, the Fifth Amendment: canonical text, the famous passages the whole world quotes. MMLU quotes famous documents; web text contains them. The overlap is exact, reproducible and evidentially worthless.
A false positive you can see is a judgment you make in one second; as a rate, it is one you cannot evaluate at all. The full write-up, including the two published numbers this project had to retract, is in three ways contamination scanning silently fails.
How we ruled contamination out — two tests
The standard defence does not save you. Document-frequency filtering discarded
757 matches at this scale and 0 at 356,317 documents — same filter, same
threshold, so it only engages once the corpus is enormous. Even then the survivors are canonical,
because a famous passage's exact ten-gram is rare: Patrick Henry's most quoted sentence
appears in one document out of nine million, as
give me liberty or give me death!?- Patrick Henry.
Tested properly against a corpus of different provenance — The Pile with every web-crawl subset removed, 251,488 documents of court opinions, patents, abstracts and digitised books — 67 of the 342 are confirmed canonical, 46 of them by court opinions. The other 275 are undetermined, not leaked: a control of nine unmistakably canonical items scored 1 of 9 against that reference, so absence from it carries no information. Presence is evidence; absence is a lead.
What Ingot refuses to do
Each of these was a defect first — caught by reading the output instead of
trusting it, or by running the tool the way a stranger would.
All 20 are written up in docs/measurements.md.
- No silent clean result. Every report names the items nothing could ever match.
- No verdict without the text. Ingot reports what overlaps; whether that is leakage is your call.
- No paraphrase in the headline. Weaker evidence sits in its own labelled section, outside the count.
- No unmeasured claims. The false-positive rate prints beside the detection rate, from the same run.
What a clean result does not prove
Canonical is not the same as harmless. A test item written from a public web page can sit in a training corpus as ordinary web text: nobody leaked it, and a model trained on that corpus still saw it. The canonicality label says where text came from, not that seeing it was free.
Questions are indexed; answers are not. Published indexes exclude answer text, so a corpus that reproduces every solution while paraphrasing its questions scans clean.
“Verbatim” means after normalization. Matching runs on lowercased tokens with punctuation stripped. For prose the difference is cosmetic; for code it is not — a HumanEval match is a run of identifiers and words, with operators and structure invisible.