How it works
Dataset Doctor
The worst data problems don't crash anything. They make a model look excellent in testing and fail in use. Below are the six ideas that let Dataset Doctor find them with plain statistics, each with the few lines of code that carry it.
- Data quality for ML
- Statistics
- Python ↔ JavaScript parity
- Runs in the browser
One analysis
- Readevery cell kept as the exact text in the file
- Profileeach column typed as numeric, binary, date, category, text or ID
- Checkper-column, target and train-vs-test checks add issues
- Scoreeach issue costs points; the penalties compound
- Fixa pandas snippet per issue, and a cleaned CSV for the safe ones
A placeholder sits apart from the data
Disguised missing values
Text placeholders like N/A or unknown are easy: they match a fixed list. Numeric ones are harder, because −1 or 999 can be a real value. Dataset Doctor only calls a number a placeholder when it is a familiar one, it repeats, and it sits clearly outside the range of everything else in the column.
hits = []
if p.type == "numeric":
counts, order = {}, []
for i in p.present_idx:
x = p.num[i]
if x is not None and x in SENTINELS:
k = js_str(x)
if k not in counts:
counts[k] = 0
counts[k] += 1
for k in js_keys(counts):
x, c = float(k), counts[k]
if c < 2 or c < 0.002 * np_:
continue
others = [y for y in p.sorted if y != x]
if len(others) < 10:
continue
lo, hi = quantile_sorted(others, 0.01), quantile_sorted(others, 0.99)
span = max(1e-9, hi - lo)
apart = (x < others[0] and lo >= 0) if x < 0 else (x > hi + 2 * span or (x >= 99 and x > others[-1]))
if apart:
hits.append({"value": int(x), "count": c})- Only the usual suspects are candidates: −1, −9, −99, …, 99, 999, 999999 and so on (the
SENTINELSlist invalues.py). - A one-off −1 is not a convention. The value must appear at least twice and in at least 0.2% of the filled rows.
- The column's real range is measured without the candidate, using the 1st and 99th percentiles so a few outliers can't stretch it.
- A negative placeholder must sit below every other value in a column that is otherwise non-negative (−1 among ages). A positive one must be far above the 99th percentile, or be at least 99 and above the maximum (999999 among incomes).
Once a value is confirmed as a placeholder it is stored in p.sentinels, and the leakage, outlier and shift checks skip it. Otherwise 999999 would show up as a huge outlier and bend every other statistic.
Score every column alone, out of fold
How predictive is one column, with no model?
A leak is a column that gives away the answer, usually because it was recorded after the outcome. To find one, Dataset Doctor measures how well each column predicts the target on its own. First every column is turned into a few small integer codes, so numbers and categories are treated the same way.
if p.type in ("numeric", "binary") and p.sorted:
s = p.sorted
edges = [s[(k * len(s)) // 10] for k in range(1, 10)]
edges = [e for i, e in enumerate(edges) if i == 0 or e != edges[i - 1]]
sentinels = p.sentinels
for i in range(n):
x = p.num[i]
if x is None or x in sentinels:
codes[i] = 0
continue
b = 0
while b < len(edges) and x >= edges[b]:
b += 1
codes[i] = b + 1
return codes
top = {v: j + 2 for j, v in enumerate(top_order(p.counts)[:50])}
for i in range(n):
v = p.raw[i]
codes[i] = 0 if is_missing_token(v) else top.get(v, 1)- Nine cut points at the deciles give up to ten equal-count bins. Repeated cut points are merged, so a column with few distinct values gets fewer bins.
- Blanks and confirmed placeholders share code 0. If being blank predicts the outcome, that still shows up in the score.
- Categories keep their 50 most common values as codes 2 to 51.
- Everything rarer collapses into code 1, which stops a column of unique values from getting a code per row.
Each row's prediction is the target rate among rows with the same code, counted only from the other four folds. The row never sees its own label, so a unique ID column can't memorise the answer and scores like a coin flip. The probabilities then give an AUC for a category target (an R² for a numeric one).
key, f = codes[i], fold[i]
tot, own = totals[key], tab[key][f]
train_n = tot_all - tot_f[f]
cnt2 = 0
for c4 in range(C):
cnt2 += tot[c4] - (own[c4] if own else 0)
pr = []
for c5 in range(C):
pc = div(prior[c5] - prior_f[f][c5], train_n)
pr.append(div((tot[c5] - (own[c5] if own else 0)) + M_SMOOTH * pc, cnt2 + M_SMOOTH))
probs.append(pr)
labs.append(c)- Counts are kept per (code, fold).
totis the code's total over all folds andownis the part from this row's fold. - Subtracting
ownleaves only the other four folds: the training data for this row. - The class rate across all training rows, again without this fold.
- An m-estimate with
M_SMOOTH = 2: the code's rate pulled toward that prior by two pseudo-rows, so a code seen once can't claim certainty.
A leak has to stand out
Strong features are not leaks
A high score alone proves nothing. On the scikit-learn breast-cancer data a dozen measurements each reach an AUC above 0.9, and they are all real. What gives a leak away is the gap to the rest: its error rate is several times smaller than the next-best column's.
standout = (1 − scorenext best) / (1 − score)
# A leak stands out: its error (1 - AUC, or 1 - R²) is several times smaller than the
# next-best feature's. Two or more near-perfect features are flagged together.
hi_t, med_t, perfect = (0.98, 0.9, 0.995) if is_class else (0.95, 0.85, 0.99)
n_perfect = sum(1 for s in strength if s["score"] >= perfect)
for k, s in enumerate(strength):
nxt = strength[1 if k == 0 else 0] if len(strength) > (1 if k == 0 else 0) else None
ratio = div(1 - js_max(0, nxt["score"]), js_max(1e-6, 1 - s["score"])) if nxt else math.inf
s["standout"] = ratio
s["flag"] = bool((k == 0 and s["score"] >= med_t and ratio >= 3) or (n_perfect >= 2 and s["score"] >= perfect))- Thresholds for classification (AUC) and regression (R²): a medium flag needs 0.9 or 0.85, a high-severity one 0.98 or 0.95, and “near-perfect” means 0.995 or 0.99.
- The columns are already sorted by score, so the top column compares itself with the second, and every other column with the top.
- The ratio of errors.
1e-6keeps a perfect score from dividing by zero. - The rule: the top column, strong enough, with at least 3× lower error. Two leaks would hide each other under that rule, so two or more near-perfect columns are flagged together.
In the demo, cancellation_reason scores 1.000 while the next-best column scores 0.611, and it is flagged. On the iris, wine, breast-cancer and diabetes datasets, nothing is flagged as a leak. What it does report is that the first three files are sorted by class, which is true and is exactly the kind of thing that bites when you split without shuffling.
Copies and contradictions
Train/test overlap and conflicting labels
Two of the checks come down to the same trick: turn each row into a key and look it up in a hash set. A test row whose key appears in the training file measures memory, not skill. The key leaves out ID columns, because the same customer copied under a new ID is still a copy.
# overlap: identical rows (ignoring ID columns and the label) and shared IDs
shared = [c for c in train.columns if c in tcols]
ids = [p.name for p in prof if p.is_id and p.name in shared]
key_cols = [c for c in shared if c not in ids and (target is None or c != target.name)]
ji = [train.columns.index(c) for c in key_cols]
jt = [tcols.index(c) for c in key_cols]
train_keys = {"\x01".join(r[j] for j in ji) for r in train.rows}
overlap = [i for i, r in enumerate(test.rows) if "\x01".join(r[j] for j in jt) in train_keys]- The key is every shared column except IDs and the label. A copied row with a different label is still a copy.
- Column positions are looked up by name in each file, so the two files may order their columns differently.
- The raw cell text joined with a control character that can't appear in a value. One pass over each file, so the check is linear in the number of rows.
Conflicting labels use the same key inside one file, but group the rows and collect their labels. Identical features with two different labels mean noisy labels or a missing feature, and no model can get all of them right.
def _check_conflicts(t, prof, n, labeled, y, is_class, strength, table, issues):
"""Groups of rows with identical features (ignoring IDs and leaks) but different labels."""
leak_cols = [s["column"] for s in strength if s["flag"]]
feat_idx = [p.index for p in prof if p is not t and not p.is_id and p.name not in leak_cols]
if not feat_idx:
return
groups = {}
for i in labeled:
r = table.rows[i]
key = "\x01".join(r[j] for j in feat_idx)
g = groups.get(key)
if g is None:
g = groups[key] = {"rows": [], "labels": set()}
g["rows"].append(i)
g["labels"].add(y[i] if is_class else js_str(y[i]))
conflict_rows, conflict_groups = [], 0
for g in groups.values():
if len(g["labels"]) > 1:
conflict_groups += 1
conflict_rows.extend(g["rows"])- The leakage check runs first and its flagged columns are left out of the key. A leak differs with the label by definition, so it would hide every conflict.
- The same row key as the overlap check, built from the remaining features.
- Each group collects the set of labels it has seen.
- More than one label in a group is a conflict. The demo's 12 planted contradictory groups are found this way, and 30 copied rows by the overlap check.
A score that compounds, fixes you can rerun
From findings to a number and a cleaned file
The health score is a triage summary, not a probability. Each issue costs points by severity (high 15, medium 7, low 2, notes 0), and each penalty takes its percentage off what is left, so many small problems lower the score a lot without ever reaching zero: a file with many small problems lands in the 30s, and the categories show where the points went.
score = 100 · Πissues (1 − penalty / 100)
def score_issues(issues):
"""Set each issue's penalty, sort issues (severity, penalty, category) and return the score."""
penalties = {c: 0 for c in CATEGORY_ORDER}
for issue in issues:
f = 0.5 + 0.5 * min(1, 10 * issue["affected"])
if issue["type"] in FULL_WEIGHT:
f = 1
issue["penalty"] = js_round(SEVERITY_WEIGHTS[issue["severity"]] * f * 10) / 10
penalties[issue["category"]] = penalties.get(issue["category"], 0) + issue["penalty"]
issues.sort(key=lambda i: (SEVERITY_RANK[i["severity"]], -i["penalty"], _category_index(i["category"])))
s = 1
for issue in issues:
s = s * (1 - issue["penalty"] / 100)
return {"overall": js_round(100 * s), "penalties": penalties, "order": list(CATEGORY_ORDER)}- Half weight when an issue touches almost no rows, rising to full weight once 10% of rows are affected.
- Leaks, train/test overlap, conflicting labels and sorting by the label always cost in full: a single one can invalidate an evaluation.
- Points are also added up per category (Model readiness, Train vs test, Validity…), so the report can show where they went.
- The compounding product. The demo files with the test file attached score 17 out of 100.
Every issue carries a pandas snippet, and the safe ones also carry an action, a small instruction the engine can apply itself. Here is the placeholder fix from section 01 being applied to the raw text of the file.
for issue in sel:
a = issue["action"]
if a.get("missing"):
m = a["missing"]
j, cnt = cols.index(m["column"]), 0
for r in rows:
v = r[j]
t = trim(v)
if t != "" and t.lower() in MISSING_SET:
r[j] = ""
cnt += 1
continue
x = parse_num(v)
if x is None:
x = parse_num_text(v)
if x is not None and x in m["numbers"]:
r[j] = ""
cnt += 1
log.append(m["column"] + ": " + str(cnt) + " placeholder values blanked")- Text placeholders such as
unknownbecome a truly empty cell. - Numbers are parsed both plain and as written for humans, so
"999,999"is caught as well as999999. - Only the numbers the check confirmed as placeholders are blanked. A real −1 in some other column is never touched.
- Every fix writes a log line, so the cleaned CSV comes with a record of what changed.
The cleaned CSV only applies fixes that are safe to automate: blanking placeholders, parsing numbers, merging spellings, rewriting dates, dropping duplicates and flagged columns, and a seeded shuffle. Missing values and outliers are left alone, because filling or clipping them is a modelling decision.
One engine, two languages
The browser version, and how it stays identical to Python
GitHub Pages can't run Python, so the whole engine also exists in JavaScript as web/doctor.js. The page loads it into a Web Worker, a background thread, so a large file doesn't freeze the page, and the file never leaves the visitor's machine. When workers aren't available (the page opened from disk), app.js runs the same engine on the main thread.
/* Runs the analysis off the main thread so the page stays responsive on big files. */
importScripts("doctor.js", "engine.js");
var engine = makeEngine(self.DatasetDoctor);
self.onmessage = function (e) {
var res;
try { res = engine(e.data); } catch (err) { res = { error: String(err && err.message || err) }; }
res.id = e.data.id;
self.postMessage(res);
};- The worker loads the engine and wraps it in a small command interface:
load,setTest,analyzeandfix. - An exception becomes an error message instead of killing the worker, so one bad file doesn't break the page.
- Each reply carries the request's
id, which is how the page matches answers to the promises waiting for them.
Two copies of an engine drift apart unless something checks them. tests/parity.py runs both on the demo, the scikit-learn datasets, 15 deliberately awkward edge-case files and the cleaned output, and requires the same issues, scores and text, and a byte-identical cleaned CSV. That forces the Python side to copy JavaScript's behaviour wherever the languages differ, collected in jscompat.py.
def js_round(x):
"""Math.round: round half toward +infinity."""
if x != x or x in (math.inf, -math.inf):
return x
r = math.floor(x)
return r + 1 if x - r >= 0.5 else r- Python's
round(2.5)gives 2 (ties go to the even number); JavaScript'sMath.round(2.5)gives 3. A score of 16.5 would print differently in the two versions. - NaN and infinities pass through untouched, as they do in JavaScript.
- Round half up toward +∞, so −2.5 becomes −2, exactly like
Math.round. The same file has JavaScript's number-to-string,toFixed, string trimming and object key order.
Even the fold each row lands in for the leakage check is shared: fold_of in stats.py is a Knuth multiplicative hash, and the JavaScript tests confirm its fast 32-bit version matches an exact BigInt one. The parity test covers 29 cases in all.
What it checks
Seventeen checks, in plain language
Most data-quality tools tell you a column has missing values. Dataset Doctor looks for the problems that quietly break a machine-learning project: a column that leaks the answer, test rows that are copies of training rows, labels that contradict each other, a file sorted by outcome, placeholders like −1 pretending to be data. A model trained on data like that looks excellent in testing and then fails in use.
You drop in a CSV, say which column a model would predict, and optionally add the file you'll test on. You get a list of problems in plain language, ordered by how much they matter. Each one comes with the evidence (example values and line numbers), a suggested pandas fix, and a one-click cleaned CSV or a Python cleaning script you can review and rerun.
It started as a small Streamlit prototype with five checks. This version adds the checks below, explains every finding, and writes the fix.
| check | what it catches |
|---|---|
| Leakage | A column that predicts the target far better than every other column, scored out of sample. Usually something recorded after the outcome. |
| Train vs test overlap | Rows (ignoring IDs) or IDs that appear in both files, which inflate test scores. |
| Distribution shift | Columns whose values or category mix differ between training and test (KS test, total variation distance with a chi-square test), and categories the training file never saw. |
| Conflicting labels | Identical rows with different targets: label noise, or a missing feature. |
| Sorted by target | Row position predicts the label, so an unshuffled split puts different classes in train and test. |
| Class imbalance | A rare class a model can ignore while still scoring high accuracy. |
| Missingness that predicts the target | Whether a value is blank says something about the outcome. Keep a flag instead of silently filling it. |
| Disguised missing values | "N/A", "unknown", "-" and numeric placeholders like −1, 999 or 999999 that sit apart from the real values. |
| Numbers stored as text | "$1,049.00", "12%", "(45)", so the column can't be used as a number. |
| Mixed types | A numeric column with a few stray words in it. |
| Dates | Mixed formats (2024-03-14 next to 03/14/2024), day/month ambiguity, impossible dates (February 30) and dates in the future. |
| Category spellings | US / USA / "United States" / "us ", plus likely typos ("Premum" for "Premium"). |
| Duplicates | Exact duplicate rows, and rows identical except for their ID. |
| Outliers | Values far from the bulk (robust z-score above 5, on a log scale for long-tailed columns), with placeholders excluded. |
| Useless columns | Empty, constant or nearly constant columns, identifiers, and pairs of columns that carry the same information. |
| Personal data | Email addresses, phone numbers, US social security numbers, card numbers (Luhn-checked) and IP addresses. |
| File structure | Rows with the wrong number of fields, usually an unquoted comma. |
The demo data
A churn file with problems planted on purpose
The demo is a synthetic customer-churn file generated by tests/make_demo.py with a fixed seed. The problems are planted on purpose and recorded in demo_truth.json:
- a batch of cancelled customers appended to the end of the file;
- country written five ways and a typo in the plan names;
- prices stored as text;
- −1 for unknown ages, 999999 for unknown incomes, and a few incomes with two extra zeros;
- a quarter of the dates in US format;
- a cancellation-reason column that leaks the outcome;
- a constant column, email addresses, 40 duplicated rows and 12 contradictory ones.
The test file was collected later: customers are older, there's a new "Enterprise" plan, and 30 training customers slipped into it. The test suites check that every planted problem is found and that the honest columns aren't flagged.
Limits
What to keep in mind when reading a report
- Every check is a heuristic with a threshold. A flagged outlier may be real; a flagged leak may be a legitimately strong feature that happens to stand out. Read the evidence before acting.
- The leakage check only sees one column at a time. A leak spread across two columns, each harmless alone, won't be caught.
- Shift tests use p-values on large samples, where tiny differences become "significant". That's why they also require a minimum effect size (KS distance or total variation of at least 0.1).
- The cleaned CSV only applies fixes that are safe to automate (see section 05). Missing values and outliers are left alone.
- It reads CSV and TSV. Save spreadsheets as CSV first. Files up to a few hundred thousand rows are comfortable in a browser tab; 100,000 rows take about four seconds.
Reproduce
Command line, Streamlit app and tests
The Python package doctor/ comes with a command-line tool and a Streamlit app, and gives the same results as this site.
python -m pip install numpy pandas streamlit pytest python analyze.py data/demo_train.csv --target churned --compare data/demo_test.csv python analyze.py data/demo_train.csv --script clean.py --fix cleaned.csv --markdown report.md streamlit run app.py python -m pytest -q tests/ # 54 tests python tests/parity.py # Python vs JavaScript, 29 cases (needs node) node tests/test_engine.js # JavaScript tests
Read further
Where everything lives
- doctor/checks.py per-column and whole-file checks, disguised missing values
- doctor/target.py leakage, conflicting labels, imbalance, ordering
- doctor/compare.py train vs test: overlap, shift, unseen categories
- doctor/scoring.py penalties and the health score
- doctor/fixes.py cleaning script and cleaned CSV
- doctor/jscompat.py JavaScript-identical rounding, formatting and ordering
- web/doctor.js the same engine in JavaScript, run by this site
- analyze.py command-line tool
- app.py Streamlit app
- doctor/__init__.py analyze() and the public API
- web/app.js the project page
- tests/make_demo.py demo data generator
- tests/parity.py Python vs JavaScript parity check
- README.md full documentation