# Dataset Doctor Dataset Doctor checks a CSV for the data problems that quietly break machine-learning models, explains each one in plain language, and writes the fix. Ordinary data profilers tell you a column has missing values. Dataset Doctor also looks for the things that make a model look excellent in testing and then fail in use: - a column that **leaks the answer** (for example something recorded after the outcome) - **test rows that are copies of training rows**, and columns that **drift** between the two files - **identical rows with different labels**, a file **sorted by the label**, and **class imbalance** - **disguised missing values** like `N/A`, `unknown`, `-1` or `999999` - **numbers stored as text** (`$1,049.00`, `12%`), **mixed date formats**, and **category spellings** (US / USA / "United States") For every issue you get the evidence (example values and line numbers), a suggested pandas fix, and, for the fixes that are safe to automate, a cleaned CSV. A cleaning script collects all the fixes so you can review and rerun them. It runs in two places with the same engine: - **In the browser** at [williamc.me/projects/dataset-doctor](https://williamc.me/projects/dataset-doctor/). Nothing is uploaded. - **In Python**: a command-line tool, a Streamlit app and an importable package. A parity test checks that both give identical results. ## Quick start ```bash cd dataset-doctor python3 -m venv .venv source .venv/bin/activate python -m pip install numpy pandas streamlit pytest python analyze.py data/demo_train.csv --target churned --compare data/demo_test.csv ``` Output (abridged): ```text ================================================== DATASET DOCTOR ================================================== Health score: 17/100 Rows: 2402 Columns: 15 Target: churned (classification) Issues -------------------------------------------------- [HIGH] cancellation_reason predicts churned suspiciously well [HIGH] 30 test rows also appear in the training file [HIGH] age is distributed differently in the test file [HIGH] plan has values in test that training never saw [MEDIUM] Rows are partly sorted by churned [MEDIUM] 12 groups of identical rows with different churned [MEDIUM] monthly_spend stores numbers as text [MEDIUM] signup_date mixes date formats [MEDIUM] country spells the same thing in different ways (2 groups) ... ``` More options: ```bash python analyze.py data.csv # target guessed from column names (target, label, churned, price, ...) python analyze.py data.csv --no-target # skip leakage and label checks python analyze.py data.csv --script clean.py # pandas cleaning script python analyze.py data.csv --fix cleaned.csv # cleaned copy of the file python analyze.py data.csv --markdown report.md --json report.json streamlit run app.py # the web app, locally ``` From Python: ```python import doctor report = doctor.analyze("data/demo_train.csv", target="churned", compare="data/demo_test.csv") print(report["score"]["overall"]) for issue in report["issues"]: print(issue["severity"], issue["title"]) print(doctor.cleaning_script(report, filename="demo_train.csv")) ``` `analyze` also accepts a pandas DataFrame. ## What it checks | Check | What it catches | |---|---| | Leakage | A column that predicts the target far better than every other column, scored out of sample | | Train vs test overlap | Rows (ignoring IDs) or IDs that appear in both files | | Distribution shift | Numeric columns (KS test) and category mixes (total variation distance + chi-square) that differ between files; categories the training file never saw | | Conflicting labels | Identical rows with different targets | | Sorted by target | Row position predicts the label, so an unshuffled split goes wrong | | Class imbalance | A class rare enough that accuracy becomes meaningless | | Missingness that predicts the target | Whether a value is blank says something about the outcome | | Disguised missing values | Text placeholders, and numeric placeholders (−1, 999, 999999, ...) that sit apart from the real values | | Numbers stored as text, mixed types | Currency signs, %, thousands separators, stray words in numeric columns | | Dates | Mixed formats, day/month ambiguity, impossible dates, future dates | | Category spellings | Same value written differently, plus likely typos (one edit away from a much more common value) | | Duplicates | Exact duplicates, and rows identical except for their ID | | Outliers | Robust z-score above 5 (on a log scale for long-tailed columns), with placeholders excluded | | Useless columns | Empty, constant or nearly constant columns, identifiers, columns that duplicate another | | Personal data | Emails, phone numbers, US SSNs, card numbers (Luhn-checked), IP addresses | | File structure | Rows with the wrong number of fields | ## How it works **Reading the file.** Every cell is kept as the exact text in the file, so nothing is silently converted before it's checked. Each column is then typed as numeric, binary, date, categorical, text or ID, and the checks run on that. **The leakage check.** For every column, Dataset Doctor measures how well that column *alone* predicts the target, with no model. Numbers are cut into 10 quantile bins and categories keep their 50 most common values. Each row is scored using only the other four fifths of the data (5-fold, out of fold), which is what stops an ID column from looking perfect. The score is an AUC for category targets and an R² for numeric ones. Strong features aren't leaks, so a column is flagged only when it **stands out**: its error (1 − AUC, or 1 − R²) is at least 3× smaller than the next-best column's, and its score is at least 0.9 AUC or 0.85 R². Two or more near-perfect columns are flagged together. On the scikit-learn iris, wine, breast-cancer and diabetes datasets nothing is flagged as a leak, even though breast cancer has a dozen features above 0.9 AUC. **The score.** Each issue costs points by severity (high 15, medium 7, low 2, notes 0), scaled between half and full by the share of rows affected. Leaks, train/test overlap, conflicting labels and sorting always cost in full. Penalties compound (each takes its percentage off what's left), and the report shows which category the points went to. It is a triage summary, not a calibrated probability. **Fixes.** Each issue has a suggested pandas snippet. The cleaned CSV applies only the fixes that are safe to automate: - blank out placeholders - parse numbers stored as text - merge spellings - rewrite dates as YYYY-MM-DD - drop duplicates, constant, ID, personal-data and leaking columns - shuffle Missing values and outliers are left for you to decide. ## Project structure ```text dataset-doctor/ ├── analyze.py command-line tool ├── app.py Streamlit app ├── index.html the web page (williamc.me/projects/dataset-doctor) ├── doctor/ Python package │ ├── __init__.py analyze() and the public API │ ├── csvio.py CSV reading and writing │ ├── values.py number, date, missing-value and spelling rules │ ├── profiler.py per-column profile and type inference │ ├── checks.py per-column and whole-file checks │ ├── target.py leakage, conflicting labels, imbalance, ordering │ ├── compare.py train vs test │ ├── scoring.py score │ ├── fixes.py cleaning script and cleaned CSV │ ├── report.py Markdown report and summary │ ├── stats.py, fmt.py, jscompat.py statistics, formatting, JavaScript-identical rounding/ordering ├── web/ │ ├── doctor.js the same engine in JavaScript (runs in the browser) │ ├── engine.js, worker.js runs the analysis in a background thread │ └── app.js the page ├── data/ │ ├── sample.csv the original tiny example │ ├── demo_train.csv, demo_test.csv demo with planted problems │ └── demo_truth.json what was planted └── tests/ ├── test_doctor.py pytest suite (54 tests) ├── parity.py Python vs JavaScript, 29 cases, results must be identical ├── edge_cases.py awkward files for the parity check ├── test_engine.js JavaScript tests ├── run_js.js runs the JavaScript engine from node └── make_demo.py regenerates the demo data ``` ## Tests ```bash python -m pytest -q tests/ python tests/parity.py # needs node node tests/test_engine.js ``` Set `REAL_DATA` to a folder holding `iris.csv`, `wine.csv`, `breast_cancer.csv` and `diabetes.csv` (exported from `sklearn.datasets` with the target in a column named `target`) to include the false-alarm checks on clean data. The demo data is synthetic and generated with a fixed seed by `tests/make_demo.py`. Every problem planted in it is listed in `data/demo_truth.json`. The tests check that each one is found and that the honest columns are not flagged. If you change a check, change it in both `doctor/` and `web/doctor.js`, then run the parity test. ## Limitations - Every check is a heuristic with a threshold. An outlier may be a real observation, and a flagged leak may be a legitimately strong feature. Read the evidence before acting. - The leakage check looks at one column at a time. A leak spread across two columns, each harmless alone, won't be caught. - Shift tests also require a minimum effect size, because on large files tiny differences become "significant". - Files are read as CSV/TSV. Save spreadsheets as CSV first. In the browser, about 100,000 rows take around four seconds. ## Design philosophy Dataset Doctor separates detection from explanation. The detection is deterministic statistics that can be tested; each finding then carries a plain-language explanation and a fix. An AI model could later help explain findings or suggest fixes, rather than being responsible for detecting every issue directly. This keeps the system easy to test, understand and improve. ## Ideas for later - Leaks spread across pairs of columns (two-feature interactions) - Time-aware checks: future information in time-series splits, gaps and duplicated timestamps - Excel and Parquet input - Saved reports, so you can compare a dataset over time ## License No license has been selected yet.