# Shortcut Hunt

A puzzle game about data leakage and shortcut learning, and a set of automated shortcut checks for your own CSV.

Each of the eight levels trains a small model in your browser on a synthetic dataset and scores it on a held-out set. The score is suspiciously good. You look at sample rows (or sample images) and guess what the model is really using. Then the reveal: the shortcut is removed (a column dropped, pixels masked, backgrounds shuffled, the split redone by time, near-copies deduplicated), the model retrains live, and the score collapses to its honest level, with the model's weights, a counterfactual where it helps, and a short note on how the same thing happens in real projects.

| level | the shortcut | with it | without |
|---|---|---|---|
| 1 The ID that knew too much | row IDs assigned after sorting by label | 99.4% | 72.2% |
| 2 Tomorrow's weather, today | `cancel_fee`, written after the outcome | 93.9% | 72.8% |
| 3 Déjà vu | 240 of 320 test rows are near-copies of training rows | 90.9% | 63.7% |
| 4 The watermark | a faint 2×2 mark on one class of images | 100.0% | 61.7% |
| 5 Background noise | background brightness tracks the class | 96.7% | 68.3% |
| 6 The timestamp | month as a category, random split | 86.2% | 56.6% (time split) |
| 7 The label in the notes | the word "escalated" in a free-text note | 98.0% | 71.3% |
| 8 Accuracy that lies | 95% of rows are one class | 94.6% | 50.0% balanced accuracy |

**Bring your own CSV** (bottom of the page) runs the four hunts on a file with a two-class target: ID-like columns whose order tracks the label, columns with a single-feature AUC that is too good to be true, test rows already present in training after a random split, and a label rate that moves with the calendar (random split vs. time split of a logistic regression). Nothing is uploaded. The write-up is honest about which of these fire on clean data.

Facilitator mode (`?host` in the URL, or the button) hides right/wrong until the host presses Reveal, for playing with a room. Keys: 1–9 pick, R reveal, N next, H facilitator mode.

## Layout

    hunt/models.py        logistic regression (deterministic full-batch GD), 1-NN, decision stump, AUC, metrics
    hunt/levels.py        the eight level generators (fixed seeds), materialize(before/after), run_level
    hunt/images.py        28x28 shape masks, the packed image format, compose
    hunt/checks.py        the CSV shortcut checks + a small CSV reader
    hunt/rng.py           mulberry32, identical in Python and JavaScript
    data/make_levels.py   writes web/levels/*.json (13–36 KB each)
    data/make_demo_csv.py writes web/demo.csv (four planted shortcuts)
    experiments/run_levels.py   results table, out/*.png, the false-positive study
    web/models.js, engine.js, checks.js   line-for-line ports; worker.js runs them off the main thread
    web/page.js           the game, its canvas charts, the CSV tool
    index.html            the project page; how-it-works/index.html the write-up
    tests/                pytest tests, parity.py (Python vs JavaScript), run_js.js

## Run it

```bash
pip install -r requirements.txt
python data/make_levels.py          # regenerate the level files (byte-identical)
python data/make_demo_csv.py        # regenerate the demo CSV
python experiments/run_levels.py    # prints the table above, writes out/*.png and out/results.json
python -m pytest -q                 # 22 tests, about 3 s
python tests/parity.py              # needs node; every level and six CSV files through both engines
```

The page needs a web server (Web Workers do not run from `file://`): from the site root,
`python3 -m http.server 8000`, then open `http://localhost:8000/projects/shortcut-hunt/`.

## How the port stays exact

Nothing in the game is random: datasets ship ready-made, logistic regression is full-batch gradient descent
from zero weights for a fixed 500 steps with a computed step size (1 / Lipschitz constant, via 60 steps of power
iteration), 1-NN and the stump break ties by lowest index. The CSV tool's only random numbers (the 70/30 split and
the two folds) come from mulberry32 in both languages. `tests/parity.py` compares every accuracy, weight, check
id, severity and sentence. Change a model or a check in both `hunt/` and `web/` and run it.