# Model Router

Which models should handle which requests so that quality stays above a target and cost is as
low as possible? Give it an eval log (one row per prompt, with each candidate model's correctness,
cost and optional latency and confidence) and it returns the cheapest routing plan that meets your
target, **with a statistical guarantee** measured on prompts the tuning never saw.

Python package `router/` (numpy, scipy only for the tests), a CLI, and a line-for-line JavaScript
port in `web/router.js` that powers the project page (`index.html`) entirely in the browser.

## Use it on your own log

```bash
pip install -r requirements.txt
python -m router sample                                   # the two accepted CSV layouts
python -m router plan evals.csv --target 0.9 --delta 0.05 # the plan, certified at 95% confidence
python -m router plan evals.csv --target 0.9 --json plan.json --signal learned --cal 0.6 --safety 2
```

CSV layouts (column names are case-insensitive; `cost_usd`, `price`, `tokens`, `is_correct`,
`score`, `p_correct`, `question` and similar are recognised):

```
# wide: one row per prompt
prompt_id,prompt,tiny_correct,tiny_cost,tiny_latency,tiny_confidence,frontier_correct,frontier_cost,...
# long: one row per prompt x model
prompt_id,model,correct,cost,latency,confidence,prompt
```

`correct` is 0/1 (or a score in [0, 1]); `cost` is in any unit; `confidence` (0–1, the model's own
estimate that it was right) is what a cascade gates on; `prompt` text lets the tool learn a router
instead. Models are ordered by mean cost. Prompts missing a model are dropped.

## What it computes

1. **Single-model baselines** and their Pareto frontier of (cost, quality).
2. **Oracle**: the cheapest model that is right per prompt; the floor on cost at every quality.
3. **Cascade**: cheap → expensive; a request stops at the first model whose confidence clears its
   threshold (cost accumulates). Every cascade on a grid of 24 quantile thresholds per model is
   enumerated on a calibration split; the cheapest one above the target (plus a margin) is picked.
   Without a confidence column, P(right | prompt text) is learned (hashed bag of words → logistic
   regression, out-of-fold on the calibration split) and used as the gate.
4. **Learned router**: predict first, call one model. Same predictions, one threshold.
5. **Guarantee**: the chosen plan is evaluated once on the held-out split; Clopper–Pearson (0/1) or
   Hoeffding (scores) lower bound on quality at your δ, Hoeffding upper bound on cost. When the
   target cannot be certified it says so, and how many held-out rows would be needed.
6. **Sensitivity**: certified cost against the target, the cheapest certified single model at each
   target, the oracle's price, and where the cascade stops being the cheapest certified option.

## Results on the demo (`web/demo.csv`, 1,500 prompts × 4 models, target 90%, δ = 0.05)

| policy | quality | 95% lower bound | cost / 1k |
|---|---|---|---|
| tiny / small / mid / frontier alone | 61.6 / 73.3 / 82.1 / 92.4% | 58.6 / 70.5 / 79.7 / 90.6% | $0.012 / 0.051 / 0.402 / 3.691 |
| Oracle | 98.5% | 97.6% | $0.374 |
| **Cascade** (tiny ≥ 0.970 → small ≥ 0.916 → mid ≥ 0.854 → frontier) | 92.1% | **90.3%** | **$1.531** (41% of frontier) |
| Learned router (text only) | 92.7% | 90.9% | $3.660 |

The cascade certifies 90% at 41% of the frontier model's cost (40–60% across split seeds).
Full numbers, figures and limits are on the How it works page.

## Checks

```bash
python -m pytest -q          # 24 tests, ~30 s: bounds vs scipy and by simulation, cascade vs
                             # row-by-row simulation, search optimality, monotonicity, parity with JS
python tests/parity.py       # browser engine against the Python on five files (needs node)
python tools/make_figures.py # out/*.png and the numbers quoted on the page
python data/make_demo.py     # regenerate web/demo.csv
```

If you change the maths, change it in `router/` and `web/router.js` both; `tests/parity.py` will
tell you if they drift apart.