# clfkit: classifiers you can trust inside a tool `clfkit` is a small Python toolkit and CLI for putting a classifier into a tool when you need to know how far to trust it. Accuracy alone doesn't tell you that. A tool also needs to answer these questions: | Question a tool must answer | clfkit piece | |---|---| | Is "90% confident" really 90%? | **calibration**: reliability diagram, ECE, temperature / Platt / isotonic | | Which labels are plausible, with a guarantee? | **conformal prediction sets**: LAC, APS, Mondrian (per-class) | | When should it say "I don't know"? | **selective classification**: abstention threshold with a certified error bound | | Has the incoming data changed? | **drift detection**: per-feature tests, score drift, classifier two-sample test, label-shift estimate | | How brittle is a prediction? | **robustness**: exact L2 certificates for linear models, noise-stability check for any model | | When is a cosine similarity "similar"? | **embedding threshold calibration**: null distribution, precision-certified threshold | It works on any CSV. Numeric, categorical and free-text columns are detected automatically, and text becomes TF-IDF features. It also works on precomputed embeddings (`.npy`) from any model. The dependencies are numpy, scipy, pandas, scikit-learn and matplotlib. It runs on CPU and is single-threaded by default (the random forest uses at most 2 workers). --- ## Install ```bash cd classifier-toolkit pip install -e . # gives the `clfkit` command pip install pytest # for the tests ``` ## Quick start (exact commands) ```bash # Train: model comparison, calibration, conformal and abstention thresholds, all evaluated on a held-out test split clfkit train data.csv --target label --text-col text --out model_dir # (precomputed embeddings instead of a CSV) clfkit train --embeddings emb.npy --labels labels.txt --out model_dir # Self-contained HTML report (plots are embedded as base64 PNG, so it's a single file you can email) clfkit report model_dir # -> model_dir/report.html # Predict: label, conformal set, abstain flag and per-class probabilities for each row clfkit predict model_dir new.csv --alpha 0.1 [--score aps|lac] [--mondrian] [--max-risk 0.05] # -> model_dir/predictions.csv (pred_label, confidence, prediction_set "a|b", set_size, abstain, p_...) # Drift: compare a new UNLABELLED batch with the reference data clfkit drift model_dir new_batch.csv [--fail-on-drift] # exits with code 1 on drift (for cron/CI); drift.json is included in the next report # Embedding similarity threshold with a precision guarantee clfkit similarity emb.npy labels.npy --target-precision 0.95 [--center] ``` Run the full demo (breast cancer, digits, synthetic text tickets with three drift scenarios, and embeddings) and the tests: ```bash python examples/demo_end_to_end.py # ~12 s; writes examples/output/*/report.html python -m pytest -q # 20 tests, ~10 s ``` --- ## The theory, briefly **The data split is what makes the guarantees valid.** `train` divides the rows into four parts: ``` train (60%) model selection (internal 80/20) and final fit calib_fit (10%) fits the probability calibrator calib_conf (10%) conformal quantiles and abstention threshold <- the guarantees come from here test (20%) honest evaluation and the drift reference ``` None of these parts is reused for a second purpose. For example, the conformal quantile is computed on data that the temperature was not fitted on. Use `--calib-size` to change the share (with less data, the guarantees get looser; see the caveats below). ### Calibration A model is calibrated if, among the inputs where it says 70%, the label is right 70% of the time. - **ECE** bins predictions by confidence and averages |accuracy − confidence|. It depends on the binning and is biased, so use it to compare methods on the same data. The report also gives NLL and Brier score, which are proper scoring rules. - **Temperature scaling** (Guo et al., 2017) computes softmax(log p / T) with one scalar T fitted by NLL. It never changes the argmax and is hard to overfit. T > 1 fixes overconfidence. In the demo digits model T is 0.82: an L2-regularised logistic regression on small data is underconfident, and temperature scaling sharpens it. - **Platt** (per-class logistic on the logit) and **isotonic** (per-class monotone fit), both one-vs-rest and renormalised. Isotonic is more flexible but needs more data. In the demos it has the worst NLL because it outputs hard 0s and 1s on small calibration sets. ### Conformal prediction sets The idea comes from Vovk, Gammerman & Shafer, *Algorithmic Learning in a Random World* (2005); the split (inductive) form is what's used here. It works like this: 1. Choose a score s(x, y), where larger means "y fits x worse". 2. Let q be the ⌈(n+1)(1−α)⌉-th smallest score on the n calibration points. 3. Predict C(x) = {y : s(x,y) ≤ q}. **Guarantee:** if calibration and test points are exchangeable, P(Y ∈ C(X)) ≥ 1−α. This holds for any model; a bad model just gives bigger sets. With continuous scores (no ties), coverage is also ≤ 1−α + 1/(n+1). The guarantee is **marginal**: it averages over test points and over the random calibration set. It is not a guarantee for each input or each class. - **LAC** (Sadinle, Lei & Wasserman, 2019) uses s = 1 − p_y. It gives the smallest sets on average, but coverage is uneven: hard classes are under-covered. - **APS** (Romano, Sesia & Candès, NeurIPS 2020) uses s = (probability of the classes ranked above y) + U·p_y, with U random. Sets adapt to how hard the input is. RAPS (Angelopoulos et al., 2021) adds a size penalty; it is not implemented here. - **Mondrian / class-conditional** uses one quantile per class, which gives P(Y ∈ C | Y = k) ≥ 1−α for **every** class k. The cost is larger sets. It also needs at least 1/α − 1 calibration points per class; with fewer, q_k = ∞ and the set includes everything. - **Empty sets are legitimate.** When the model is more accurate than 1−α, the method can return an empty set so that it doesn't over-cover. That is why the average set size can be below 1. In a tool, treat an empty set as "abstain" or lower α. ### Abstention with a guarantee The rule is: answer if confidence ≥ λ, otherwise abstain. Selective risk is the error rate among answered inputs: R(λ) = P(wrong | conf ≥ λ). For a fixed λ, the errors among the n_λ answered held-out points follow a Binomial(n_λ, R(λ)) distribution, so the Clopper-Pearson upper bound applies. clfkit evaluates a **fixed, data-independent grid** of 53 thresholds with a Bonferroni correction (δ/53). That makes all the bounds hold at the same time, so picking the lowest certified λ (the one that answers the most) after looking at the data is still valid. This is the same idea as Geifman & El-Yaniv (2017) and the "Learn then Test" framework (Angelopoulos et al., 2021). - **Guaranteed:** with probability ≥ 1−δ over the held-out draw, the true error rate among answered inputs is ≤ the target, for future data from the same distribution. - **Not guaranteed:** - whether any single input is correct - the error rate you observe on one finite batch. In the demo, the certified threshold gave 5.6% error on a 600-row batch against a 5% target. That's within sampling noise; the guarantee is about the population rate. - the fraction of inputs answered - anything after drift - **Data requirement:** even with zero errors, the bound is 1 − (δ/m)^{1/n}. To certify a risk r you need n ≳ ln(m/δ)/r answered held-out points; for r = 2% that's about 349. If no λ certifies the target, `predict` abstains on everything and says so. ### Drift Each check compares a new unlabelled batch with the reference sample (the test split): - **Per-feature tests.** Numeric features get a KS test. Categorical features get a chi-square test. Text features get a KS test on document length and on the out-of-vocabulary rate. All p-values are **Holm-adjusted** to control the family-wise error rate. - **Score drift:** a KS test on the model's confidence. - **Classifier two-sample test** (Lopez-Paz & Oquab, ICLR 2017): train a classifier to tell reference rows from new rows. If nothing changed, its held-out accuracy is Binomial(n, ½), so a binomial test gives an exact p-value. It catches multivariate drift that the per-feature tests miss. - The alarm is raised at level 0.01, split by Bonferroni across the three families. - **Label shift (BBSE**, Lipton, Wang & Smola, ICML 2018): if p(y) changes but p(x|y) does not, then C·w = μ, where C is the joint confusion matrix on labelled reference data and μ is the distribution of predicted labels on the new batch. Solving gives w = q(y)/p(y) and the new class proportions. This is only meaningful under the label-shift assumption. The output includes the condition number of C; a large value means the estimate is unreliable. ### Robustness - **Exact L2 certificate (linear models).** The region where class c wins is the polyhedron {z : (w_c − w_k)·z + b_c − b_k ≥ 0 for all k}. The distance from z to its boundary is therefore **exactly** r(z) = min_{k≠c} (f_c − f_k)/‖w_c − w_k‖. For binary problems this reduces to |w·z + b|/‖w‖. The tests check that no perturbation inside the radius changes the label, and that one 0.1% beyond it does. Three scope limits apply: - The radius is measured in the model's **feature space**: standardised numerics, one-hot categoricals, TF-IDF. - It covers the argmax of the **logistic-regression baseline** (which clfkit always fits), not the conformal sets and not tree models. - Edits that aren't small in L2, such as swapping a word, are not covered. - **Empirical stability (any model):** the fraction of predictions unchanged under random Gaussian noise or random token dropout. Random noise is much weaker than adversarial noise, so this is a smoke test, **not** a certificate. ### When is a cosine similarity significant? Take two independent uniformly random directions in R^d and let c be the cosine between them. Then (1+c)/2 ~ Beta((d−1)/2, (d−1)/2), so the standard deviation of c is exactly 1/√d. **In 768 dimensions, a cosine of 0.2 is 5.5 standard deviations out (p ≈ 1e-8).** Real embedding models are anisotropic, though: vectors share a common direction. In the demo's synthetic anisotropic embeddings, **100% of unrelated pairs have a cosine above 0.2**, with a mean of 0.62. So the isotropic null is only a sanity check. Measure the null on pairs you know are unrelated, and consider mean-centring the embeddings. `clfkit similarity` picks the lowest threshold whose precision lower bound is ≥ the target. It uses the same Bonferroni plus Clopper-Pearson approach as abstention. Two caveats: - Precision depends on the base rate of related pairs in deployment. - The bound assumes independent pairs, so it is approximate when pairs share items. --- ## Expected output (real numbers from `python examples/demo_end_to_end.py`, seed 0) **Synthetic support tickets.** 3000 rows (text, channel, numeric), 4 classes. Default `clfkit train`, α = 0.1, max-risk 0.05: ``` hgb val_acc=0.939 val_logloss=0.280 val_ece=0.051 <- auto-selected (best log-loss) logreg val_acc=0.908 val_logloss=0.442 val_ece=0.200 rf val_acc=0.933 val_logloss=0.453 val_ece=0.251 TEST accuracy=0.958 ECE=0.013 NLL=0.128 (calibration: temperature, T=1.38; uncalibrated ECE 0.024) conformal lac coverage=0.903 (target 0.90) avg_size=0.92 worst_class=0.868 conformal lac_mondrian coverage=0.912 avg_size=0.93 worst_class=0.875 conformal aps coverage=0.925 avg_size=1.05 worst_class=0.889 conformal aps_mondrian coverage=0.925 avg_size=1.05 worst_class=0.882 abstain if conf<0.50: certified risk<=0.05 (UB 0.0498); test coverage=0.992, test risk=0.0353 ``` Drift on three 600-row batches: | batch | alarm | what fired | BBSE estimate | |---|---|---|---| | same distribution | no | nothing (C2ST acc 0.485, p = 0.78) | ≈ 0.25 each (true 0.25) | | label shift (billing 0.55, others 0.15) | **yes** | C2ST p = 6e-9; `n_prev_tickets` (Holm p = 0.003) | billing 0.57, account 0.17, shipping 0.15, technical 0.12 | | new vocabulary (25% unseen words) | **yes** | `text[oov_rate]` p ≈ 0; C2ST p = 2e-13 | unchanged ≈ 0.25 (correct: labels didn't shift) | **Digits** (10 classes, loaded through the `.npy` embedding loader), α = 0.1. The class-conditional gap, averaged over 50 random calibration/test re-splits: ``` LAC worst class '8' 0.766, best class 0.981 <- marginal coverage is 0.90, but class 8 is under-covered LAC Mondrian worst class '0' 0.898, best class 0.932 <- every class is about 0.90 ``` **Breast cancer** (binary), α = 0.05: APS coverage 0.947 with an average set size of 1.04. The abstention threshold of 0.80 is certified for ≤ 15% risk and gave 4.5% test error while answering 98% of inputs. A 2% target can't be certified with 86 held-out points; even an error-free model would need ≥ 349. The logistic regression's median certified L2 radius is 1.72 (in standardised units). **Embeddings:** isotropic P(cos ≥ 0.2) is 5.5e-2 at d = 64, 3.9e-5 at d = 384, 1.1e-8 at d = 768 and 1.2e-15 at d = 1536. On anisotropic synthetic embeddings (d = 384), no threshold certifies 95% precision before centring. After centring, cos ≥ 0.12 gives precision 0.983 (lower bound 0.962). **Runtime** on a shared 2-CPU box: the full demo takes about 12 s and `pytest` about 10 s. `clfkit train` on the 3000 tickets takes about 5 s; `report` about 3 s, `predict` about 1.5 s and `drift` about 2.3 s, mostly Python and sklearn import time. On a heavily loaded machine, training took about 40 s. --- ## Tool ideas 1. **Email triage with abstain-to-human.** Train on past emails labelled by folder or priority (text column). Auto-file emails only when `abstain == False`; the rest go to a "needs a human" queue. The certified `--max-risk` caps the misfiling rate among auto-filed mail. *Uses: text loader, calibration, selective classification, drift (`--fail-on-drift` in a nightly cron).* 2. **Support-ticket routing with prediction sets.** Route a ticket when its set has one team; when the set is {billing, account}, send it to both or to a shared queue. Use `--mondrian` so small teams aren't systematically under-covered. *Uses: conformal APS and Mondrian, report per-class coverage.* 3. **Document-type classifier with a drift alarm.** Classify invoices, contracts and letters from extracted text. Run a weekly `clfkit drift` on the incoming batch: the OOV-rate test and C2ST catch a new vendor template, and BBSE tells you whether only the document mix changed. *Uses: text loader, drift, BBSE.* 4. **Receipt and expense categoriser.** Merchant (categorical), amount (numeric) and line text → expense category. Show the user the prediction set as a short pick-list instead of a single guess. *Uses: mixed-type CSV detection, conformal sets, calibrated probabilities.* 5. **Lab-notebook entry tagging.** Embed entries with any encoder, then `clfkit train --embeddings`. Tag entries automatically only above the certified threshold, and suggest the conformal set otherwise. *Uses: embedding loader, abstention, conformal.* 6. **Semantic "is this a duplicate?" checker.** Use `clfkit similarity` on labelled duplicate and non-duplicate pairs to get a cosine cutoff with a precision lower bound, instead of guessing 0.8. *Uses: empirical null, precision-certified threshold, centring.* 7. **Sensor or instrument fault classifier with a sanity layer.** Train on tabular sensor features. Use the linear model's certified radius to flag readings within noise distance of the decision boundary, and use score drift to detect sensor degradation. *Uses: robustness certificate, empirical stability, score drift.* 8. **Paper and arXiv abstract triage for reading lists.** Classify abstracts as read, skim or skip. Calibrated probabilities let you rank by P(read), and the abstention rate on a new month's papers tells you when your interests have shifted and the model needs retraining. *Uses: text TF-IDF, calibration, selective risk–coverage, drift.* 9. **Quality gate for an LLM labeller.** Treat LLM-produced labels as predictions with self-reported scores. Calibrate them on a small human-labelled set, then use conformal sets or abstention to decide which items still need human review. *Uses: calibration and conformal on any probability vector, through the Python API (`SplitConformal`, `TemperatureScaler`).* Every module also works on raw arrays without the CLI. For example, `SplitConformal(alpha=.1, score="aps").fit(P_cal, y_cal).predict_sets(P_new)`. --- ## File layout ``` classifier-toolkit/ ├── README.md, pyproject.toml, requirements.txt, .gitignore ├── clfkit/ │ ├── __init__.py thread caps, version │ ├── data.py CSV/embedding loading, column-type detection, preprocessing, 4-way split │ ├── models.py logreg / hist-gradient-boosting / random forest; model comparison │ ├── calibration.py ECE, NLL, Brier, reliability bins; temperature, Platt, isotonic │ ├── conformal.py split conformal: LAC, APS, Mondrian; coverage evaluation │ ├── selective.py risk-coverage, Clopper-Pearson, certified abstention threshold │ ├── drift.py KS/chi2 + Holm, score drift, C2ST, BBSE label-shift estimation │ ├── robustness.py exact certified L2 radius (linear), certified-accuracy curve, noise stability │ ├── similarity.py cosine null distribution, pair sampling, precision-certified threshold │ ├── pipeline.py Bundle: train / evaluate / predict / drift; save/load │ ├── report.py self-contained HTML report (base64 PNG plots) │ ├── datasets.py synthetic text tickets, tabular data, anisotropic embeddings │ └── cli.py `clfkit` command ├── examples/demo_end_to_end.py └── tests/ conformal coverage, calibration and selective guarantees, exact radius, drift/BBSE/similarity, CLI round trip ``` ## Caveats - **Every guarantee assumes exchangeability** between the calibration data and future data. Under drift they no longer hold, which is why the drift monitor exists. If drift fires, recalibrate on fresh labelled data. For pure label shift, weighted conformal methods exist (e.g. Podkopaev & Ramdas, 2021); they are not implemented here. - **The default calibration split (2 × 10%) is small for small datasets.** Conformal still holds for any n (sets just get bigger or become all-inclusive). Abstention certificates and Mondrian sets need a few hundred held-out points, or ≥ 1/α per class. - **TF-IDF is a baseline text representation.** For real text tools, embed with a sentence encoder and use `--embeddings`. - **`bundle.joblib` is a pickle.** Only load model directories you created yourself.