model logreg · 10 classes · split sizes {'train': 718, 'calib_fit': 359, 'calib_conf': 360, 'test': 360} · calibration: temperature
0.961test accuracy0.017test ECE0.919APS coverage (target 0.90)1.06APS avg set sizeInternal 80/20 split of the training rows; sorted by validation log-loss (a proper scoring rule).
(none)
Each calibrator is fitted on calib_fit and evaluated on test. ECE: binned |accuracy - confidence| (15 bins). NLL and Brier are proper scores (lower is better).
| method | accuracy | ece | nll | brier | mean_confidence |
|---|---|---|---|---|---|
| none | 0.961 | 0.032 | 0.161 | 0.068 | 0.938 |
| temperature | 0.961 | 0.017 | 0.160 | 0.066 | 0.954 |
| platt | 0.944 | 0.031 | 0.907 | 0.094 | 0.969 |
| isotonic | 0.942 | 0.046 | 1.334 | 0.102 | 0.983 |
Thresholds from calib_conf (n=360); coverage measured on test. Guarantee: marginal coverage ≥ 0.90 (and ≤ 0.903 for continuous scores) on average over calibration draws, if data are exchangeable. Per-class coverage is not guaranteed by the marginal method; Mondrian conformal fixes this per class at the cost of larger sets.
| method | coverage | avg_set_size | singletons | empty | worst_class_cov |
|---|---|---|---|---|---|
| lac | 0.900 | 0.911 | 0.911 | 0.089 | 0.714 |
| lac_mondrian | 0.914 | 0.975 | 0.942 | 0.042 | 0.800 |
| aps | 0.919 | 1.058 | 0.839 | 0.061 | 0.857 |
| aps_mondrian | 0.928 | 1.139 | 0.792 | 0.053 | 0.857 |
Answer only if confidence ≥ 0.8250. On calib_conf: 330 answered, 5 wrong, simultaneous upper bound on selective risk 0.0492 ≤ target 0.05 (Clopper-Pearson, Bonferroni over 53 fixed thresholds, δ=0.05).
On test: coverage 0.914, selective risk 0.0152. AURC 0.0043.
Guaranteed: with prob. ≥ 1-δ over the held-out draw, the true error rate among answered inputs is ≤ target, for future data from the same distribution. Not guaranteed: per-input correctness, the realised rate on a small batch, the fraction answered, or anything under drift.
Exact L2 certificate for the logistic-regression baseline in its preprocessed features (standardised numerics / one-hot / TF-IDF): no perturbation of norm below the radius can change its predicted label. Clean accuracy 0.961; median certified radius 1.320.
Empirical stability of the final model (logreg) under random noise (numeric: Gaussian with sd = σ × feature sd; text: each token dropped with prob. σ). Random noise is much weaker than adversarial noise: this is a smoke test, not a certificate.
| sigma | agreement | stable_all |
|---|---|---|
| 0.050 | 0.998 | 0.994 |
| 0.100 | 0.988 | 0.972 |
| 0.200 | 0.981 | 0.964 |
| 0.500 | 0.941 | 0.869 |
Generated by clfkit. Raw numbers: summary.json.