model rf · 2 classes · split sizes {'train': 284, 'calib_fit': 85, 'calib_conf': 86, 'test': 114} · calibration: temperature
0.947test accuracy0.052test ECE0.947APS coverage (target 0.95)1.04APS avg set sizeInternal 80/20 split of the training rows; sorted by validation log-loss (a proper scoring rule).
| model | val_accuracy | val_log_loss | val_ece | fit_seconds |
|---|---|---|---|---|
| rf | 0.912 | 0.175 | 0.061 | 0.336 |
| logreg | 0.930 | 0.229 | 0.047 | 0.029 |
| hgb | 0.912 | 0.386 | 0.074 | 0.072 |
Each calibrator is fitted on calib_fit and evaluated on test. ECE: binned |accuracy - confidence| (15 bins). NLL and Brier are proper scores (lower is better).
| method | accuracy | ece | nll | brier | mean_confidence |
|---|---|---|---|---|---|
| none | 0.947 | 0.029 | 0.348 | 0.077 | 0.934 |
| temperature | 0.947 | 0.052 | 0.352 | 0.088 | 0.986 |
| platt | 0.947 | 0.049 | 0.355 | 0.089 | 0.986 |
| isotonic | 0.947 | 0.058 | 1.460 | 0.107 | 0.995 |
Thresholds from calib_conf (n=86); coverage measured on test. Guarantee: marginal coverage ≥ 0.95 (and ≤ 0.961 for continuous scores) on average over calibration draws, if data are exchangeable. Per-class coverage is not guaranteed by the marginal method; Mondrian conformal fixes this per class at the cost of larger sets.
| method | coverage | avg_set_size | singletons | empty | worst_class_cov |
|---|---|---|---|---|---|
| lac | 0.956 | 1.018 | 0.982 | 0.000 | 0.944 |
| lac_mondrian | 0.947 | 1.088 | 0.912 | 0.000 | 0.931 |
| aps | 0.947 | 1.035 | 0.895 | 0.035 | 0.944 |
| aps_mondrian | 0.991 | 1.149 | 0.851 | 0.000 | 0.976 |
Answer only if confidence ≥ 0.8000. On calib_conf: 77 answered, 2 wrong, simultaneous upper bound on selective risk 0.1382 ≤ target 0.15 (Clopper-Pearson, Bonferroni over 53 fixed thresholds, δ=0.05).
On test: coverage 0.982, selective risk 0.0446. AURC 0.0266.
Guaranteed: with prob. ≥ 1-δ over the held-out draw, the true error rate among answered inputs is ≤ target, for future data from the same distribution. Not guaranteed: per-input correctness, the realised rate on a small batch, the fraction answered, or anything under drift.
Exact L2 certificate for the logistic-regression baseline in its preprocessed features (standardised numerics / one-hot / TF-IDF): no perturbation of norm below the radius can change its predicted label. Clean accuracy 0.930; median certified radius 1.716.
Empirical stability of the final model (rf) under random noise (numeric: Gaussian with sd = σ × feature sd; text: each token dropped with prob. σ). Random noise is much weaker than adversarial noise: this is a smoke test, not a certificate.
| sigma | agreement | stable_all |
|---|---|---|
| 0.050 | 1.000 | 1.000 |
| 0.100 | 0.997 | 0.991 |
| 0.200 | 0.982 | 0.974 |
| 0.500 | 0.953 | 0.877 |
Generated by clfkit. Raw numbers: summary.json.