clfkit report

model hgb · 4 classes · split sizes {'train': 1800, 'calib_fit': 300, 'calib_conf': 300, 'test': 600} · calibration: temperature

0.958test accuracy0.013test ECE0.925APS coverage (target 0.90)1.05APS avg set size

1. Model comparison

Internal 80/20 split of the training rows; sorted by validation log-loss (a proper scoring rule).

model	val_accuracy	val_log_loss	val_ece	fit_seconds
hgb	0.939	0.280	0.051	0.987
logreg	0.908	0.442	0.200	0.128
rf	0.933	0.453	0.251	0.920

2. Calibration

Each calibrator is fitted on calib_fit and evaluated on test. ECE: binned |accuracy - confidence| (15 bins). NLL and Brier are proper scores (lower is better).

method	accuracy	ece	nll	brier	mean_confidence
none	0.958	0.024	0.145	0.064	0.980
temperature	0.958	0.013	0.128	0.062	0.966
platt	0.963	0.012	0.130	0.061	0.967
isotonic	0.958	0.022	0.356	0.064	0.969

3. Conformal prediction sets

Thresholds from calib_conf (n=300); coverage measured on test. Guarantee: marginal coverage ≥ 0.90 (and ≤ 0.903 for continuous scores) on average over calibration draws, if data are exchangeable. Per-class coverage is not guaranteed by the marginal method; Mondrian conformal fixes this per class at the cost of larger sets.

method	coverage	avg_set_size	singletons	empty	worst_class_cov
lac	0.903	0.915	0.915	0.085	0.868
lac_mondrian	0.912	0.927	0.927	0.073	0.875
aps	0.925	1.047	0.858	0.058	0.889
aps_mondrian	0.925	1.050	0.855	0.058	0.882

4. Abstention (selective classification)

Answer only if confidence ≥ 0.5000. On calib_conf: 293 answered, 4 wrong, simultaneous upper bound on selective risk 0.0498 ≤ target 0.05 (Clopper-Pearson, Bonferroni over 53 fixed thresholds, δ=0.05).

On test: coverage 0.992, selective risk 0.0353. AURC 0.0037.

Guaranteed: with prob. ≥ 1-δ over the held-out draw, the true error rate among answered inputs is ≤ target, for future data from the same distribution. Not guaranteed: per-input correctness, the realised rate on a small batch, the fraction answered, or anything under drift.

5. Robustness

Exact L2 certificate for the logistic-regression baseline in its preprocessed features (standardised numerics / one-hot / TF-IDF): no perturbation of norm below the radius can change its predicted label. Clean accuracy 0.960; median certified radius 0.088.

Empirical stability of the final model (hgb) under random noise (numeric: Gaussian with sd = σ × feature sd; text: each token dropped with prob. σ). Random noise is much weaker than adversarial noise: this is a smoke test, not a certificate.

sigma	agreement	stable_all
0.050	0.987	0.966
0.100	0.975	0.940
0.200	0.964	0.926
0.500	0.873	0.704

6. Drift check

DRIFT DETECTED (level 0.01; reference n=600, new n=600). Alarms: {"features": true, "scores": false, "c2st": true}

Score drift (KS on confidence): stat 0.073, p=7.94e-02; mean confidence 0.966 → 0.962.
Classifier two-sample test: held-out accuracy 0.648 (chance 0.5), p=1.68e-13.

feature	test	statistic	p	p_holm
text[oov_rate]	KS	0.99	0	0
channel	chi2	2.12	0.347	1
text[length]	KS	0.0367	0.815	1
n_prev_tickets	KS	0.035	0.856	1

Label-shift estimate (valid only if p(x|y) is unchanged; confusion-matrix condition number 1.1):

class	reference	estimated_new (BBSE)	predicted_new
account	0.252	0.248	0.238
billing	0.252	0.239	0.248
shipping	0.240	0.252	0.247
technical	0.257	0.260	0.267

Generated by clfkit. Raw numbers: summary.json.