William Chang /Projects/Classifier Toolkit Back to project

How it works

Classifier Toolkit

A classifier's accuracy says nothing about whether you can trust one particular answer. clfkit turns held-out data into statements you can rely on: calibrated probabilities, label sets with a coverage guarantee, and an error bound for the answers it chooses to give. Below are the six ideas that make it work, and how the browser demo reproduces them, each with the few lines of code that carry it.

  • Statistical guarantees
  • Uncertainty quantification
  • Hypothesis testing
  • numpy & scikit-learn

From a CSV to a prediction you can act on

  1. Splitfour disjoint parts, each with one job
  2. Fitcompare three models, refit the best
  3. Calibrateone temperature, fitted on its own split
  4. Certifyconformal quantiles and the abstain threshold
  5. Watchtest each new batch for drift

The ideas

  1. 01Give every guarantee its own data
  2. 02Fix confidence with one number
  3. 03Return a set, not a guess
  4. 04Say “I don't know” with a bound
  5. 05Notice when the world changes
  6. 06An exact radius, checked by tests
  7. 07The same maths in the browser
  8. →What's in the toolkit
  9. →Results
  10. →Where it fits
  11. →Limits
  12. →Reproduce
  13. →Read further
01

Give every guarantee its own data

The four-way split

Every guarantee on this page has the same fine print: it holds only if the numbers it is computed from were not used to fit anything else. So clfkit train cuts the rows into four parts up front: 60% to train and select the model, 10% to fit the calibrator, 10% for the conformal and abstention thresholds, and 20% for an honest test that also serves as the drift reference.

clfkit/data.pylines 197–215
def make_split(
    y: np.ndarray, test_size: float = 0.2, calib_size: float = 0.2, seed: int = 0
) -> Split:
    """Stratified train / calib_fit / calib_conf / test split (fractions of all rows).

    ``calib_size`` is split equally between ``calib_fit`` and ``calib_conf``.
    """
    idx = np.arange(len(y))
    rest, test = train_test_split(
        idx, test_size=test_size, random_state=seed, stratify=_safe_stratify(y)
    )
    rel_cal = calib_size / (1.0 - test_size)
    train, cal = train_test_split(
        rest, test_size=rel_cal, random_state=seed + 1, stratify=_safe_stratify(y[rest])
    )
    cal_fit, cal_conf = train_test_split(
        cal, test_size=0.5, random_state=seed + 2, stratify=_safe_stratify(y[cal])
    )
    return Split(np.sort(train), np.sort(cal_fit), np.sort(cal_conf), np.sort(test))
  1. The test split is carved off first and never touched again until evaluation.
  2. The calibration share is a fraction of all rows, so it is rescaled to a fraction of what remains after the test split.
  3. The calibration rows are halved: one half fits the calibrator, the other half computes the guarantees. Every split is stratified, so rare classes appear in each part.
clfkit/pipeline.pylines 137–141
        P_calfit_raw = proba(split.calib_fit)
        cal = CALIBRATORS[calibration]().fit(P_calfit_raw, y[split.calib_fit])
        P_conf = cal.transform(proba(split.calib_conf))
        P_test_raw = proba(split.test)
        P_test = cal.transform(P_test_raw)
  1. The calibrator only ever sees calib_fit.
  2. The conformal and abstention data are passed through that already-fitted calibrator, so the quantiles are computed on probabilities the calibrator was not tuned on.
02

Fix confidence with one number

Temperature scaling

A model is calibrated if, among the inputs where it says 70%, it is right 70% of the time. Temperature scaling (Guo et al., 2017) divides the log-probabilities by a single number T before the softmax. T above 1 softens an overconfident model, T below 1 sharpens an underconfident one. With only one parameter it is hard to overfit, and since dividing by a positive number keeps the order of the scores, it never changes the predicted label.

pT = softmax(log p / T)

clfkit/calibration.pylines 98–119
class TemperatureScaler:
    """Single-parameter temperature scaling: p_T = softmax(log p / T)."""

    name = "temperature"

    def __init__(self):
        self.T_ = 1.0

    def fit(self, probs: np.ndarray, y: np.ndarray):
        z = _logits(probs)
        idx = np.arange(len(y))

        def loss(log_t):
            return -np.mean(log_softmax(z / np.exp(log_t), axis=1)[idx, y])

        # Search over log T so that T stays positive; [1/20, 20] is plenty.
        res = minimize_scalar(loss, bounds=(np.log(0.05), np.log(20.0)), method="bounded")
        self.T_ = float(np.exp(res.x))
        return self

    def transform(self, probs: np.ndarray) -> np.ndarray:
        return softmax(_logits(probs) / self.T_, axis=1)
  1. Log-probabilities work as logits, because softmax ignores a constant shift. That lets the scaler sit on top of any model that outputs probabilities.
  2. The loss is the negative log-likelihood of the true labels: a proper scoring rule, so it is minimised by honest probabilities.
  3. The search runs over log T, which keeps T positive without a constraint. A bounded 1-D search is all a single parameter needs.
  4. Applying it is one line. Platt scaling (a per-class logistic fit on the logit) and isotonic scaling (a per-class monotone fit) are in the same file behind the same fit / transform interface. Isotonic is more flexible but needs more data.

The headline number is ECE (expected calibration error): bin the predictions by confidence and average the gap between accuracy and confidence. It depends on the binning and is biased, so it is good for comparing methods on the same data, not as an absolute truth. The report gives NLL and Brier score next to it; those are proper scoring rules.

Result

On the 3,000 synthetic support tickets the fitted T is 1.38 and test ECE (expected calibration error) drops from 0.024 to 0.013. On the digits model T is 0.82: there the logistic regression was underconfident, and temperature scaling sharpens it.

03

Return a set, not a guess

Split conformal prediction

Conformal prediction gives each label a score for how badly it fits the input, finds a cutoff on held-out data, and returns every label under the cutoff. If the held-out and future data are exchangeable (roughly, drawn the same way), the set contains the true label at least 1 − α of the time. That holds for any model. A bad model just produces bigger sets.

q = the ⌈(n+1)(1−α)⌉-th smallest calibration score,   C(x) = {y : s(x, y) ≤ q}

clfkit/conformal.pylines 50–60
def conformal_quantile(scores: np.ndarray, alpha: float) -> float:
    """The finite-sample-corrected (1-alpha) quantile used by split conformal.

    Returns +inf when n is too small for the requested alpha
    (ceil((n+1)(1-alpha)) > n), i.e. "include everything".
    """
    n = len(scores)
    k = math.ceil((n + 1) * (1 - alpha))
    if n == 0 or k > n:
        return float("inf")
    return float(np.sort(scores)[k - 1])
  1. The (n+1) instead of n is the finite-sample correction. It treats the future point as one more score in the ranking, which is what makes the guarantee exact rather than asymptotic.
  2. With too few points for the requested α, no finite cutoff is valid, so the set becomes every label. The code returns “include everything” rather than an overconfident answer.
  3. Otherwise the cutoff is just one sorted score. No model internals are involved.

clfkit has two scores. LAC (s = 1 − py) gives the smallest sets on average but under-covers hard classes. APS adds up the probability of the labels ranked above y, plus a random share of py, so sets grow on hard inputs and shrink on easy ones. Empty sets are legitimate: when the model is more accurate than 1 − α, an empty set is how the method avoids over-covering, which is why the average set size can be below 1. In a tool, treat an empty set as “abstain”.

The guarantee is marginal: averaged over all inputs. A model can hit 90% overall while missing one hard class far more often. Mondrian conformal fixes this by computing one cutoff per class from that class's calibration points only, at the cost of larger sets.

clfkit/conformal.pylines 101–113
    def fit(self, probs_cal: np.ndarray, y_cal: np.ndarray):
        rng = np.random.default_rng(self.seed)
        n, K = probs_cal.shape
        u = rng.uniform(size=n)
        s = all_class_scores(probs_cal, self.score, u)[np.arange(n), y_cal]
        self.n_cal_ = n
        self.n_classes_ = K
        if self.mondrian:
            self.qhat_ = np.array(
                [conformal_quantile(s[y_cal == k], self.alpha) for k in range(K)]
            )
        else:
            self.qhat_ = np.full(K, conformal_quantile(s, self.alpha))
  1. Each calibration point is scored only at its true label. u is the per-point random draw that APS, the adaptive score, uses to break ties.
  2. Mondrian: one quantile per class k, from the scores of points whose label is k. This gives coverage of at least 1 − α within every class.
  3. The plain version uses one shared cutoff, copied K times so both cases are applied by the same comparison at prediction time.
The gap it closes

On handwritten digits at a 90% target, averaged over 50 re-splits, plain LAC covers digit 8 only 76.6% of the time while its best class gets 98.1%. Mondrian brings every class to between 89.8% and 93.2%.

04

Say “I don't know” with a bound

Certified selective classification

The rule is simple: answer if confidence ≥ λ, otherwise hand the input to a human. The hard part is choosing λ so that the error rate among answered inputs is guaranteed below a target. For a fixed λ, the errors among the answered held-out points are binomial, so an exact Clopper–Pearson bound applies. The catch is that λ is picked after looking at the data, which would normally void the bound.

clfkit/selective.pylines 48–61
# Fixed grid: dense near 1 where high-precision thresholds live. Chosen without
# looking at data, which is what makes the union bound valid.
DEFAULT_GRID = np.unique(
    np.concatenate([np.round(np.arange(0.0, 0.95, 0.025), 4), 1 - np.logspace(-1.3, -3, 15)])
)


def clopper_pearson_upper(k: int, n: int, delta: float) -> float:
    """One-sided (1 - delta) upper confidence bound on a binomial proportion."""
    if n == 0:
        return 1.0
    if k >= n:
        return 1.0
    return float(beta.ppf(1 - delta, k + 1, n - k))
  1. The candidate thresholds are fixed in the code, before any data is seen. That is what lets the union bound below cover a choice made after looking.
  2. 53 thresholds: a coarse grid from 0 to 0.925, plus 15 points packed close to 1, where high-confidence thresholds live.
  3. The one-sided Clopper–Pearson upper bound: a Beta quantile, exact for binomial counts at any sample size.
clfkit/selective.pylines 89–105
    m = len(grid)
    rows = []
    for lam in grid:
        sel = conf >= lam
        n_sel = int(sel.sum())
        k = int((~correct[sel].astype(bool)).sum())
        rows.append(
            {
                "threshold": float(lam),
                "n_selected": n_sel,
                "coverage": n_sel / len(conf),
                "errors": k,
                "empirical_risk": k / n_sel if n_sel else float("nan"),
                "risk_upper_bound": clopper_pearson_upper(k, n_sel, delta / m),
            }
        )
    return rows
  1. For each threshold: which held-out points would be answered, and how many of those are wrong.
  2. Bonferroni: each bound is computed at δ/53. Then all 53 bounds hold at once with probability at least 1 − δ, so taking the lowest certified threshold (the one that answers the most) is still valid.

What it promises: with probability at least 1 − δ, the true error rate among answered inputs is at most the target, for future data from the same distribution. It promises nothing about a single input, about the error you happen to see on one small batch, about how many inputs get answered, or about anything after drift.

What data it takes

Even with zero errors, the bound cannot go below 1 − (δ/53)1/n. To certify 2% risk you need about 349 answered held-out points. The breast cancer model has 86, so it is certified at 15% instead, and if no threshold certifies, predict abstains on everything and says so. clfkit similarity reuses this same machinery to pick a cosine cutoff with a precision lower bound.

05

Notice when the world changes

Drift tests on unlabelled batches

Every guarantee above assumes future data looks like the calibration data. clfkit drift checks that on a new batch without labels, in three families: a KS or chi-square test per feature (Holm-adjusted, since there are many), a KS test on the model's confidence, and a classifier two-sample test. The alarm fires at level 0.01, split three ways by Bonferroni. The two-sample test is the one that catches changes no single feature shows.

clfkit/drift.pylines 112–125
    rng = np.random.default_rng(seed)
    n = min(Z_ref.shape[0], Z_new.shape[0], max_per_side)
    ia = rng.choice(Z_ref.shape[0], n, replace=False)
    ib = rng.choice(Z_new.shape[0], n, replace=False)
    import scipy.sparse as sp

    stack = sp.vstack if sp.issparse(Z_ref) else np.vstack
    Z = stack([Z_ref[ia], Z_new[ib]])
    lab = np.r_[np.zeros(n), np.ones(n)]
    tr, te = train_test_split(np.arange(2 * n), test_size=0.5, random_state=seed, stratify=lab)
    clf = LogisticRegression(C=1.0, max_iter=2000).fit(Z[tr], lab[tr])
    correct = int((clf.predict(Z[te]) == lab[te]).sum())
    res = binomtest(correct, len(te), 0.5, alternative="greater")
    return {"accuracy": correct / len(te), "n_test": int(len(te)), "p": float(res.pvalue)}
  1. Both samples are cut to the same size, so a classifier that learned nothing scores exactly 50%.
  2. Train a logistic regression to tell “old” rows from “new” on half the data, and count how many of the other half it gets right.
  3. If nothing changed, that count is Binomial(n, ½), so a one-sided binomial test gives an exact p-value with no approximation.

When only the class mix moved (more billing tickets, say), Black Box Shift Estimation (Lipton, Wang & Smola, 2018) recovers the new proportions from the model's predictions alone. If C is the joint confusion matrix on labelled reference data and μ is the share of each predicted label in the new batch, then C·w = μ, where w is the ratio of new to old class frequencies.

clfkit/drift.pylines 135–142
def bbse_estimate(C: np.ndarray, p_ref: np.ndarray, y_pred_new: np.ndarray) -> dict:
    """Estimate new class proportions under label shift (hard-prediction BBSE)."""
    K = C.shape[0]
    mu = np.bincount(y_pred_new, minlength=K) / len(y_pred_new)
    w, *_ = np.linalg.lstsq(C, mu, rcond=None)
    w = np.clip(w, 0.0, None)
    q = w * p_ref
    q = q / q.sum() if q.sum() > 0 else np.full(K, 1.0 / K)
  1. The only thing measured on the new batch: how often the model predicts each class.
  2. Solve C·w = μ by least squares and clip negative weights, which can appear from sampling noise.
  3. Reweight the old class frequencies and renormalise. The function also reports C's condition number, since a near-singular C makes the estimate unreliable.
Result

On three 600-row ticket batches: the unchanged batch raised no alarm (two-sample accuracy 0.485, p = 0.78). The label-shift batch did (p = 6e-9), and BBSE estimated billing at 0.57 against a true 0.55. The new-vocabulary batch fired on the out-of-vocabulary rate, and BBSE correctly left the mix at about 0.25 each.

06

An exact radius, checked by tests

Robustness certificates for linear models

How far can an input move before the prediction flips? For a linear model the answer is exact. Class c wins wherever its score beats every other class, which is an intersection of half-spaces, so the distance to losing is the distance to the nearest of their boundaries. clfkit always fits a logistic-regression baseline, and this certificate covers its predicted label, measured in its feature space (standardised numerics, one-hot categories, TF-IDF).

r(z) = mink≠c (fc(z) − fk(z)) / ‖wc − wk‖

clfkit/robustness.pylines 60–76
    F = np.asarray(Z @ W.T) + b  # (n, K)
    pred = F.argmax(1)
    n, K = F.shape
    # Pairwise ||w_i - w_j|| computed once (K x K) - avoids an (n x d) array.
    G = W @ W.T
    sq = np.diag(G)
    pair_norm = np.sqrt(np.clip(sq[:, None] + sq[None, :] - 2 * G, 0.0, None))
    radius = np.full(n, np.inf)
    runner = np.zeros(n, dtype=int)
    for k in range(K):
        norm = pair_norm[pred, k]
        gap = F[np.arange(n), pred] - F[:, k]
        with np.errstate(divide="ignore", invalid="ignore"):
            r = np.where((k != pred) & (norm > 0), gap / norm, np.inf)
        better = r < radius
        radius = np.where(better, r, radius)
        runner = np.where(better, k, runner)
  1. Scores for every row and class in one matrix product; works for dense or sparse (TF-IDF) inputs.
  2. ‖wi − wj‖ for every pair of classes from the Gram matrix W·Wᵀ, a K × K array, instead of building a difference vector per row.
  3. The formula above, term by term: score gap over weight-difference norm, for every row at once.
  4. Keep the smallest distance, and remember which class was the runner-up. That gives the direction of the fastest flip, which the tests use.

“Exact” is a strong claim, so the test suite checks it both ways for 60 points in a 2-class and a 4-class model:

tests/test_robustness.pylines 28–39
    rng = np.random.default_rng(0)
    for i in range(len(Z)):
        r = rad[i]
        # 1) Random perturbations strictly inside the radius never flip the label.
        U = rng.normal(size=(200, Z.shape[1]))
        U /= np.linalg.norm(U, axis=1, keepdims=True)
        scales = r * rng.uniform(0, 0.999, size=(200, 1))
        assert np.all(_predict(W, b, Z[i] + scales * U) == pred[i])
        # 2) The worst-case direction, just inside: still the same label.
        assert _predict(W, b, (Z[i] + 0.999 * r * D[i])[None])[0] == pred[i]
        # 3) Just beyond the radius along that direction: label flips (tight).
        assert _predict(W, b, (Z[i] + 1.001 * r * D[i])[None])[0] != pred[i]
  1. 200 random directions per point, each scaled to land strictly inside the radius. None may change the label.
  2. The worst direction, at 99.9% of the radius: still the same label.
  3. The same direction at 100.1% of the radius: the label must flip. This is what shows the radius is tight, not just safe.
Scope

The certificate covers small L2 changes in feature space. Swapping one word moves a TF-IDF vector a long way, so that is not covered. For the non-linear models clfkit only measures how often predictions survive random noise, which is a smoke test, not a certificate.

07

The same maths in the browser

How the demo on the project page works

The demo doesn't run a model in the browser. A Python script trains the three demo models exactly as the end-to-end demo does, then exports each one's uncalibrated probabilities and labels for the three held-out splits to JSON. Everything after that (fitting the temperature, conformal cutoffs, the abstention certificate, the drift tests) is redone in plain JavaScript in demo/clfkit.js, a line-by-line port of the Python functions with no dependencies. Your own CSV goes through the same functions; it is shuffled and split 25% / 25% / 50%, the proportions clfkit uses for its held-out parts.

APS uses one random number per row to break ties. To get the same sets as Python, the export includes the exact draws clfkit makes:

examples/export_demo_data.pylines 59–61
    # The uniforms SplitConformal draws: seed for calibration, seed + 7919 for test.
    u_cal = np.random.default_rng(b.seed).uniform(size=len(b.y_conf))
    u_test = np.random.default_rng(b.seed + 7919).uniform(size=len(b.y_test))
  1. SplitConformal seeds its calibration draws with the bundle seed and its test draws with seed + 7919. Exporting both streams means JavaScript needn't reproduce numpy's random generator. For an uploaded CSV the page draws its own from a small seeded generator.

The conformal part ports almost word for word. The one thing to get right is doing the floating-point operations in the same order, so that scores tie exactly where Python's tie:

demo/clfkit.jslines 110–129
  // conformal_quantile: the ceil((n+1)(1-alpha))-th smallest score; +Infinity if n is too small
  function conformalQuantile(scores, alpha) {
    var n = scores.length, k = Math.ceil((n + 1) * (1 - alpha));
    if (n === 0 || k > n) return Infinity;
    var s = Float64Array.from(scores).sort();
    return s[k - 1];
  }
  // all_class_scores: LAC s = 1 - p_y; APS s = mass ranked above y + u * p_y
  function classScores(p, score, u) {
    var K = p.length, out = new Array(K), j;
    if (score === "lac") { for (j = 0; j < K; j++) out[j] = 1 - p[j]; return out; }
    var order = []; for (j = 0; j < K; j++) order.push(j);
    order.sort(function (a, b) { return p[b] - p[a]; }); // stable, like argsort(-p, kind="stable")
    var cum = 0;
    for (var r = 0; r < K; r++) {
      var v = p[order[r]]; cum += v;
      out[order[r]] = (cum - v) + u * v;
    }
    return out;
  }
  1. The same finite-sample index as conformal_quantile. The slider sets α = (100 − coverage)/100, which is the same double as Python's 0.1, 0.05 and so on, so k comes out the same.
  2. A JavaScript sort with a comparator is stable, which matches argsort(-p, kind="stable") when two classes have the same probability.
  3. A running sum, minus the current term, plus u times the current term: the same three operations, in the same order, as numpy's cumsum(sorted_p) - sorted_p + u * sorted_p.

The abstention certificate needs scipy.stats.beta.ppf. The port computes the regularised incomplete beta with a continued fraction and inverts it by bisection until the interval can't shrink any further:

demo/clfkit.jslines 205–218
  function betaPpf(q, a, b) { // inverse of I_x(a, b) by bisection, to double precision
    var lo = 0, hi = 1;
    for (var i = 0; i < 200; i++) {
      var mid = 0.5 * (lo + hi);
      if (mid === lo || mid === hi) break;
      if (betainc(a, b, mid) < q) lo = mid; else hi = mid;
    }
    return 0.5 * (lo + hi);
  }
  // clopper_pearson_upper: one-sided (1 - delta) upper bound on a binomial proportion
  function clopperPearsonUpper(k, n, delta) {
    if (n === 0 || k >= n) return 1;
    return betaPpf(1 - delta, k + 1, n - k);
  }
  1. Bisection is slow by numerical standards (about 60 steps), but it can't fail, and 53 thresholds × 60 steps is still instant.
  2. The same edge cases as clopper_pearson_upper: no answered points, or all wrong, gives a bound of 1.

The drift tests need scipy's exact two-sample Kolmogorov–Smirnov p-value. scipy counts lattice paths and returns 1 minus the share that stays inside the band, which loses precision for tiny p. The port instead walks the same lattice with probabilities and adds up the mass that leaves the band, so small p-values stay accurate:

demo/clfkit.jslines 245–273
  // scipy.stats.ks_2samp (two-sided, exact): D, then P(D >= d) by walking the lattice of
  // merge orders and adding up the probability of every path that leaves the band.
  function ks2samp(a, b) {
    var x = Float64Array.from(a).sort(), y = Float64Array.from(b).sort(), n1 = x.length, n2 = y.length;
    var i = 0, j = 0, dInt = 0; // D * n1 * n2 as an integer, so the band test below is exact
    while (i < n1 || j < n2) {
      var v = i < n1 && (j >= n2 || x[i] <= y[j]) ? x[i] : y[j];
      while (i < n1 && x[i] === v) i++;
      while (j < n2 && y[j] === v) j++;
      dInt = Math.max(dInt, Math.abs(i * n2 - j * n1));
    }
    var D = dInt / (n1 * n2);
    if (dInt === 0) return { statistic: 0, p: 1 };
    var row = new Float64Array(n2 + 1), pOut = 0;
    row[0] = 1;
    for (i = 0; i <= n1; i++) {
      var next = new Float64Array(n2 + 1);
      for (j = 0; j <= n2; j++) {
        var m = row[j];
        if (!m) continue;
        var left = n1 + n2 - i - j;
        if (!left) continue;
        if (i < n1) { var pr = m * (n1 - i) / left; if (Math.abs((i + 1) * n2 - j * n1) >= dInt) pOut += pr; else next[j] += pr; }
        if (j < n2) { var pc = m * (n2 - j) / left; if (Math.abs(i * n2 - (j + 1) * n1) >= dInt) pOut += pc; else row[j + 1] += pc; }
      }
      row = next;
    }
    return { statistic: D, p: Math.min(1, pOut) };
  }
  1. The statistic is kept as the integer |i·n₂ − j·n₁| at each distinct value, after all tied values are counted. This is exactly what scipy rounds D·lcm to, so the band test has no floating-point fuzz.
  2. Merging two sorted samples in a random order is a random walk through an n₁ × n₂ grid. From (i, j) the next item comes from the first sample with probability (n₁ − i)/(items left).
  3. A step that lands outside the band adds its probability to the p-value and stops there. Every term is positive, so nothing cancels. Even for the 25%-new-words batch, where p is around 1e-317, the port agrees with scipy.

The per-feature tests also need a chi-square tail (a regularised incomplete gamma), Holm's adjustment, and BBSE's least-squares solve, done through a small Jacobi eigen-decomposition so the condition number comes for free.

Checked against Python. A Node script ran the port on the exported data and a Python script ran clfkit on the same arrays: 3 models × 20 conformal settings (LAC and APS, with and without Mondrian, α from 0.01 to 0.2), 12 abstention targets each, and 6 drift batches. Coverage, set sizes, worst-class coverage and every certified threshold were identical. Fitted temperatures agreed to a relative 2e-12 (tickets 1.380165780, breast cancer 0.324255861, digits 0.819628144), cutoffs to 9e-13, ECE to 1e-13, Clopper–Pearson bounds to 2e-15, KS and chi-square p-values to a relative 3e-15, and BBSE estimates to 4e-16.

What the browser leaves out. The drift panel runs the per-feature tests, the confidence test and BBSE, but not the classifier two-sample test, because that needs the model's TF-IDF feature matrix for every row. The alarm still uses clfkit's level 0.01 split three ways, so it is slightly more conservative than it needs to be. That test matters: on a 500-ticket batch with billing at 55%, the browser's checks don't fire (past-tickets p = 0.016, 0.065 after Holm), while in Python the two-sample test catches the same kind of shift at p = 6e-9. BBSE still recovers the mix (0.55 estimated for billing). The drift panel uses the temperature Python fitted, not the slider's. Platt and isotonic calibration are not ported.

→

What's in the toolkit

One question per piece

A small Python toolkit and command-line tool. It works on any CSV, detecting numeric, categorical and free-text columns automatically (text becomes TF-IDF features), or on precomputed embeddings from any model. It compares three model types, calibrates the winner, and writes a self-contained HTML report you can email.

Question a tool must answerclfkit piece
Is “90% confident” really 90%?calibration: reliability diagram, ECE, temperature / Platt / isotonic
Which labels are plausible, with a guarantee?conformal prediction sets: LAC, APS, Mondrian (per class)
When should it say “I don't know”?selective classification with a certified error bound
Has the incoming data changed?per-feature tests, score drift, classifier two-sample test, label-shift estimate
How brittle is a prediction?exact L2 certificates for linear models, noise stability for any model
When is a cosine similarity “similar”?null distribution and a precision-certified threshold

Similarity. Two random directions in d dimensions have a cosine with standard deviation exactly 1/√d, so in 768 dimensions a cosine of 0.2 is 5.5 standard deviations out. Real embeddings share a common direction, though, so clfkit measures the null on known-unrelated pairs instead, and picks the lowest cosine cutoff whose precision lower bound meets the target, with the same Bonferroni and Clopper–Pearson machinery as abstention.

→

Results

From the end-to-end demo

Synthetic support tickets. 3,000 tickets (free text, channel, a number), 4 teams. Default clfkit train, evaluated on the held-out test split.

95.8%test accuracy
0.013calibration error (ECE), down from 0.024
90.3%conformal coverage, target 90%
≤ 5%certified error when it answers; answers 99.2%

Drift on three 600-row batches of tickets (the full Python checks, including the two-sample test):

batchalarmwhat firedestimated class mix
Same distributionnonothing (two-sample test p = 0.78)≈ 0.25 each (true 0.25)
Label shift: billing 0.55yestwo-sample test p = 6e-9; n_prev_ticketsbilling 0.57, others 0.12–0.17
New vocabulary, 25% unseen wordsyesout-of-vocabulary rate p ≈ 0; two-sample p = 2e-13unchanged ≈ 0.25 (correct)

Digits. The per-class coverage gap and how Mondrian closes it are in section 03.

Breast cancer. APS reaches 94.7% coverage at α = 0.05 with an average set size of 1.04. A 15% risk target is certified at threshold 0.80, giving 4.5% test error while answering 98% of inputs. A 2% target can't be certified with only 86 held-out points (see section 04).

Embeddings. On synthetic anisotropic embeddings, every unrelated pair has cosine above 0.2 (mean 0.62), and no threshold certifies 95% precision. After mean-centring, cos ≥ 0.12 gives precision 0.983 with a lower bound of 0.962.

Sample reports. These were generated by running the demo, and the numbers match the README.

Support ticketsText and tabular features, model comparison, calibration, conformal, abstention, three drift checks.197 KB Breast cancerBinary, small data: what can and can't be certified.202 KB Digits10 classes through the embeddings loader; LAC vs Mondrian per class.195 KB
→

Where it fits

Tools it was built for

  • Email or ticket triage with abstain-to-human. Auto-file only when it doesn't abstain; the certified risk caps the misfiling rate. Route a ticket to two teams when its prediction set has two.
  • Document classifier with a drift alarm. A weekly clfkit drift catches a new vendor template, and the label-shift estimate says whether only the document mix changed.
  • Quality gate for an LLM labeller. Calibrate the LLM's scores on a small human-labelled set, then use conformal sets or abstention to decide which items still need review. (The CSV upload on the project page does exactly this with any model's scores.)
  • “Is this a duplicate?” Get a cosine cutoff with a precision lower bound instead of guessing 0.8.
→

Limits

What the guarantees don't cover

  • Exchangeability. Every guarantee assumes the calibration data and future data are exchangeable. Under drift they no longer hold, which is why the drift monitor exists: if it fires, recalibrate on fresh labelled data.
  • Marginal, not per input. Conformal coverage is an average over inputs and calibration draws; only Mondrian gives it per class. The abstention bound is about the population error rate, not any one answer or one small batch.
  • Small data. Conformal holds for any n, but sets get bigger or include everything. Certifying a low error rate takes hundreds of held-out points, and Mondrian needs at least 1/α − 1 per class.
  • Robustness scope. The exact radius covers small L2 changes to the logistic-regression baseline's input features, not word swaps and not the tree models.
  • The browser demo leaves out the classifier two-sample test and the Platt and isotonic calibrators (section 07).
→

Reproduce

Install, run the demo, rebuild the browser data

Install inside classifier-toolkit/; this gives the clfkit command (train, report, predict, drift, similarity):

pip install -e .
clfkit train data.csv --target label --text-col text --out model_dir
clfkit report model_dir

Run everything above (about 12 s) and the 20 tests:

python examples/demo_end_to_end.py
python -m pytest -q

Rebuild the JSON the browser demo loads (about 20 s):

python examples/export_demo_data.py      # writes demo/data/*.json
→

Read further

Where everything lives