Classifier Toolkit
See how far you can trust a classifier: whether its confidence is honest, which answers are plausible, when it should say “I don't know”, and when new data stops looking like the old. Try it on three trained models, or on your own model's outputs.
Model
Loading…
One row per held-out example: a probability column per class, then the true label (p_cat,p_dog,label), or a single score for a yes/no model (score,label with labels 0/1). The file stays in your browser.
Is its confidence honest?
If bars sit on the dashed line, “70% sure” is right 70% of the time.
Changing T here also feeds panels 2 and 3, so you can see what bad calibration does to them.
after scalingas trainedrows per bin
Which answers are plausible?
Instead of one guess, a short list that contains the right answer as often as you ask.
How often each class's list holds the answer, on the test rows
Some test rows and their lists
When should it answer?
Set the error rate you can live with; clfkit finds when it can answer and still promise it.
certified bound (held-out rows)error seen on test rowschosen
Has the data changed?
Make a new batch of tickets and see whether the alarm notices, without any labels.
Checking for change needs fresh batches of inputs, which this demo has for the support-tickets model only.
real mix of the batchestimated without labelsreference
Everything runs in your browser, on held-out predictions exported from the Python package, and the numbers match it to about 12 digits. One difference: the change check leaves out one of clfkit's three tests, the one that needs the model's text features.
How it's built, results and limits: How it works →