Calibrating Stop Confidence with Isotonic Regression

A stop-detection score of 0.8 invites exactly one interpretation: that roughly eight such stops in ten are real. Almost no raw scorer delivers that. Scores built from heuristics — variance below a threshold, duration above another, a geofence hit — produce numbers that rank candidates sensibly and mean nothing in particular on an absolute scale.

That matters because downstream rules are written against the absolute scale. “Auto-approve above 0.9” is a policy that assumes calibration, and an uncalibrated 0.9 that is right seven times in ten quietly approves three wrong stops in every ten it touches.

Isotonic regression fixes this with a single fitted mapping and no assumptions beyond monotonicity. This page covers fitting it, measuring whether it helped, and keeping it fitted, extending the scoring work in confidence scoring for stop detection.

Compatibility and Configuration Requirements

Fitting the Calibrator

from __future__ import annotations

import numpy as np
from sklearn.isotonic import IsotonicRegression


def fit_calibrator(raw_scores: np.ndarray, is_real: np.ndarray) -> IsotonicRegression:
    """Map raw stop scores onto observed probabilities, monotonically.

    ``raw_scores`` are the scorer's own output on held-out candidates;
    ``is_real`` is the binary label. The fitted object is small enough to
    version alongside the scorer.
    """
    if len(raw_scores) < 1_000:
        raise ValueError(f"{len(raw_scores)} labels is too few for a non-parametric fit")
    cal = IsotonicRegression(y_min=0.0, y_max=1.0, out_of_bounds="clip", increasing=True)
    cal.fit(raw_scores, is_real.astype(float))
    return cal


def expected_calibration_error(p: np.ndarray, y: np.ndarray, bins: int = 10) -> float:
    """Mean absolute gap between predicted and observed, weighted by bin size."""
    edges = np.linspace(0, 1, bins + 1)
    idx = np.clip(np.digitize(p, edges) - 1, 0, bins - 1)
    err = 0.0
    for b in range(bins):
        m = idx == b
        if m.any():
            err += m.mean() * abs(p[m].mean() - y[m].mean())
    return float(err)

increasing=True is not a formality. It encodes the claim that a higher raw score should never mean a lower probability, and if the fit is materially improved by allowing a decreasing region, the scorer has a bug worth finding rather than calibrating around.

out_of_bounds="clip" handles scores outside the fitted range, which happens the first time a new device family produces a score the calibration set never contained. Clipping is the conservative choice; extrapolating a non-parametric fit is not.

Measuring Whether It Helped

Two numbers, reported together, say whether the calibration was worth doing.

Expected calibration error is the average gap between what the score claims and what is observed, weighted by how many candidates fall in each bin. It is the direct measure of the property being fixed, and it should fall substantially — typically from around 0.1 to under 0.03.

Brier score is the mean squared error of the probability. It moves less, because it also contains the scorer’s inherent discrimination, which calibration cannot improve. A Brier score that does not move at all while calibration error falls is the expected and healthy result.

What should not change is ROC area. Isotonic regression is monotonic, so it cannot reorder candidates. If ROC area moves, something other than calibration happened — usually the held-out set overlapping the tuning set.

from sklearn.metrics import brier_score_loss, roc_auc_score

def report(raw, calibrated, y) -> dict:
    return {
        "ece_raw": expected_calibration_error(raw, y),
        "ece_cal": expected_calibration_error(calibrated, y),
        "brier_raw": brier_score_loss(y, raw),
        "brier_cal": brier_score_loss(y, calibrated),
        "auc_raw": roc_auc_score(y, raw),
        "auc_cal": roc_auc_score(y, calibrated),   # must equal auc_raw
    }

Assert the last equality. It is a cheap check that the pipeline did what it claims, and it fires immediately if the calibrator was fitted on the wrong column.

Keeping It Fitted

A calibrator has a shelf life, because everything feeding it moves. New tracker hardware changes the variance distribution the scorer reads; a new depot changes the mix of round types; a change to the minimum-dwell threshold changes which candidates exist at all.

Refit monthly on a rolling window of labels, and monitor the calibration error of the live calibrator between refits. A rising error is the signal that the underlying score has drifted, and it usually appears weeks before anybody notices the downstream rules behaving oddly.

def monthly_refit(labels_window) -> IsotonicRegression:
    cal = fit_calibrator(labels_window["raw_score"].to_numpy(), labels_window["is_real"].to_numpy())
    ece = expected_calibration_error(cal.predict(labels_window["raw_score"]), labels_window["is_real"])
    if ece > 0.05:
        raise SystemExit(f"post-fit calibration error {ece:.3f} — the score itself has a problem")
    return cal

That guard matters. If a freshly fitted calibrator still cannot reach a low calibration error, the problem is not calibration — the score is non-monotonic in the label, and no monotonic mapping can fix it. That is a finding about the scorer, and it should stop the pipeline rather than ship a calibrator that papers over it.

Using a Calibrated Score

Once calibrated, the score supports things an uncalibrated one cannot.

Thresholds that mean something. “Auto-approve above 0.9” now approves a population that is right about nine times in ten, and that claim can be checked.

Expected counts. Summing calibrated probabilities over a day estimates the number of real stops without labelling any of them, which is a useful sanity check against the detector’s own count.

Cost-weighted decisions. With a genuine probability, the review threshold can be derived from the relative cost of a false accept and a false reject rather than chosen by feel.

def review_threshold(cost_false_accept: float, cost_false_reject: float) -> float:
    """The probability at which reviewing and accepting cost the same."""
    return cost_false_accept / (cost_false_accept + cost_false_reject)

None of these is available from a raw score, and all three are commonly attempted with one — which is the practical case for spending an afternoon on calibration.

Common Pitfalls Specific to This Technique

Fitting on the data the scorer was tuned on. The calibrator then learns the tuning rather than the miscalibration, and the reported improvement disappears in production.

Fitting per depot on small samples. Isotonic regression will fit a few hundred points perfectly and generalise badly. Fit globally unless each stratum genuinely has thousands of labels.

Shipping the calibrator without versioning it against the scorer. A calibrator fitted on the old score applied to a new one is worse than no calibration, and nothing about the output looks wrong.

Treating a poor post-fit calibration error as a calibration problem. It is a scorer problem: no monotonic mapping can fix a score that is not monotonic in the label.

Related

• Confidence scoring for stop detection — parent topic, including why binary thresholds fail
• Anomaly scoring for unexpected fleet stops — a second score with the same calibration need
• Interpreting OSRM match confidence scores — the same argument applied to a matcher’s own score
• Building a ground-truth set for map-matching evaluation — assembling the labels this needs
• Stop Detection & Dwell Time Analytics — parent section