How it works

Agent Blame

A trace shows what each step of a failed LLM pipeline did. Blame needs a counterfactual: had this step been better, with everything else the same, would the answer have been right? Below are the six ideas that make that question answerable, and a seventh on how the browser demo works, each with the few lines of code that carry it.

Causal inference
Shapley values
Concentration bounds
LLM tooling

Blaming one failure

Runthe pipeline once, every random draw addressed by a hash
Hold the noisereplay the recorded draws, or infer them from the choices
Swapre-run with each subset of steps using its fix
SplitShapley values share the repair among the steps
Aggregateaverage over failures, stop once the top culprit is clear

Randomness with an address

Swapping one step never shifts another step's random numbers

LLM steps sample, so re-running a failed pipeline with one step changed also changes the luck of every other step. Then you can't tell whether the fix helped or the dice did. Agent Blame never pulls numbers from a running random stream. Each draw is a hash of its address: the world, the step, a key and the option.

Formally the pipeline is a structural causal model. Step i computes y_i = f_i(parents_i, u_i), where u_i is its own sampling noise, and a failure is one realisation of all the u. The counterfactual is answered in Pearl's three steps: recover the noise of the failed run (abduction), switch the steps in S to their fixes (action), and re-run with the same noise (prediction). If each counterfactual re-sampled the LLM calls instead, the gain from a fix would mix its effect with the luck of a fresh sample, and the question would quietly change from “what caused this failure” to “what helps on average”.

agentblame/noise.pylines 45–56

def _u01(*parts: Any) -> float:
    """Deterministic uniform in the open interval (0, 1) from a hashed address."""
    h = hashlib.blake2b(digest_size=8)
    for p in parts:
        h.update(repr(p).encode())
        h.update(b"\x1f")
    x = int.from_bytes(h.digest(), "little")
    return (x + 0.5) / 2.0**64


def _gumbel_from_u(u: float) -> float:
    return -math.log(-math.log(u))

The address parts are hashed with BLAKE2b. A separator byte between parts keeps ("ab", "c") and ("a", "bc") apart.
The 64-bit digest becomes a number strictly inside (0, 1). The half-step offset means it is never exactly 0 or 1, so the logs below can't blow up.
The inverse CDF turns a uniform into a standard Gumbel variable, the noise used for categorical choices in idea 02.

Every draw a step makes goes through one function. It looks up an explicit value first, and only falls back to the hash if there is none. That one check is how a logged production run, or noise inferred from what happened, gets plugged back in.

agentblame/noise.pylines 195–200

    def _raw(self, addr: str, element: Hashable, kind: str) -> float:
        ov = self.source.overrides.get((self.step, addr))
        if ov is not None and element in ov:
            return float(ov[element])
        u = _u01(self.source.seed, self.step, addr, element)
        return _gumbel_from_u(u) if kind == "gumbel" else u

Overrides win: a world built from a trace (NoiseSource.from_trace) or by abduction replays those exact values.
Otherwise the value depends only on (seed, step, address, element). Running a different variant of some other step can't touch it.

Common random numbers

Because every coalition of fixes sees the same noise, differences between runs are differences in the fixes, not in luck. A test checks that swapping in a stronger answer step leaves the noise identical at every shared address and the upstream output unchanged.

Same luck, new odds

Gumbel-max replay, and recovering the noise from what was chosen

A sample from probabilities softmax(l) can be drawn as the index of the largest l_k + g_k, where each g_k is Gumbel noise. Treat g as the step's luck. A better prompt or model changes the logits l, the luck stays the same, and the new argmax is the counterfactual choice. If the logits don't change, the original choice comes back exactly.

choice = argmax_k (l_k + g_k) → counterfactual = argmax_k (l′_k + g_k)

agentblame/noise.pylines 240–245

        addr = self._address(key)
        g = [self._raw(addr, lab, "gumbel") for lab in labels]
        idx = gumbel_max(lv, g)
        chosen = labels[idx] if as_label else idx
        self.draws.append(Draw(self.step, addr, "categorical", labels, g, lv, chosen))
        return chosen

The noise is keyed by option label, not position. A fixed step that scores the same options gets the same Gumbel per option.
The sample itself is just an argmax over logit plus noise.
The draw is recorded with its logits and the chosen option. That record is what replay and abduction work from.

Often only the choice was logged, not the noise. Abduction, the first of Pearl's three counterfactual steps, then samples noise that is consistent with the choice. The construction comes from A* sampling: the winning perturbed value is Gumbel(logsumexp l), and every other option gets a Gumbel truncated below it (Maddison, Tarlow & Minka, 2014). Gumbel-max replay for counterfactuals is from Oberst & Sontag (ICML 2019). This is a modelling choice: observational data alone cannot identify counterfactuals.

agentblame/noise.pylines 100–115

    fin = [x for x in logits if math.isfinite(x)]
    mx = max(fin)
    lse = mx + math.log(sum(math.exp(x - mx) for x in fin))
    g = rng.gumbel(size=len(logits) + 1)          # one batch call: cheaper than many
    top = lse + g[-1]
    noise = np.empty(len(logits))
    for j, lj in enumerate(logits):
        if j == observed:
            noise[j] = top - lj
        elif math.isfinite(lj):
            z = lj + g[j]                            # z ~ Gumbel(lj), truncate below top
            m = min(top, z)
            noise[j] = m - math.log1p(math.exp(-abs(top - z))) - lj
        else:
            noise[j] = g[j]
    return noise

Log-sum-exp of the finite logits, shifted by the max for numerical safety.
The top perturbed value. Its distribution doesn't depend on which option won, so it can be drawn first.
The observed option is given exactly that top value, so it is guaranteed to win again.
Every other option is a Gumbel around its own logit, truncated to stay below the top, computed stably with log1p. Only the noise (value minus logit) is returned.

Validated

One test draws 60,000 prior noise vectors, keeps those that reproduce each observed choice, and compares the counterfactual choices with those from abduction. They agree to within 0.02 per option. Another test checks that an abducted world reproduces the factual run on 40 cases.

Share the repair fairly

Exact Shapley values over every subset of fixes

A pipeline is a list of named steps. Each step has the implementation that actually ran and one or more alternatives: a better prompt, a stronger model, a reranker, an oracle. For a set S of steps, v(S) is the score when the steps in S run their fix and the rest run as they did, all under the same noise. v(∅) is the failure, v(all) the fully fixed run. A step's Shapley value is its average marginal effect over every order in which the fixes could be applied, and the values add up to v(all) − v(∅). It is the unique split that adds up to the total, treats interchangeable steps equally and gives zero to a fix that never matters. That handles the two cases that break simpler rules: over-determination (two fixes that each suffice) and conjunction (two fixes that only work together).

agentblame/attribution.pylines 206–215

    w = [math.factorial(s) * math.factorial(n - s - 1) / math.factorial(n) for s in range(n)]
    vals = [game.value(m) for m in range(1 << n)]
    phi = {}
    for i, p in enumerate(game.players):
        bit = 1 << i
        tot = 0.0
        for m in range(1 << n):
            if not m & bit:
                tot += w[bin(m).count("1")] * (vals[m | bit] - vals[m])
        phi[p] = tot

The weight of a subset of size s: the fraction of orderings in which exactly those s fixes come before step i.
Subsets are bitmasks. For each subset without step i, add the weighted gain from also fixing i. The bit count gives the subset size.

That is 2ⁿ pipeline runs per case. Most of them share a prefix: fixing the reviewer doesn't change anything upstream of it. So every step execution is cached on what determines its output.

agentblame/pipeline.pylines 197–210

            key = (s.name, v, noise.world_id, _fingerprint(args))
            hit = self._cache.get(key) if self.use_cache else None
            if hit is not None:
                out, step_draws = hit
                self.meter.cache_hits += 1
            else:
                handle = noise.handle(s.name)
                out = s.call(v, args, handle)
                step_draws = handle.draws
                cost = s.variants[v][1]
                self.meter.charge(s.name, cost)
                spent += cost
                if self.use_cache:
                    self._cache[key] = (out, step_draws)

The cache key: step, variant, noise world and the step's actual inputs. Addressed noise (idea 01) is what makes the world id a valid part of a key.
A hit reuses the stored output and its recorded draws, and is free.
A miss runs the step, charges its cost to the meter, and stores the result. The cost meter is what batch budgets read later.

Why not simpler rules

In the refund-policy example the planner fix and the retriever fix each repair the answer alone, and the writer and reviewer fixes only work together. Fixing a step alone credits planner and retriever fully (200% in total), leave-one-out gives every step zero, and “last step to touch” blames the reviewer. Shapley gives 5/12, 5/12, 1/12, 1/12, matching a hand calculation in the tests. Caching cuts the 16 coalitions from 64 step runs to 14.

refund-policy step	Shapley	fix alone
Planner	+0.417 (5/12)	+1.000
Retriever	+0.417 (5/12)	+1.000
Writer	+0.083 (1/12)	0.000
Reviewer	+0.083 (1/12)	0.000

The Planner fix adds an “effective 2026” constraint to the search; the Retriever fix drops superseded documents; the Writer fix cites sources; the Reviewer fix re-checks cited claims against the live policy registry, so it only helps together with the Writer fix. If the Reviewer instead checks every claim (mechanism B), the Writer becomes a null player and the split is 1/3, 1/3, 0, 1/3. Both are in the HTML report, and both can be tried on the project page.

Sample orders, with a guarantee

Monte Carlo Shapley, empirical-Bernstein intervals and adaptive stopping

For many steps, 2ⁿ subsets is too many. Instead, sample random orders of the steps and fix them one at a time. Each order gives one marginal contribution per step, and their average is an unbiased estimate of the Shapley value (Castro, Gómez & Tejada, 2009). The question is when to stop.

agentblame/attribution.pylines 281–299

    while t < max_permutations:
        perm = rng.permutation(n)
        row = np.empty(n)
        mask, prev = 0, evaluate(0, t, 0)
        for pos, i in enumerate(perm, start=1):
            mask |= 1 << int(i)
            cur = evaluate(mask, t, pos)
            row[i] = cur - prev
            prev = cur
        marg.append(row)
        t += 1
        spent = game.pipeline.meter.total - c0
        over_budget = budget is not None and spent >= budget
        if t >= next_check or over_budget or t == max_permutations:
            k += 1
            dk = delta / (n * k * (k + 1)) if adaptive else delta / n
            X = np.array(marg)
            means = X.mean(axis=0)
            rad = np.array([_radius(bound, X[:, j], R, dk) for j in range(n)])

A random order of the steps.
Add steps one at a time and record each one's gain. These gains telescope to v(all) − v(∅), so the estimates always add up exactly, like true Shapley values.
Confidence is checked on a geometric schedule. Checkpoint k gets error budget δ / (n·k·(k+1)), which sums to at most δ over all steps and checkpoints, so every interval holds at once even though the code decides when to stop by looking at them.
One radius per step from the chosen bound, over all marginals so far.

Hoeffding's bound is available too; the default is empirical Bernstein (Maurer & Pontil 2009). Most marginal contributions are exactly zero, so the sample variance V is small and this radius is much tighter than Hoeffding's, which only uses the range R.

radius = √(2·V·ln(4/δ) / t) + 7·R·ln(4/δ) / (3(t − 1))

Result

On the 8-step demo the adaptive “top1” rule stopped after 692 permutations, once the drafter's interval [+0.344, +0.682] cleared every other step's. Its estimate was within 0.024 of the exact values. With 8 steps and caching, exact Shapley costs only 126 runs, so sampling saves little there. It pays off when 2ⁿ is large.

From one failure to many

Blame across a test set, and stopping once the culprit is clear

One failure is an anecdote. blame_batch runs every test case, analyses only the failures, and reports per step how often fixing it alone resolves a failure, its mean Shapley value with a confidence interval, and its share of all the repair. Analysing a failure costs many runs, so it can stop early.

agentblame/batch.pylines 96–119

    for case in cases:
        n_seen += 1
        factual = pipeline.run(case.inputs, noise=NoiseSource(case.seed))
        if not is_failure(factual.score):
            continue
        n_fail += 1
        cb = analyze_case(pipeline, case, fixes=fixes, players=players, mode=mode,
                          n_worlds=n_worlds, crn=crn, method=method, mc=mc,
                          seed=seed, factual=factual)
        analyzed.append(cb)
        if budget is not None and pipeline.meter.total - c0 >= budget:
            reason = "budget"
            break
        if stop_when_separated and len(analyzed) >= next_check:
            k += 1
            ps = analyzed[0].players
            dk = delta / (len(ps) * k * (k + 1))
            X = np.array([[c.shapley.values[p] for p in ps] for c in analyzed])
            means = X.mean(axis=0)
            rad = np.array([_radius(stop_bound, X[:, j], value_range, dk) for j in range(len(ps))])
            if _separated(means, rad, "top1"):
                reason = f"top step separated ({stop_bound}) after {len(analyzed)} failures"
                break
            next_check = max(len(analyzed) + 1, int(math.ceil(min_failures * growth ** k)))

Every case is run once as it happened. Passing cases cost one pipeline run and are skipped.
A failure gets the full analysis. Its factual trace is passed along so it isn't run again.
The same anytime union bound as idea 04, now across cases: stop when the top step's interval on the mean Shapley value is disjoint from all others.

The honest trade-off

In the 8-step stream the finite-sample Bernstein rule needed 1,038 failures to separate the drafter; the asymptotic normal rule stopped after 61, with no finite-sample guarantee. On the 200-case RAG demo, “last step to touch” blamed the verifier for 149 of 154 failures, while Shapley put the retriever and query rewriter on top.

Real LLMs: move the dice to the client

The optional Anthropic adapter

A hosted model samples on the server, and the Messages API doesn't return token log-probabilities, so its noise can't be held fixed. The adapter's answer for decisions such as routing, tool choice or a verdict: ask the model to score each option at temperature 0, then sample the choice locally through the noise handle.

agentblame/adapters/anthropic_adapter.pylines 207–218

    def make(vname: str, cfg: Mapping[str, Any]):
        p = cfg.get("prompt", prompt)
        suffix = _SCORE_INSTRUCTIONS.format(options=", ".join(options))   # appended after .format
        mdl = cfg.get("model")
        k = cfg.get("sharpness", sharpness)

        def fn(noise, **inputs):
            text = llm.complete(p.format(**inputs) + suffix, step=name, variant=vname,
                                inputs=inputs, temperature=0.0, model=mdl)
            scores = _parse_scores(text, options)
            return noise.categorical({o: k * s for o, s in scores.items()})
        return fn

A temperature-0 call, cached, so the scores are a fixed function of the prompt and inputs.
The JSON scores are parsed; a missing option scores 0.
The scores become logits for Gumbel-max (idea 02). A better prompt changes the scores, the per-option noise stays, and replay is exact.

Free-text steps are cached too. The key includes a seed drawn from the step's noise, so replaying the recorded world returns the factual response without a network call.

agentblame/adapters/anthropic_adapter.pylines 129–135

        model = model or self.model
        key = ResponseCache.key(step=step, variant=variant, inputs=inputs, model=model,
                                prompt=prompt, system=system, temperature=temperature,
                                seed=seed if temperature > 0 else None)
        hit = self.cache.get(key)
        if hit is not None:
            return hit

At temperature 0 the seed is dropped, so all worlds share one response. Above 0 the seed picks a sample slot.
Cache first. With offline=True a miss raises instead of calling the API, which is how the tests run with a fake client and no network.

Limitation

When a sampled free-text step's prompt or inputs change, the cache key changes and the result is a fresh sample. Blame for that step is then interventional (“how often does this help”), not a counterfactual about this failure.

The demo in the browser

Replays exported once, attribution computed live

The project page lets you switch steps to their fixes and watch the blame move. It does not run the pipelines in the browser. A script runs the real package once, for every failing case of four scenarios (the refund policy with both reviewer fixes, all 154 RAG failures under three noise settings, 12 cases of the 8-step agent, the 30 quickstart failures), and for every combination of fixes stores the score and each step's output. That is 506 games and about 340 KB of JSON.

examples/export_web_demo.pylines 45–60

def export_game(game, show, S, py):
    """All 2^n coalitions: v(S), shown output and fingerprint of every step."""
    n = game.n
    steps = game.pipeline.steps
    v, out, fp = [], [], []
    for m in range(1 << n):
        v.append(game.value(m))
        tr = game.trace(m)
        out.append([S(show(s.name, tr.outputs[s.output])) for s in steps])
        fp.append([_fingerprint(tr.outputs[s.output]) for s in steps])
    # Last-touch compares fingerprints (exact output equality). Ship them only
    # for steps where the shown text is not a faithful stand-in for them.
    need = [any((out[a][j] == out[b][j]) != (fp[a][j] == fp[b][j])
                for a in range(1 << n) for b in range(a))
            for j in range(len(steps))]
    fps = {steps[j].name: [S(fp[m][j]) for m in range(1 << n)] for j in range(len(steps)) if need[j]}

Each coalition goes through the package's own BlameGame: value is the score (averaged over 8 worlds for the inferred and fresh noise settings), trace the run in the reference world whose outputs fill the step cards.
Last-step blame compares exact output fingerprints, not display text. The fingerprints are shipped only for a step where the short display text would give a different answer. In the four scenarios that never happens, so none are shipped.

The browser does the attribution itself. The JavaScript follows the Python line by line, with the same weights, the same loop order and the same floating-point operations, so it gives bit-identical numbers rather than numbers that agree to a few decimals.

demo/blame.jslines 16–29

function exactShapley(v, n) {
  const w = [];
  for (let s = 0; s < n; s++) w.push(factorial(s) * factorial(n - s - 1) / factorial(n));
  const phi = [];
  for (let i = 0; i < n; i++) {
    const bit = 1 << i;
    let tot = 0;
    for (let m = 0; m < (1 << n); m++) {
      if (!(m & bit)) tot += w[popcount(m)] * (v[m | bit] - v[m]);
    }
    phi.push(tot);
  }
  return phi;
}

The weight of a subset of size s, computed as in exact_shapley: whole-number factorials, then one division.
Subsets as bitmasks in increasing order, the same accumulation order as the Python loop. Fix-alone and leave-one-out are one line each next to it.

demo/blame.jslines 48–57

function lastTouch(game, steps) {
  const full = game.out.length - 1;
  let last = null;
  steps.forEach(function (s, j) {
    const f = game.fp && game.fp[s];
    const a = f ? f[0] : game.out[0][j], b = f ? f[full] : game.out[full][j];
    if (a !== b) last = s;
  });
  return last;
}

Like last_touch: walk the steps in run order and keep the last one whose output differs between the failed run (mask 0) and the run with every fix (the full mask).

Because the scores are plain data, the page can also let you edit them: flip one combination's pass/fail, or set its success rate, and every rule is recomputed. That shows what each rule responds to without re-running anything. The RAG tab also recomputes the batch table over all 154 failures (mean Shapley, share of repair, last-step counts) with your edits included.

Checked against Python

On all 506 exported games the browser's Shapley, fix-alone and leave-one-out values are bit-identical to the package's, and last-step blame agrees on every one. The RAG batch means differ from the package by at most 1.7 × 10⁻¹⁶, because numpy sums in a different order. Monte Carlo Shapley is not in the browser: with at most 8 steps, exact enumeration of 256 coalitions is instant.

→

Results

From the three demos

Stochastic RAG, 200 cases

rewrite → retrieve → answer → verify, where every step is a mocked LLM that samples with Gumbel-max and the fixes only shift its logits. 154 of 200 cases fail.

step	mean Shapley [95% CI]	fixing alone resolves	share	last-touch blames
retrieve	+0.319 [0.269, 0.369]	26.0%	36.7%	2
rewrite	+0.312 [0.260, 0.364]	27.3%	35.8%	0
answer	+0.181 [0.133, 0.229]	16.2%	20.8%	0
verify	+0.058 [0.038, 0.079]	5.2%	6.7%	149

Recovering the noise from the observed choices alone (8 posterior worlds) gives almost the same ranking and sizes. The interventional version with fresh noise gives smaller numbers because it asks a different question. The naive “last step to touch” rule blames verify for 149 of 154 failures.

Sharing noise across a permutation cuts the variance of the Monte Carlo estimator 2.0–2.8× at about half the step cost, a 3.7–5.5× efficiency gain at equal budget.

8-step agent, Monte Carlo with adaptive stopping

intake → router → planner → search → extractor → calculator → drafter → guardrail on a finance question. Exact Shapley over 256 coalitions costs 126 step runs with caching (about 6,500 without). Monte Carlo error shrinks like 1/√permutations and the simultaneous intervals always cover the exact values. The adaptive rule stops after 692 permutations with drafter at +0.513 [0.344, 0.682], separated from every other step. The router's “fix” can mis-route under some noise, so its Shapley value can be negative: a fix that makes things worse.

Sample reports

Refund policyBoth reviewer mechanisms, exact Shapley next to the naive baselines.11 KB Stochastic RAG200 cases: batch blame with CIs, then a drill-down into every failure.1.4 MB

These were generated by running the demos, and the numbers match the README. The package writes such reports itself: terminal tables and a self-contained HTML file with bars, CI whiskers and a per-case drill-down.

→

Limits

Honest caveats

Blame is always relative to the fixes you supply. A step with no good alternative can't be blamed.
Hosted LLM APIs don't expose their sampling noise or (for the Anthropic Messages API) token log-probabilities. For a sampled free-text call whose input changes, the counterfactual is a fresh sample, which is interventional rather than counterfactual. Routing, tool choice and verdicts can be made exactly replayable by scoring the options and sampling locally (idea 06).
With 8 steps and prefix caching, exact Shapley is cheap, so Monte Carlo saves little in that demo. It pays off when 2ⁿ is large or steps can't share prefixes.
The finite-sample batch stopping rule is valid but conservative: it needed about 1,000 failures to separate a 0.18 gap, where the asymptotic rule stopped after 61.
The browser demo only covers the cases that were exported. It looks up stored replays; to blame your own pipeline you need the Python package.

→

Reproduce

Install, test, run the demos

The only core dependency is numpy. It runs on CPU and needs no API keys; the Anthropic adapter is optional. Inside agent-blame/:

pip install -e .
python -m pytest -q                         # 35 tests, about 3 s
python examples/quickstart.py               # short API tour
python -m agentblame.demos.refund_policy    # under a second
python -m agentblame.demos.stochastic_rag   # under a minute
python -m agentblame.demos.eight_step       # about 15 s
python examples/export_web_demo.py          # rebuilds demo/data.json for the project page

The demos write their HTML reports to agentblame_out/.

→

Read further

Where everything lives

agentblame/noise.py addressed noise, Gumbel-max, posterior Gumbels, abduction
agentblame/pipeline.py steps and variants, the pipeline runner, traces, cost meter, step cache
agentblame/attribution.py the blame game, exact and Monte Carlo Shapley, bounds, baselines
agentblame/batch.py blame over a batch of failures, CIs, adaptive stop
agentblame/report.py terminal tables and the self-contained HTML report
agentblame/adapters/anthropic_adapter.py cached LLM steps, client-side choice sampling
tests/test_shapley.py efficiency, symmetry, the 5/12 hand check, Monte Carlo coverage
tests/test_noise.py Gumbel replay, abduction against rejection sampling
agentblame/demos/refund_policy.py example (a), the refund-policy pipeline
agentblame/demos/stochastic_rag.py example (b), 200 stochastic RAG cases, noise modes, CRN variance
agentblame/demos/eight_step.py example (c), 8-step agent, Monte Carlo and adaptive stopping
examples/quickstart.py minimal API tour
examples/export_web_demo.py exports the replays behind the project page
demo/blame.js the browser demo: attribution and UI
demo/data.json the exported replays (340 KB)
pyproject.toml
README.md full write-up, theory and roadmap, and every number

Back to Agent Blame All projects