# agentblame: counterfactual blame for LLM / agent pipelines When a multi-step LLM pipeline (planner → retriever → writer → reviewer, or an agent loop) gets an answer wrong, a trace shows you what each step *did*. `agentblame` tells you which step *caused* the failure. It does this by replaying the pipeline with some steps swapped for better versions, holding the randomness fixed, and splitting the credit for the repair with Shapley values. * **Define a pipeline** as named steps. Each step has its as-run implementation and one or more alternatives (better prompt, stronger model, reranker, oracle…). * **Counterfactual replay with shared noise.** Every random choice a step makes goes through a noise handle. Swapping one step never changes the noise the other steps see (common random numbers). A Gumbel-max helper makes sampled LLM choices replayable under new logits. * **Attribution.** Exact Shapley (2ⁿ coalitions, memoized, with step-level caching), Monte Carlo permutation Shapley with simultaneous Hoeffding / empirical-Bernstein CIs and adaptive stopping, and three naive baselines (fix-alone, leave-one-out, last-step-to-touch) for comparison. * **Batch view.** Over many failing test cases you get "fixing step X alone resolves Y % of failures" and mean Shapley blame with CIs. It is budget-aware: each step run has a cost, cached runs are free, and analysis can stop as soon as the top culprit is statistically separated. * **Reports.** Terminal tables and a self-contained HTML report (no external assets) with bars, CI whiskers and a per-case drill-down. Core dependency: `numpy`. CPU only. No API keys needed. The Anthropic adapter is optional and lazily imported. --- ## Quick start ```bash cd agent-blame pip install --break-system-packages -e . # or: pip install -e . inside a venv pip install --break-system-packages pytest python -m pytest -q # 35 tests, ~3 s python examples/quickstart.py # minimal API tour, <1 s python -m agentblame.demos.refund_policy # example (a), <1 s python -m agentblame.demos.stochastic_rag # example (b), ~30 s CPU (≈1 min wall on a busy 2-CPU box) python -m agentblame.demos.eight_step # example (c), ~13 s ``` The demos write HTML reports to `./agentblame_out/` (change this with `--out DIR`). `stochastic_rag` uses at most 2 worker processes. ### Minimal API ```python from agentblame import step, Pipeline, Case, blame_batch, analyze_case, format_batch @step("retriever", cost=2.0) # parameter names = upstream outputs / case inputs def retriever(planner, noise): # `noise` is optional; gives addressed random draws return noise.categorical({"gold": 1.0, "stale": 1.3}) # Gumbel-max sample @retriever.variant("reranker", cost=4.0) # first variant = default "fix" for blame def retriever_fixed(planner, noise): return noise.categorical({"gold": 3.0, "stale": 0.2}) # same Gumbels, new logits pipe = Pipeline([planner, retriever, writer], metric=lambda out: float(out["writer"] == "ok")) batch = blame_batch(pipe, [Case("q1", {"question": "..."}, seed=1), ...]) print(format_batch(batch)) ``` `analyze_case(..., mode=...)` picks how the noise is held fixed: | mode | what is known | question answered | |---|---|---| | `"recorded"` (default) | the factual noise (simulated, seeded, or logged/cached) | "Would *this* failure have been avoided had S been fixed?" (exact, deterministic) | | `"posterior"` | only the factual *choices* | same counterfactual question; noise is abducted from the observed choices (`n_worlds` posterior samples, shared across coalitions) | | `"prior"` | nothing | "How often would fixing S help on a fresh run of this input?" (interventional, not counterfactual) | --- ## The theory in plain language **Pipeline as a structural causal model.** Step *i* computes `y_i = f_i^{variant}(parents_i, u_i)`, where `u_i` is its exogenous randomness (sampling noise). A failure is one realization of all the `u`. The counterfactual "had steps S used their fixes, would the run have succeeded?" is answered in Pearl's three steps. *Abduction* recovers `u` for the failed run. *Action* applies do(variant = fix) to the steps in S. *Prediction* re-runs the pipeline with the same `u`. This gives a game `v(S)` ∈ [0, 1]. `v(∅)` is the factual outcome (0 for a failure) and `v(all)` is the outcome with every fix applied. **Why the noise must be shared.** If each counterfactual re-samples the LLM calls independently, `v(S ∪ {i}) − v(S)` mixes the effect of fixing *i* with the luck of a fresh sample. Some failures then get "fixed" by changing a step that did not matter, and the question quietly changes from "what caused *this* failure" to "what tends to help on average". Holding the noise fixed removes that confound. Addressed noise does this: a draw is a hash of (world, step, key, option), so swapping one step does not shift any other step's random numbers. It also lowers estimator variance (common random numbers): differences between correlated runs are less noisy than differences between independent runs. **Gumbel-max replay.** A categorical sample from logits `l` can be written as `argmax_k (l_k + g_k)` with i.i.d. standard Gumbel `g`. Treat `g` as the step's exogenous noise. The counterfactual under new logits `l'` (a better prompt or model) is then `argmax_k (l'_k + g_k)` with the *same* `g`, so unchanged logits always reproduce the factual sample. This "Gumbel-max SCM" was proposed for counterfactual off-policy evaluation by Oberst & Sontag (ICML 2019). It is a *modelling choice*: observational data cannot identify counterfactual distributions, and the Gumbel-max model is one reasonable SCM that satisfies their "counterfactual stability" property. When only the chosen option was logged (not `g`), `abduct()` samples `g` from its posterior given the observed argmax. It uses the top-down construction from A* sampling (Maddison, Tarlow & Minka 2014): the max is Gumbel(logsumexp `l`), and the others are Gumbels truncated below it. A test checks this against rejection sampling. **Shapley values.** `φ_i = Σ_{S ∌ i} |S|!(n−|S|−1)!/n! · (v(S∪{i}) − v(S))` is step *i*'s average marginal effect over all orders in which the fixes could be applied. It is the unique attribution satisfying *efficiency* (Σφ = v(all) − v(∅)), *symmetry*, *null player* (a fix that never matters gets 0) and *additivity*. This handles the two cases that break simpler rules: * *Over-determination*: Planner and Retriever are each enough on their own. "Fix-alone" gives both full credit (total > 100 %). "Leave-one-out" (but-for from the fully fixed state) gives both zero. * *Conjunction*: the Writer fix (cite sources) and Reviewer fix (check cited claims) only help together. Fix-alone gives both zero. Shapley gives the redundant causes 5/12 each and the conjunctive pair 1/12 each. The naive "last step to touch the wrong value" heuristic blames the Reviewer, because it emitted the final wrong answer. **Monte Carlo Shapley with guarantees.** For larger *n*, `mc_shapley` samples random permutations. Each one gives one marginal contribution per step, an unbiased estimate (Castro, Gómez & Tejada 2009). The per-permutation marginals telescope to `v(all) − v(∅)`, so the estimate is exactly efficient. Marginals lie in [−1, 1] for a 0/1 metric. The library reports either a Hoeffding radius or an empirical-Bernstein radius `sqrt(2·V·ln(4/δ)/t) + 7·R·ln(4/δ)/(3(t−1))` (Maurer & Pontil 2009). The Bernstein radius is much tighter when most marginals are exactly 0, which is typical. Checks run on a geometric schedule with `δ_k = δ / (n·k·(k+1))`. A union bound over steps and checkpoints makes all intervals hold simultaneously with probability ≥ 1 − δ, *including under adaptive stopping*. The stopping rule is best-arm-identification style: stop when the top step's interval is disjoint from all others (`stop="top1"`), when all adjacent pairs separate (`"ranking"`), when every radius is ≤ `width`, or when the cost `budget` runs out. Tied players (e.g. two interchangeable causes) never separate, so `"ranking"` then runs to the budget. **Batches.** `blame_batch` runs each case, replays its failures, and reports three numbers per step: the mean fix-alone gain ("fixing X alone resolves Y % of failures"), the mean Shapley value with a CI across cases (normal or empirical-Bernstein), and the Shapley share of the total repairable failure mass. With `stop_when_separated=True` it stops analysing failures once the top step separates, using the same anytime union bound. With `stop_bound="normal"` it uses an asymptotic heuristic that stops much earlier but is not anytime-valid. --- ## Examples and real output All numbers below come from the runs used to write this README (Python 3.13, numpy 2.5, 2-CPU machine shared with other jobs). ### (a) Refund-policy pipeline: `python -m agentblame.demos.refund_policy` Planner → Retriever → Writer → Reviewer answers "90 days" from a superseded 2023 policy. The correct answer is 30 days. The fixes: * Planner: add a 2026 date constraint. * Retriever: drop superseded docs. * Writer: cite sources. * Reviewer: re-check *cited* claims against the policy registry. The value function is checked exhaustively in the tests: `v(S) = 1` iff S ∋ Planner, or S ∋ Retriever, or S ⊇ {Writer, Reviewer}. ``` Mechanism A -- Reviewer only checks *cited* claims case refund-q1: factual score=0 v(no fix)=0.000 v(all fixes)=1.000 [exact, 16 coalitions, cost 14] step fix-alone leave-1-out Shapley planner +1.000 +0.000 +0.417 (= 5/12) retriever +1.000 +0.000 +0.417 (= 5/12) writer +0.000 +0.000 +0.083 (= 1/12) reviewer +0.000 +0.000 +0.083 (= 1/12) last-step-to-touch blames: reviewer Mechanism B -- Reviewer checks *all* claims planner +0.333 retriever +0.333 writer +0.000 reviewer +0.333 ``` Hand check for Planner: its marginal is `1 − v(S)` over S ⊆ {R, W, Rv}, and `v(S) = 0` only for ∅, {W}, {Rv}. That gives 1/4 + 1/12 + 1/12 = 5/12. For the Writer the marginal is 1 only at S = {Rv}, weight 1/12. Changing the Reviewer's *mechanism* (B) makes the Writer a null player and puts the Reviewer among the sufficient fixes. Caching keeps the cost to 14 step runs instead of 16 × 4 = 64. ### (b) Stochastic RAG, 200 cases: `python -m agentblame.demos.stochastic_rag` rewrite → retrieve → answer → verify. Every step is a mocked LLM that samples from logits with Gumbel-max. Fixes shift the logits (stronger model, reranker, grounding check), so they are not deterministic either. 154 of 200 cases fail. ``` 1) Counterfactual blame, recorded noise (exact replay of each failure) 200 cases run, 154 failures, 154 analyzed, 20 not fixable by any combination; cost 6017 step fixing alone resolves mean Shapley [95% CI] share last-touch retrieve 26.0 % +0.319 [+0.269, +0.369] 36.7% 2 rewrite 27.3 % +0.312 [+0.260, +0.364] 35.8% 0 answer 16.2 % +0.181 [+0.133, +0.229] 20.8% 0 verify 5.2 % +0.058 [+0.038, +0.079] 6.7% 149 2a) same, noise abducted from observed choices only (8 posterior worlds) retrieve +0.316 rewrite +0.277 answer +0.186 verify +0.077 2b) interventional, fresh noise (8 prior worlds) -- a different question retrieve +0.244 rewrite +0.202 answer +0.130 verify +0.056 ``` Abduction from the observed choices alone recovers almost the same ranking and magnitudes as exact replay. The interventional version gives smaller numbers because it asks a different question: it ignores what actually went wrong in each failure. Last-step-to-touch blames `verify` for 149/154 failures. **Common random numbers vs. independent re-sampling** (100 failing cases, 4 worlds per coalition, 8 replicates; Var = variance across replicates, averaged over cases and steps; cost = step cost units per replicate): ``` mode estimator Var CRN Var indep indep/CRN cost CRN cost indep Var*cost gain posterior fix-alone 0.02567 0.02134 0.83x (by-product of exact-avg) posterior exact-avg 0.01303 0.00920 0.71x 15314 67800 3.13x posterior mc-perm 0.00947 0.01903 2.01x 36720 67962 3.72x prior fix-alone 0.02905 0.10150 3.49x (by-product of exact-avg) prior exact-avg 0.01797 0.01555 0.87x 13422 67200 4.33x prior mc-perm 0.01021 0.02858 2.80x 34872 67962 5.46x ``` How to read this: * **Recorded-noise replay has zero variance.** It is deterministic, and it is the right estimand when you have the noise. * **Permutation MC (the scalable estimator) benefits clearly from CRN.** With one world shared across a permutation, the variance is 2.0–2.8× lower at about half the step cost. That is a 3.7–5.5× efficiency gain. * **"Fix-alone" in prior mode:** 3.5× lower variance, because `v({i})` and `v(∅)` are positively correlated under shared noise. In posterior mode `v(∅)` is identically 0 (abduction reproduces the failure), so there is nothing to correlate and CRN gives no gain (0.83×, within noise). * **Exact Shapley over world-averaged values (`exact-avg`):** at an equal *number of runs*, independent sampling has 13–29 % lower variance. Its 16 coalition values use 16× more independent worlds, and errors in same-signed Shapley terms average out instead of adding up. But shared worlds let the step cache serve most runs, so CRN costs 4.4–5× less, and at equal *cost* it is still 3.1–4.3× more efficient. CRN is not a variance-reduction guarantee for every linear combination. Its main job here is to make the estimand counterfactual. ### (c) 8-step agent, Monte Carlo + adaptive stopping: `python -m agentblame.demos.eight_step` intake → router → planner → search → extractor → calculator → drafter → guardrail, for a finance question whose answer needs routing AND evidence AND a correct number AND a delivered, unblocked draft. The router fix (an LLM classifier) can itself mis-route under some noise. In those cases its Shapley value is *negative*: a "fix" that makes things worse. ``` Case c000: exact Shapley over 2^8 = 256 coalitions (step cost with caching 126; without caching ~6528) intake +0.117 router +0.117 planner +0.117 search +0.117 extractor 0 calculator 0 drafter +0.533 guardrail 0 estimator perms coalitions cost max|err| max CI radius stop MC, fixed 16 perms (Bernstein) 16 77 125 0.217 2.412 max_permutations MC, fixed 64 perms (Bernstein) 64 182 126 0.123 0.694 max_permutations MC, fixed 256 perms (Bernstein) 256 255 126 0.029 0.231 max_permutations MC, fixed 1024 perms (Bernstein) 1024 256 126 0.015 0.086 max_permutations MC adaptive top1 (hoeffding) 692 256 126 0.024 0.177 separated (top1) MC adaptive top1 (bernstein) 692 256 126 0.024 0.169 separated (top1) MC adaptive top1, budget=100 4 27 100 0.217 12.075 budget Adaptive estimate: drafter +0.513 [+0.344, +0.682]; all other CIs below +0.282. Batch over 400 cases (exact): drafter +0.285 [+0.266, +0.304] (33.1 % share) ranks first; fix-alone resolves 0 % for every step (no single fix suffices); last-touch blames guardrail 400/400. Adaptive batch (stream of up to 3000 cases, delta=0.05): bernstein 1038 failures analysed, cost 155995, top = drafter normal 61 failures analysed, cost 9005, top = drafter ``` Error shrinks roughly like 1/√perms, and the simultaneous CIs always cover the exact values here. Note the honest caveat in the cost column. With *n* = 8 and prefix caching, exact Shapley costs only 126 units (vs ~6 500 uncached), so MC saves little in this demo. MC pays off when 2ⁿ is large or steps are expensive and cannot share prefixes. The finite-sample batch stopping rule is valid but conservative: the range term of the Bernstein bound needs about 1 000 cases to separate a 0.18 gap. The asymptotic rule stops after 61. ### Optional real-LLM adapter: `agentblame/adapters/anthropic_adapter.py` * `AnthropicLLM` wraps `anthropic.Anthropic().messages.create`. The SDK is imported on the first real call only. Responses are cached on (step, variant, inputs, model, prompt, temperature, seed) in a JSON file, and `offline=True` forbids any network call. * `llm_text_step` builds free-text steps. The seed is drawn from the step's noise handle, so replaying the recorded world reproduces the factual response exactly from the cache. * `llm_choice_step` is the recommended pattern for decisions such as routing, tool choice or verdicts. The model scores each option at temperature 0, and agentblame samples the choice locally with Gumbel-max, so counterfactual replay is exact. * **Limitation:** hosted LLM APIs do not expose their sampling noise, and the Anthropic Messages API does not return token log-probabilities. For a sampled free-text call whose prompt or inputs change, the counterfactual is therefore a fresh sample. That makes it interventional, not counterfactual. Exact Gumbel replay at the token level needs logprobs plus control of the sampler (e.g. a self-hosted model). Even temperature 0 is not guaranteed bit-reproducible on hosted inference. Caching pins the factual response. Tests use a fake client and never touch the network. --- ## Runtime | command | wall time here | |---|---| | `pytest -q` (35 tests) | ~3 s | | `examples/quickstart.py` | 0.3 s | | `demos.refund_policy` | < 1 s | | `demos.eight_step` | ~13 s | | `demos.stochastic_rag` | ~58 s wall / 29 s CPU (2 processes; the machine was shared) | `stochastic_rag` can be shrunk with `--replicates 4 --var-cases 50`. --- ## Roadmap to a product 1. **Zero-effort instrumentation.** Auto-wrap steps from existing tracing (OpenTelemetry spans, LangGraph / LlamaIndex / custom agent loops). Log the noise that matters: seeds, sampled choices, tool results, retrieved docs. Every production failure then becomes replayable (`NoiseSource.from_trace`, `abduct`). 2. **A catalogue of fixes.** Oracle injection (feed the gold output of a step: the cleanest "was this step the cause" test), stronger model, prompt variants, retries, a human-labelled correction. Blame is always *relative to the fixes considered*, and the UI must say so. 3. **Client-side sampling for decisions.** Push every discrete decision (route, tool, document, stop/continue) through `noise.categorical` over model scores or logprobs. Where logprobs exist (self-hosted / vLLM), add token-level Gumbel replay. 4. **Scale and cost.** Stratified or antithetic permutation sampling, KernelSHAP-style regression estimators, sharing coalitions across cases, and bandit allocation of budget across *cases* (spend replays where the ranking is uncertain). Also tighter anytime-valid bounds (betting confidence sequences) to cut the ~1 000-case requirement of the conservative rule. 5. **Interactions.** Shapley interaction indices to surface pairs like Writer+Reviewer that only work together, and agent loops with repeated steps, where blame should go to (step, iteration). 6. **Product surface.** A CI check that blames regressions to a step, weekly dashboards ("retriever caused 37 % of failures, fixing it alone resolves 26 %"), and per-failure drill-down linked to traces. The HTML report here is the seed of that UI. 7. **Validation.** Benchmarks of synthetic pipelines with known ground-truth causal structure (like the examples here), plus user studies comparing time-to-fix with and without counterfactual blame. --- ## File layout ``` agent-blame/ ├── pyproject.toml # pip install -e . (deps: numpy; extras: test, anthropic) ├── README.md ├── agentblame/ │ ├── __init__.py # public API │ ├── noise.py # addressed noise, Gumbel-max, posterior Gumbels, abduction │ ├── pipeline.py # Step / @step / variants, Pipeline runner, Trace, cost meter, step cache │ ├── attribution.py # BlameGame v(S), exact & MC Shapley, bounds, baselines, analyze_case │ ├── batch.py # blame_batch: aggregate over failures, CIs, adaptive stop │ ├── report.py # terminal tables, self-contained HTML │ ├── adapters/anthropic_adapter.py # optional cached LLM steps (lazy SDK import) │ └── demos/ │ ├── refund_policy.py # example (a) │ ├── stochastic_rag.py # example (b) │ └── eight_step.py # example (c) ├── examples/quickstart.py # minimal API tour ├── tests/ # 35 pytest tests (~3 s) └── agentblame_out/ # HTML reports written by the demos ``` ## References (for the techniques used) * J. Pearl, *Causality* (2009): abduction–action–prediction counterfactuals. * L. S. Shapley, "A value for n-person games" (1953). * J. Castro, D. Gómez, J. Tejada, "Polynomial calculation of the Shapley value based on sampling", *Computers & OR* (2009). * A. Maurer, M. Pontil, "Empirical Bernstein bounds and sample variance penalization", COLT (2009). * C. Maddison, D. Tarlow, T. Minka, "A* sampling", NeurIPS (2014). * M. Oberst, D. Sontag, "Counterfactual off-policy evaluation with Gumbel-max structural causal models", ICML (2019).