Refund-policy pipeline: who caused the stale answer?
Planner → Retriever → Writer → Reviewer. The as-run pipeline answers '90 days' from a superseded policy; the correct answer is 30 days.
Planner and Retriever are each sufficient on their own (over-determination), so leave-one-out blames neither; fix-alone blames only them; last-step-to-touch blames the Reviewer. Shapley gives the redundant causes 5/12 each and splits the remaining 1/6 between Writer and Reviewer, which only work together.
Changing how the Reviewer works (mechanism B) changes the blame: Writer becomes a null player and the Reviewer joins the set of sufficient fixes.
Mechanism A: Reviewer checks only cited claims
refund-q1 factual=0, v(∅)=0.00, v(all)=1.00, top: planner
| step | fix alone | leave-one-out | Shapley | Shapley bar (±CI) | last touch |
|---|---|---|---|---|---|
| planner | +1.000 | +0.000 | +0.417 | ||
| retriever | +1.000 | +0.000 | +0.417 | ||
| writer | +0.000 | +0.000 | +0.083 | ||
| reviewer | +0.000 | +0.000 | +0.083 | ◆ |
exact; 16 coalitions; cost 14; exhaustive
Coalition values v(S) (S = set of fixed steps):
| {} | 0.000 |
| {planner} | 1.000 |
| {retriever} | 1.000 |
| {planner, retriever} | 1.000 |
| {writer} | 0.000 |
| {planner, writer} | 1.000 |
| {retriever, writer} | 1.000 |
| {planner, retriever, writer} | 1.000 |
| {reviewer} | 0.000 |
| {planner, reviewer} | 1.000 |
| {retriever, reviewer} | 1.000 |
| {planner, retriever, reviewer} | 1.000 |
| {writer, reviewer} | 1.000 |
| {planner, writer, reviewer} | 1.000 |
| {retriever, writer, reviewer} | 1.000 |
| {planner, retriever, writer, reviewer} | 1.000 |
Outputs: factual vs. all-fixed
| value | factual | all fixes |
|---|---|---|
| question | 'How many days do I have to request a refund?' | 'How many days do I have to request a refund?' |
| planner | {'keywords': ['refund', 'policy'], 'year': None} | {'keywords': ['refund', 'policy'], 'year': 2026} |
| retriever | {'id': 'policy-2023', 'year': 2023, 'superseded': True, 'text': 'Refund policy: refund policy allows refunds within 90 days of purchase.'} | {'id': 'policy-2026', 'year': 2026, 'superseded': False, 'text': 'Refunds are accepted within 30 days of purchase (effective 2026).'} |
| writer | {'claims': [{'text': 'Refunds are accepted within 90 days.', 'cite': None}]} | {'claims': [{'text': 'Refunds are accepted within 30 days.', 'cite': 'policy-2026'}]} |
| reviewer | 'Refunds are accepted within 90 days.' | 'Refunds are accepted within 30 days.' |
Mechanism B: Reviewer checks all claims
refund-q1 factual=0, v(∅)=0.00, v(all)=1.00, top: planner
| step | fix alone | leave-one-out | Shapley | Shapley bar (±CI) | last touch |
|---|---|---|---|---|---|
| planner | +1.000 | +0.000 | +0.333 | ||
| retriever | +1.000 | +0.000 | +0.333 | ||
| writer | +0.000 | +0.000 | +0.000 | ||
| reviewer | +1.000 | +0.000 | +0.333 | ◆ |
exact; 16 coalitions; cost 4; exhaustive
Coalition values v(S) (S = set of fixed steps):
| {} | 0.000 |
| {planner} | 1.000 |
| {retriever} | 1.000 |
| {planner, retriever} | 1.000 |
| {writer} | 0.000 |
| {planner, writer} | 1.000 |
| {retriever, writer} | 1.000 |
| {planner, retriever, writer} | 1.000 |
| {reviewer} | 1.000 |
| {planner, reviewer} | 1.000 |
| {retriever, reviewer} | 1.000 |
| {planner, retriever, reviewer} | 1.000 |
| {writer, reviewer} | 1.000 |
| {planner, writer, reviewer} | 1.000 |
| {retriever, writer, reviewer} | 1.000 |
| {planner, retriever, writer, reviewer} | 1.000 |
Outputs: factual vs. all-fixed
| value | factual | all fixes |
|---|---|---|
| question | 'How many days do I have to request a refund?' | 'How many days do I have to request a refund?' |
| planner | {'keywords': ['refund', 'policy'], 'year': None} | {'keywords': ['refund', 'policy'], 'year': 2026} |
| retriever | {'id': 'policy-2023', 'year': 2023, 'superseded': True, 'text': 'Refund policy: refund policy allows refunds within 90 days of purchase.'} | {'id': 'policy-2026', 'year': 2026, 'superseded': False, 'text': 'Refunds are accepted within 30 days of purchase (effective 2026).'} |
| writer | {'claims': [{'text': 'Refunds are accepted within 90 days.', 'cite': None}]} | {'claims': [{'text': 'Refunds are accepted within 30 days.', 'cite': 'policy-2026'}]} |
| reviewer | 'Refunds are accepted within 90 days.' | 'Refunds are accepted within 30 days.' |
Legend: fix alone = v({i}) − v(∅); leave-one-out = v(all) − v(all∖{i}); Shapley splits v(all) − v(∅) fairly over all orders of applying fixes; ◆ = naive last-step-to-touch blame. Generated by agentblame.