Bayesian build-outcome confidence: telling Claude when its plan is likely to fail before it costs you a build

Devin shows its user a confidence score. Claude Code does not — and yet every long-running Claude Code setup is quietly generating exactly the data needed to compute one. Every task the agent runs either builds or it doesn’t; the tests pass or they don’t; the retry counter tells you how hard the step fought. Log it all, and the question “how likely is this plan step to fail?” stops being a feeling and becomes arithmetic.

We call the mechanism Bayesian build-outcome confidence, and it is the most statistical of the twelve guardrail layers we run on .NET engagements. This post shows the real implementation: the outcome log, the scoring formula (pseudo-counts plus time decay — deliberately boring math), the plan-time risk analyzer that complements it, and — because moat terms deserve honest edges — where the whole approach stops working.

You cannot be Bayesian without evidence

The foundation is unglamorous: a log. Every task run appends a record to build-outcomes.json:

{
  timestamp: "2026-07-18T14:22:07Z",
  feature_id: "F-142",
  task_id: "TASK_11",
  task_type: "backend",          // backend | frontend | infrastructure | test | docs
  model: "sonnet",               // which tier executed the step
  effort: "high",
  pattern_used: "api/endpoint",  // which implementation pattern the step followed
  build_result: "success",
  test_result: "success",        // success | failure | skipped
  retry_count: 0,
  duration_ms: 184000,
  files_changed: ["Api/Features/Billing/Endpoints/CancelPlan/..."],
  error_summary: null
}

Two fields do most of the later work. pattern_used names which implementation pattern the step followed — api/endpoint, ef/migration, blazor/component — which is what lets outcomes aggregate into something with statistical mass instead of a pile of one-off anecdotes. And retry_count captures the failures that didn’t end up failing: a step that succeeded on its third attempt is telling you something a green checkmark hides.

The writer is the quality gate — the Stop hook that runs the build after each response — so logging is nobody’s job and therefore actually happens. One production detail worth copying: writes go through a lock file with exponential backoff, because two concurrent sessions appending JSON to the same file will eventually corrupt it, and a corrupted evidence log fails the whole layer silently.

The score: pseudo-counts and time decay

Confidence updating runs over the log, groups outcomes by pattern, and updates each pattern’s confidence metadata. The core is small enough to show whole:

// Bayesian score with pseudo-counts (2 pseudo-successes to avoid 0/1 extremes)
const adjustedSuccesses = conf.successes + 2;
const adjustedTotal = conf.evidence_count + 2;
const rawScore = adjustedSuccesses / adjustedTotal;

// Time decay — a pattern validated months ago means less today
const weeksSinceLastSuccess = (now - lastValidatedTs) / (7 * 24 * 3600 * 1000);
const decayedScore = rawScore * Math.pow(1 - conf.decay_rate, weeksSinceLastSuccess);

conf.score = Math.max(0, Math.min(1, decayedScore));

Three deliberate choices in eleven lines:

Pseudo-counts, not raw ratios. A pattern with one observed failure would score 0.0 on raw frequency; one success would score a perfect 1.0. Neither number deserves belief. Adding two pseudo-successes (a Beta prior, if you want the formal name) keeps early scores away from the extremes until real evidence accumulates — a pattern at 3 successes, 1 failure scores (3+2)/(4+2) ≈ 0.83, not 0.75, and moves smoothly as evidence arrives.

Time decay. Codebases drift. A pattern validated forty times in March has not been validated against July’s dependency bumps, renamed conventions, and new analyzer rules. Confidence decays ~1% per week since last validation, which means an unused pattern quietly slides from “trusted” toward “re-verify before leaning on this” without anyone maintaining it.

Atomic writes into the pattern files themselves. The score lives inside the same pattern snapshot JSON the implementing subagent already reads — so consuming confidence requires no extra lookup. The agent learns the pattern and its trustworthiness in one read. Each snapshot also carries a contradictions list: places where observed code diverged from the documented pattern, which is the qualitative sibling of the quantitative score.

The other half: plan-time cascade risk

Pattern confidence is retrospective — it tells you how a kind of step has gone historically. The second component is prospective: before execution, a risk analyzer reads each planned task and classifies it low / medium / high for the specific failure mode that hurts most on .NET solutions — the cross-domain cascade, where a change fans out through shared components into dozens of build errors the agent then chases for a whole session.

The features are counted straight from the task spec: how many files it touches, how many cross-domain consumers it reaches (shared components — selectors, editors, shared UI — used by multiple feature areas), and a rough error estimate:

estimated_errors = file_count × 1.2 + consumers × 8

high:   file_count > 60  OR  consumers ≥ 4  OR  estimated_errors > 100
medium: file_count > 30  OR  consumers ≥ 2  OR  estimated_errors > 50

Those thresholds were not designed; they were calibrated — tuned against the post-mortem archive of a real migration task that produced roughly a hundred mid-migration build errors and burned a session failing to drive them monotonically to zero. The analyzer does not claim to predict the error count. It claims to recognize the shape of that incident before it happens again, which is a humbler and more useful promise.

What the numbers change, concretely

Scores that nobody acts on are dashboard decoration. Here is a representative three-step plan on a .NET solution, and what the two mechanisms do to it:

Step 1 — add a cancel-subscription endpoint. Pattern api/endpoint, 40+ logged outcomes, confidence 0.94, four files touched, zero shared consumers. Proceed without ceremony — this is the mechanical work the whole system exists to wave through.

Step 2 — change the shared MoneyDisplay component’s rounding. Nine files, but five cross-domain consumers → estimated errors ≈ 51 → medium-high risk. The response is not “don’t do it”; it is batch differently: the step gets grouped with its consumer updates so one build verdict covers the whole blast radius, and the build gate runs on the batch instead of letting errors dribble out across five subsequent steps.

Step 3 — first use of a brand-new webhooks/receiver pattern. Zero evidence. Here the pseudo-count choice shows its one sharp edge honestly: with no observations, the score starts at (0+2)/(0+2) = 1.0 — optimistic by construction — and only decay and evidence move it. So the routing rule for new patterns keys on evidence_count, not score: fewer than a handful of observations → the step runs at higher scrutiny (stronger model tier, human review of the diff) regardless of the number. A confidence system must know when it has no basis for confidence.

The general shape: high confidence + low risk → proceed; anything else → batch, escalate the model tier, or pause for a human — decided before the tokens are spent, not after the build breaks. On our own template build, this layer flagged two architectural steps whose pattern evidence contradicted the plan’s assumptions; both would have cost roughly a sprint each to unwind past the point of execution.

Prior art, and what’s actually different

Predicting build failures is not a new idea. USPTO patent 10,684,851 (“predicting build failures”) covers modeling failure probability from change features in CI, and AutoStan (arXiv:2603.27766) is the recent academic treatment of the same territory. Devin, from the product side, popularized showing a confidence number to the user.

The distinction we would defend: in all three, the number routes human attention — after the fact, or as UI transparency. In this design the number feeds the agent’s own scheduler before execution: it changes batching, model-tier routing, and pause decisions mechanically. A score that only informs is a report; a score that reorders the plan is a guardrail. The glossary entry for the term is here.

Building a minimal version this week

The full loop sounds like infrastructure; the starter version is an afternoon. In adoption order:

1. Log outcomes first, decide nothing yet. In your Stop hook (or wherever your build gate runs), append one JSON record per task: timestamp, a pattern label, build result, test result, retry count. The pattern label can be crude — endpoint, migration, component, other gets you surprisingly far. Do not build the scorer until the log has two weeks of entries; scoring an empty log teaches you nothing except how your formula handles zero.

2. Score weekly, by hand at first. A twenty-line script: group by pattern, apply the pseudo-count formula, print the table. The first time you see migration: 0.62 (13 obs) next to endpoint: 0.96 (41 obs), you will already know things about your setup that no amount of session-watching told you — usually that one pattern’s failures were being absorbed as retries nobody counted.

3. Wire in the cheapest intervention. Skip the scheduler integration initially; just inject the scores into the context the agent reads (your CLAUDE.md or pattern files) with one routing rule: steps using a pattern below 0.8, or with fewer than five observations, get their diff reviewed by a human before merge. That single rule captures most of the layer’s value at almost none of its complexity.

4. Add plan-time risk only if cascades are your failure mode. The file-count and shared-consumer heuristics matter on large layered solutions; a small modular codebase may never trip them. Calibrate thresholds from your own worst incident, not from ours.

The ordering matters because each step generates the justification (or not) for the next — which is itself the Bayesian posture: collect evidence, then commit.

The limits, stated plainly

This is a heuristic stack, not a guarantee, and it fails in knowable ways. The features are correlational — pattern_used and file counts, not causal understanding of your architecture. Small evidence counts are noisy, and the pseudo-counts only soften that, they do not fix it. The decay rate is hand-tuned, not learned. And a plan step can be high-confidence, low-risk, and still wrong in a way no build catches — confidence in building is not confidence in correctness, which is what the test suite and the human review layers are for.

What it reliably does is cheaper and narrower: it prevents the expensive class of agent mistake — the confident execution of a step whose kind has been failing, or whose blast radius matches a known incident shape. On a five-person team, that class was where whole afternoons went.

Where it sits

Bayesian build-outcome confidence is layer three of the twelve-layer system. It consumes the evidence the build gate generates, reads and writes the pattern snapshots, and hands its verdicts to the same scheduling logic the oscillation detector escalates to. Like the AUTO-TABLE layer, it starts paying for itself the day the log starts filling — and the log is the part you can start writing this week: one JSON append in your Stop hook, and in a month you have the evidence base this entire post is built on.

We stand the full loop up as part of Sprint Zero. The moat here was never the formula — it is eleven lines — but the discipline of logging every outcome and letting the numbers veto the plan.