Evidence

Every number, with its source.

Including the ones that cut against us. Decision extraction is a thirty-year field with four dead projects in it, and pretending otherwise would be a strange way to open a conversation about intellectual honesty in engineering decisions.

The thesis

Agents already notice the contradiction. They say nothing, pick a reading, and pick a different one next session. Nobody owns that arbitration layer.

ClaimFigureSource
Models detect conflicting instructions and almost never surface them87.3% F1 detection, rarely flaggedAugment Code, "Your agent's context is a junk drawer", March 2026
The whole thesis. Detection is solved; the silence is the problem.
Instruction compliance degrades past a ceilingroughly 150 to 200 instructionsFrontier model research, February 2026
Adding a rule when the agent misbehaves makes compliance worse at scale.
Bloated instruction files reduce compliancequalitativeAnthropic documentation
Instruction files are mostly policy, and a third is context64% policy, 36% descriptive, across 2,116 statementsStatement-level study of 84 CLAUDE.md and AGENTS.md files, 64 repositories, July 2026
Independent of us, and the reason only two of five statement types are ever compiled into a check. Their conclusion is ours: teams are not short of rules, the rules are just not in a form a system can observe.
A short recency window beats a longer one for decision context3 to 5 prior records is near-optimalContext Matters, EASE 2026, 750 repositories, arXiv 2604.03826
Context strategy mattered more than model size. Evidence that compiling a scoped file beats appending to a growing one.
Architectural decisions resting on stale evidence20 to 25% within two monthsFirst Principles Framework, arXiv 2601.21116
Why assumptions are nodes rather than a field.
Best model accuracy at acting on state it knows is stale55.2%STALE benchmark, arXiv 2605.06527
Expiry cannot be inferred at read time.

Why precision is the design constraint

Automated violation detection is good in aggregate and bad in specific places, and the places are known. Mentat refuses to auto-enforce exactly the two categories where two thirds of the errors live.

ClaimFigureSource
LLM detection of decision violations91.1% accuracy, 0.934 precision on non-compliantarXiv 2602.07609, 980 ADRs across 109 repositories, February 2026
Where that detection failsinfrastructure 42%, intent-oriented 26% of all errorssame
Exactly why those two statement types are never auto-enforced.
Why it failssemantic misinterpretation 44.6%, missing context 28.3%same
LLM-generated rationale against expert ground truthprecision 0.27, recall 0.65; 64 to 69% of extras still helpful, 1.6 to 3.2% misleadingRationale generation survey
Extraction over-generates but rarely poisons. Hence: extract generously, never auto-promote. These are the literature's numbers for model-based extraction and not a measurement of ours, which is deterministic and has no published figure yet.

The graveyard

Kantara, RecovAr, Adex, and DRMiner all built a decision graph, and several of them extracted from it well. What died was not the technique, it was the supply: rationale had to be written down by hand, and it never was. Mentat is a bet on one thing having changed about where the discussion happens, not on being better at reading it.

ClaimFigureSource
Rationale documentation dies on cost, not on toolingthe authors' own diagnosisDRMiner, arXiv 2405.19623
The cost to developers exceeds the value they perceive. That is what killed adoption, and it is the sentence this product is built around.
Mining rationale from issue logs, once the discussion exists65% F1, and 14x more full-match automated repairsDRMiner, ASE 2024
Cuts against the easy version of our story. Extraction is not the unsolved part and we should not imply it is. What no prior project had was a channel where the discussion happens as a side effect of the work.
Architectural decision recovery from issue trackers and history75% recall, 77% precisionRecovAr, ICSA 2018
Also a working technique. The graph was never the hard part. Prior work built it and assumed the inputs.
ADR adoption is shallowabout half of adopting repositories hold 1 to 5 recordsBuchgeher et al., IEEE Access
Agent session logs are structured and availabletranscripts, thinking traces, tool calls with file pathsSWE-chat, arXiv 2604.20779, April 2026
The capture channel that did not exist for any prior attempt.

Ambient conditions

Real and citable, but context rather than argument. Code volume is rising while structural discipline falls, which makes the constraint set the only thing holding shape.

ClaimFigureSource
Refactoring collapsemoved code 21% to 3.8% of changed lines, 2022 to 2026GitClear, The Maintainability Gap, 2026
Duplication rise+81% versus 2023same

What we do not claim

Two things are frequently cited in this space that are either mischaracterised or stale. Mentat does not use either.

DROPPED · MISCHARACTERISED

The Amazon "spec drift" outage

The March 2026 incident is widely retold as AI-generated code going wrong. Amazon attributed it to an engineer acting on inaccurate advice an agent inferred from an outdated internal wiki. That is an assumption node nobody marked dead, which is closer to our thesis than the popular version, but it is not spec drift and somebody in the room will know.

DROPPED · STALE

The Replit production database deletion

July 2025, and it has been used to sell everything in this category since. It says nothing about whether a decision was recorded, owned, or still true, so it does not appear anywhere in how Mentat is positioned.

The experiment that decides this

Twenty real agent sessions. Hand-extract every constraint a human articulated to correct the agent, with its justification, its assumption, and its scope. Pass is 15 or more usable constraints, under ten minutes per session, with fewer than 30 percent requiring invented context. Anything softer than that is a fail.

npm run gate

The gate is a script in this repository, not a claim on a slide. It runs in CI against fixture transcripts and exits non-zero when the extraction rate drops below the bar. If the capture channel stops working, the build says so.