Evidence
Every number, with its source.
Including the ones that cut against us. Decision extraction is a thirty-year field with four dead projects in it, and pretending otherwise would be a strange way to open a conversation about intellectual honesty in engineering decisions.
The thesis
Agents already notice the contradiction. They say nothing, pick a reading, and pick a different one next session. Nobody owns that arbitration layer.
| Claim | Figure | Source |
|---|---|---|
| Models detect conflicting instructions and almost never surface them | 87.3% F1 detection, rarely flagged | Augment Code, "Your agent's context is a junk drawer", March 2026 The whole thesis. Detection is solved; the silence is the problem. |
| Instruction compliance degrades past a ceiling | roughly 150 to 200 instructions | Frontier model research, February 2026 Adding a rule when the agent misbehaves makes compliance worse at scale. |
| Bloated instruction files reduce compliance | qualitative | Anthropic documentation |
| Instruction files are mostly policy, and a third is context | 64% policy, 36% descriptive, across 2,116 statements | Statement-level study of 84 CLAUDE.md and AGENTS.md files, 64 repositories, July 2026 Independent of us, and the reason only two of five statement types are ever compiled into a check. Their conclusion is ours: teams are not short of rules, the rules are just not in a form a system can observe. |
| A short recency window beats a longer one for decision context | 3 to 5 prior records is near-optimal | Context Matters, EASE 2026, 750 repositories, arXiv 2604.03826 Context strategy mattered more than model size. Evidence that compiling a scoped file beats appending to a growing one. |
| Architectural decisions resting on stale evidence | 20 to 25% within two months | First Principles Framework, arXiv 2601.21116 Why assumptions are nodes rather than a field. |
| Best model accuracy at acting on state it knows is stale | 55.2% | STALE benchmark, arXiv 2605.06527 Expiry cannot be inferred at read time. |
Why precision is the design constraint
Automated violation detection is good in aggregate and bad in specific places, and the places are known. Mentat refuses to auto-enforce exactly the two categories where two thirds of the errors live.
| Claim | Figure | Source |
|---|---|---|
| LLM detection of decision violations | 91.1% accuracy, 0.934 precision on non-compliant | arXiv 2602.07609, 980 ADRs across 109 repositories, February 2026 |
| Where that detection fails | infrastructure 42%, intent-oriented 26% of all errors | same Exactly why those two statement types are never auto-enforced. |
| Why it fails | semantic misinterpretation 44.6%, missing context 28.3% | same |
| LLM-generated rationale against expert ground truth | precision 0.27, recall 0.65; 64 to 69% of extras still helpful, 1.6 to 3.2% misleading | Rationale generation survey Extraction over-generates but rarely poisons. Hence: extract generously, never auto-promote. These are the literature's numbers for model-based extraction and not a measurement of ours, which is deterministic and has no published figure yet. |
The graveyard
Kantara, RecovAr, Adex, and DRMiner all built a decision graph, and several of them extracted from it well. What died was not the technique, it was the supply: rationale had to be written down by hand, and it never was. Mentat is a bet on one thing having changed about where the discussion happens, not on being better at reading it.
| Claim | Figure | Source |
|---|---|---|
| Rationale documentation dies on cost, not on tooling | the authors' own diagnosis | DRMiner, arXiv 2405.19623 The cost to developers exceeds the value they perceive. That is what killed adoption, and it is the sentence this product is built around. |
| Mining rationale from issue logs, once the discussion exists | 65% F1, and 14x more full-match automated repairs | DRMiner, ASE 2024 Cuts against the easy version of our story. Extraction is not the unsolved part and we should not imply it is. What no prior project had was a channel where the discussion happens as a side effect of the work. |
| Architectural decision recovery from issue trackers and history | 75% recall, 77% precision | RecovAr, ICSA 2018 Also a working technique. The graph was never the hard part. Prior work built it and assumed the inputs. |
| ADR adoption is shallow | about half of adopting repositories hold 1 to 5 records | Buchgeher et al., IEEE Access |
| Agent session logs are structured and available | transcripts, thinking traces, tool calls with file paths | SWE-chat, arXiv 2604.20779, April 2026 The capture channel that did not exist for any prior attempt. |
Ambient conditions
Real and citable, but context rather than argument. Code volume is rising while structural discipline falls, which makes the constraint set the only thing holding shape.
| Claim | Figure | Source |
|---|---|---|
| Refactoring collapse | moved code 21% to 3.8% of changed lines, 2022 to 2026 | GitClear, The Maintainability Gap, 2026 |
| Duplication rise | +81% versus 2023 | same |
What we do not claim
Two things are frequently cited in this space that are either mischaracterised or stale. Mentat does not use either.
DROPPED · MISCHARACTERISED
The Amazon "spec drift" outage
The March 2026 incident is widely retold as AI-generated code going wrong. Amazon attributed it to an engineer acting on inaccurate advice an agent inferred from an outdated internal wiki. That is an assumption node nobody marked dead, which is closer to our thesis than the popular version, but it is not spec drift and somebody in the room will know.
DROPPED · STALE
The Replit production database deletion
July 2025, and it has been used to sell everything in this category since. It says nothing about whether a decision was recorded, owned, or still true, so it does not appear anywhere in how Mentat is positioned.
The experiment that decides this
Twenty real agent sessions. Hand-extract every constraint a human articulated to correct the agent, with its justification, its assumption, and its scope. Pass is 15 or more usable constraints, under ten minutes per session, with fewer than 30 percent requiring invented context. Anything softer than that is a fail.
npm run gate
The gate is a script in this repository, not a claim on a slide. It runs in CI against fixture transcripts and exits non-zero when the extraction rate drops below the bar. If the capture channel stops working, the build says so.