---
title: "The decision-outcome benchmark · Mentat"
description: "Can a system predict which architectural decisions get reversed? Mentat publishes the task corpus, the trivial baselines, and its own low score, with Matthews correlation as the headline so a constant predictor scores zero."
url: https://trymentat.pages.dev/benchmark
---

00Benchmark

# Can anyone predict a reversal?

Every category-defining company eventually publishes the benchmark its category is measured on. Nobody has published this one, so here it is, including the part where our own predictor gets half the reversals wrong.

HEADLINE

0.612 MCC

Matthews correlation, which is zero for any predictor that answers the same way every time. That is why it is the headline and accuracy is not.

PRECISION / RECALL

1.000 / 0.500

It never cries wolf, and it misses half the reversals. That is the honest shape of a first attempt, and it is the number a competitor has to beat.

CORPUS

20 tasks

40% of them reversed. The corpus over-samples reversal on purpose, so the base rate is printed above every table.

01Results

## The baselines stay in the table.

A benchmark whose author tops it convinces nobody. Both trivial predictors are reported on every run, and the runner refuses to hide them: if ours ever fails to beat "nothing ever changes", the table says so on its own.

| Predictor | MCC | F1 | Precision | Recall | Accuracy | Brier |
| --- | --- | --- | --- | --- | --- | --- |
| **always-holds** | 0.000 | 0.000 | 0.000 | 0.000 | 0.600 | 0.400 |
| **always-reverses** | 0.000 | 0.571 | 0.400 | 1.000 | 0.400 | 0.600 |
| **mentat-graph-heuristic** | 0.612 | 0.667 | 1.000 | 0.500 | 0.800 | 0.129 |

Read the second row carefully. Answering "nothing ever reverses" scores 0.600 on accuracy with no knowledge of anything, which is why a benchmark that leads with accuracy on a skewed task is a benchmark measuring the skew.

02The predictor

## Six signals, all from the graph.

Deliberately simple, and deliberately published. Every signal it uses is one the product already tracks, so improving the benchmark and improving the product are the same work. It starts at a 0.12 base risk and adds.

+0.34

rests on a questioned or broken premise

The strongest signal the graph carries, and the entire thesis of the product. A rule whose reason stopped being true is a rule someone is about to argue with.

+0.16

no recorded justification

A decision with no stated why has nothing to defend it with the first time it is inconvenient.

+0.12

scoped to the whole repository

A repository-wide mandate collides with something eventually, and the carve-out usually wins.

+0.20

already questioned

Being in trouble is predictive of staying in trouble.

+0.08

low extraction confidence

An extracted decision nobody enriched is weaker than one a human typed.

+0.06

permissive rather than mandatory

The exception someone took during a migration is the thing that gets pulled back.

03Run it

## Reproduce it in one command.

The corpus, the scoring, and the baselines all ship in the CLI. Nothing here needs an account or an API key, and a pull request that makes the predictor worse fails the gate rather than quietly lowering the figure above.

RUN THE BENCHMARK

```
npx @mentatlabs/cli bench
```

INCLUDE THIS REPOSITORY'S OWN DECISIONS

```
npx @mentatlabs/cli bench --self
```

FROM A CLONE, WITH THE CI GATE

```
npm run bench:gate
```

20 synthetic0 donated0 self

Three rules the corpus follows. The split is skewed and the skew is printed. No task is solvable from the constraint text alone, or the benchmark would measure sentiment on architecture prose. And nothing in it is customer data: donated tasks are labelled and counted separately, so a reader can see how much of a score came from situations we wrote ourselves.

Have a decision that got reversed, or one that held when everyone expected it not to? That is the most valuable contribution this project can receive, and there is an issue template for it.

[Every other number we cite](https://trymentat.pages.dev/evidence)

Every figure on this page is imported from what npm run bench produced, and CI fails if the committed copy is not what the harness scored. None of it is typed by hand.

---

Mentat: https://trymentat.pages.dev/ · Docs: https://trymentat.pages.dev/docs · Developers: https://trymentat.pages.dev/developers · API: https://trymentat.pages.dev/openapi.json · Index for agents: https://trymentat.pages.dev/llms.txt
