review-calibration
Measures reviewer recall by planting bugs and scoring whether the review command catches them.
review-calibration
ADLC phase: calibration (cross-phase) · Gate: the reviewer catches at least --min-recall of planted bugs, proven against a controlled scorer.
What it defends against
"We do adversarial review" is a vibe until it is a number. review-calibration
plants real bugs into files changed by a commit, runs your review command, and
scores recall. A built-in echo control scores ~0 and an oracle control scores
1.0, so the recall figure is trustworthy rather than gamed by a reviewer that
merely echoes changed lines.
Those two controls run through a deterministic reference judge, which bounds the
scorer's aggregation. The figure itself is produced by the configured
judge, so the echo control runs a second time through that judge whenever it
rendered a verdict: in the default judge mode a judge that cannot reject an
echoing reviewer is an operational failure (exit 1, no figure emitted).
--scorer string makes the judge match everything by construction and so can
never be bounded — that run still completes and reports
configuredJudgeBounded: false, making the uncertified number visible to a
machine consumer. When no finding located a plant the judge was never consulted,
and configuredJudgeBounded / configuredJudgeEchoRecall are null.
Usage
adlc review-calibration --review-cmd "cmd with {base} placeholder" [options]| Flag | Description |
|---|---|
--review-cmd <cmd> | Required. Command that runs the reviewer; {base} is substituted with the commit ref. |
--commit <ref> | Commit whose changed files are the plant targets (default HEAD). |
--plants <n> | Number of bugs to plant (default 8). |
--min-recall <f> | Minimum recall fraction to pass, 0–1 (default 0.5). |
--min-precision <f> | Optional minimum precision to also require. |
--scorer judge|string | judge (default) is cheap-model semantic match; string is the gameable legacy location-only fallback (warns). |
--files <list> | Comma-separated fallback file list when the commit has no eligible code. |
--plants-file <path> | JSON array of authored plants [{file,line,original,mutated,category?,defect?,witness?}]. |
--tier cheap|mid|frontier | Model tier for the LLM judge (default cheap). |
--review-provider <name> | Provider family --review-cmd runs on; compared against the judge for independence. |
--strict | Gate-fail instead of warn when --review-provider matches the judge's provider. |
--json | Machine-readable output. |
--prompt-only | Print a representative judge prompt and exit 0 (no API key or git repo needed). |
Exit codes
--strict provider match.Example
$ adlc review-calibration --review-cmd "adversarial-review --base {base} --json" --plants 8 --min-recall 0.6
recall 62.5% (5/8 caught) · precision 83%
✗ gate fails — recall 62.5% (min 60.0%)
exit 2Go deeper
packages/review-calibration. Why the review itself must be prosecuted: Prosecution, Not Code Review.