Eval and hillclimb

eval and tune follow the eval design and hillclimbing practice from Anthropic’s guide to automating eval design and hillclimbing. Rule count is not the metric; precision on real repositories is.

Corpus and split

data/corpus/*.jsonl holds labelled units under src/eval/corpus.ts. Every item gets a deterministic 80/20 dev/holdout split from a seeded hash of its id (or its source family, so paired examples stay on the same side), not a random draw per run. An edit only ever reads dev; holdout stays unseen until a validation run.

eval coverage reports label balance and split overlap without any API calls. Overlapping source files, example families or duplicate text between dev and holdout weaken held-out evidence, even before a threshold is chosen.

Threshold tuning and promotion

tune picks the lowest act threshold per rule on dev whose precision lower bound (Wilson, 95%) clears --floor (default 0.8). tune --write records the decision in data/rules/tuning.json, but only after promotionEvidence (src/eval/promotion.ts) validates that threshold on the independent holdout split. A rule whose holdout precision does not clear the floor is written review-only, even if dev looked strong: that is the overfit guard. Rerun tune --write once holdout evidence supports the rule; nothing promotes on train performance alone.

tune ab compares two question wordings for one rule on dev with a paired McNemar test, and warns when there are fewer than 10 discordant pairs (underpowered).

Diagnostics to run with every baseline

• Grader consistency (eval consistency): asks the same question twice with the answer cache bypassed and reports the flip rate with a Wilson interval. A wording or a precondition that is not actually deterministic shows up here before it costs a promotion cycle.
• Infrastructure errors: a timeout, a rejected key or a malformed response is recorded as , or (never a raw body). It is counted in and excluded from precision and recall. treats any error as incomplete, not a failing rule.