In an agent-first workflow where an agent harness already handles diff generation, basic tests, and risk-tagging, what concrete signals in a PR (e.g., boundary movement, cross-feature coupling, verification gaps, unclear intent) should automatically escalate a change from a fast correctness lane into a high-judgment review lane, and how can teams iteratively tune those escalation rules using escaped-defect and incident data rather than gut feel?

dhh-agent-first-software-craft | Updated at

Answer

Proposed lane rule set and tuning loop:

  1. Escalation signals for judgment lane

A. Boundary / architecture moves

  • Files in boundary maps: module / service / façade boundaries, auth, money, PII, feature flags.
  • New cross-boundary calls or data flows (e.g., crossing bounded contexts, new joins across domains).
  • Schema, migration, or contract changes used by >1 feature or external client.
  • Large fan-out: diff touches many modules or entrypoints.

B. Cross-feature coupling and behavior shifts

  • Shared primitives: pricing, permissions, routing, experiment framework, email/SMS sending.
  • Condition changes on key flags / roles / plans.
  • Removal or bypass of existing guards, rate limits, or fallbacks.
  • Text changes that alter user-visible promises, legal copy, or support assumptions.

C. Verification gaps

  • Harness sees:
    • New code paths without direct tests or scenario coverage.
    • Reduced assertion density or removed checks around critical ops.
    • TODOs / FIXMEs / commented-out tests in touched files.
    • Failing non-flaky tests re-tagged as "ignored" or "quarantined" in same PR.

D. Risk and intent smell

  • PR description missing: problem, approach, rollback plan.
  • Agent cannot auto-generate a clear intent summary from diff + issue.
  • Diff size above team threshold (e.g., >400 LOC changed) or high churn in same area.
  • Owner mismatch: PR in a critical area by someone outside owning team.

E. Meta / surface-area changes

  • New CLI commands, harness tools, or API endpoints.
  • New background jobs, schedulers, or external integrations.
  • Changes to verification layer semantics (what gets tested, severity levels, policy rules).

Implementation sketch:

  • Fast lane: default. Judgment lane: triggered if any hard rule matches.
  • Make rules declarative (e.g., YAML/JSON): paths, tags, thresholds, keywords.
  • Show the reviewer a short “Why escalated?” summary.
  1. Using escaped-defect and incident data to tune rules

A. Tag and trace

  • For each PR at merge: log lane, rules that fired, domains touched.
  • For each incident / escaped defect: link to causing PR(s) and mark type:
    • correctness, boundary, coupling, verification gap, UX/domain.

B. Simple feedback rules

  • If an incident comes from a fast-lane PR:
    • Ask: which signal could have caught it? (e.g., boundary file, untested path, missing intent.)
    • Add or tighten a rule in that category (new path pattern, stricter test requirement, keyword).
  • If many judgment-lane PRs are clean and trivial:
    • Lower sensitivity: raise size thresholds, narrow path patterns, or require multiple weak signals instead of one.

C. Cadenced tuning

  • Every 4–8 weeks:
    • Sample N incidents and N random PRs per lane.
    • For incidents, record: "should have escalated?" and "missing signal?".
    • For random judgment-lane PRs, record: "over-escalated?".
  • Adjust:
    • Path lists (critical areas), size limits, test-coverage thresholds.
    • Which signals are hard gates vs soft nudges.

D. Use agents as analysts, not judges

  • Have an agent propose rule diffs based on tagged incidents ("PRs causing boundary incidents often touched X/Y; suggest escalating those paths").
  • Humans approve rule changes via normal review, same as code.
  1. Guardrails
  • Keep rule set small: start with 5–10 high-signal patterns (critical paths, schema, verification changes, new surfaces, missing intent).
  • Review false positives / negatives explicitly in tuning meetings.
  • Never let rules replace human override: allow manual promote/demote between lanes with short rationale.

This keeps correctness fast by default, but routes any change that alters boundaries, shared behavior, verification strength, or intent clarity into a lane where senior judgment is required, and it evolves based on real misses rather than folklore.