Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Experiment protocol

When to read this: You are running or reviewing Stage A/B experiments and need the shared bar for when results count toward a promotion decision.

Human interpretation: AGE-21 (the human reviewer only). Agents: build harnesses and cost sheets; do not write up cross-module conclusions.

Two stages

StageLocationPass criterion
A — Fidelitychatty2 research cratesMechanism matches paper; acceptance criteria in module page checked
B — Task valueharbor-chattyGain on representative tasks under paired stats; cost within budget

Neither stage auto-promotes; see “Promotion decision inputs” below.

Stage A checklist (per module)

  1. Read the module page acceptance criteria.
  2. Run with fixed seed + warm response cache where determinism is claimed.
  3. Separate training vs validation rollouts (critical for M3 GEPA).
  4. Log artifact token counts when comparing ACE vs GEPA (M4 vs M3).
  5. File failures as Linear issues — do not patch reserved symbols.

Stage B checklist

  1. Use Harbor adapter (AGE-34) — not in-repo sandboxes.
  2. Report paired differences with confidence, not single-run deltas.
  3. Use cost model with live ModelConfig prices — never hand-typed dollars.
  4. The human reviewer posts authoritative numbers before they appear in docs or a promotion record.

Cost accounting

cost ≈ rollouts × calls_per_rollout × (in_tok×in_price + out_tok×out_price) / 1e6
       × (1 − cache_hit_rate) × seeds

Implementation: chatty_optimize::cost. Token means blocked on M0 rig-tap wiring (AGE-5).

Until pilot means exist, do not treat Stage B dollar figures as authoritative.

Determinism requirements

ClaimRequirement
Byte-identical tracesSame seed + warm cache
Reproducible workflow scoreSerialize → reload → re-execute
Stable playbook serializationNo-op update must not reorder sections (M4)

Cross-module experiments (AGE-21)

Reserved whole for the human reviewer. Examples:

  • ACE vs GEPA rollout counts on same task
  • Combined workflow + playbook + preamble
  • Ordering across modules no single paper reports

Agents may build plumbing; the human reviewer runs and interprets.

Promotion decision inputs

After Stage A + B, the human reviewer records (fields per the archived promotion-log template):

  1. Effect size — paired stats, MDE met?
  2. Cost / latency — acceptable on desktop agent?
  3. Production bar — AGE-26 satisfied for shipping crates?
  4. Product gates — AGE-44–47 answered?
  5. Verdict — rejected / setting / default