Paper → experiment → product pipeline

When to read this: You need the end-to-end story of how SOTA agentic papers land in Chatty — what gets experimented on, what the human reviewer decides, and what becomes production code vs a user setting vs nothing.

This is the organizing frame for research documentation. Individual papers, crate promises, and ADRs hang off it.

The flow

flowchart LR
  Papers[SOTA papers] --> StageA[Stage A<br/>in-repo fidelity]
  StageA --> StageB[Stage B<br/>Harbor benchmarks]
  StageB --> Human[Human evaluation<br/>paired stats + ablations]
  Human -->|reject| Archive[Document why not]
  Human -->|setting| Setting[User-configurable<br/>opt-in feature]
  Human -->|default| Default[Ship as default<br/>when dominating]

  StageA --> Crates[Research crates<br/>trace / playbook / flow / optimize]
  Setting --> Core[chatty-core settings<br/>+ shipping crates]
  Default --> Core

Rule: papers inform the design; experiments decide what ships. Reproducing a paper is necessary but not sufficient — cross-module results (AGE-21) and product constraints (AGE-26) filter what becomes Chatty.

Two experiment stages

Stage A answers “did we implement the idea correctly?” Stage B answers “does it help on tasks users care about?” Neither stage auto-promotes to production.

See Harbor pivot for why Stage B lives outside this repo.

Three promotion outcomes

Every paper mechanism ends in exactly one bucket after experimentation:

Dominance bar for defaults: sustained gain on Stage B (or representative in-app tasks), acceptable cost/latency, and no regression on the production bar (AGE-26). A paper win on HotpotQA alone does not automatically become a Chatty default.

Settings surface: persisted models in chatty-core settings (ModelConfig, future FlowSettingsModel, playbook storage — see product gate issues below). Optimizer output (preamble, playbook bullets, workflow IR) lands through an apply policy the human reviewer must choose (AGE-45).

Paper → crate → product landing

Linear project: Self-improving chatty2.

Inference path vs offline tooling

flowchart TB
  subgraph request [Request path — must stay fast]
    UI[GPUI / TUI] --> Core[chatty-core]
    Core --> Trace[chatty-trace Recorder]
    Core --> Playbook[chatty-playbook]
    Core --> Flow[chatty-flow interpreter]
  end

  subgraph offline [Offline — never blocks chat]
    Opt[chatty-optimize<br/>GEPA / AFlow search]
    Harbor[harbor-chatty Stage B]
    Opt --> Artifacts[preamble / playbook / workflow IR]
    Artifacts -.->|apply policy| Core
  end

  Trace --> Opt
  Playbook --> Opt
  Flow --> Opt
  Core --> Harbor

Shipping crates (chatty-core, chatty-trace, chatty-playbook, chatty-flow) are held to the production bar (AGE-26). Build-time crates (chatty-optimize) are held to correct, not latency-safe.

What the human reviewer owns vs what agents build

Agents must not implement reserved symbols, close gate issues, or predict benchmark outcomes.

Open product gates (block promotion UX)

These human product decisions define how optimized artifacts reach users:

Project: Chatty agentic product integration.

Documentation layers for this pipeline

Rigorous docs mirror the pipeline — each layer answers a different question:

Completed doc layers

1. Per-paper module pages — modules/
2. Settings integration map — settings-integration-map.md
3. Experiment protocol — experiment-protocol.md

Still open

1. Link checker CI (DOC-50)
2. README → /user/ migration (DOC-15, human review)

Related reading

• RESERVED.md — symbols agents must not implement
• Production bar (AGE-26) — shipping crate requirements
• Cross-module experiments (AGE-21) — human-only interpretation
• cost-model.md — optimizer economics
• AGENTS.md — agent quick-start and workspace map