Home / Ops / More in this area

Affirmology - Quality Control: Audit, Plan, and Accountable Agents (v1)

Updated Jul 11, 2026 · Affirmology_QC_Audit_and_Plan_v1.md

Affirmology - Quality Control: Audit, Plan, and Accountable Agents (v1)

Written 2026-07-11 (DATA/engine chat) at Jeff's request, after he watched the mirror renders come out slow and arduous and got (rightly) scared about quality control. Jeff's steer: the #1 problem is DRIFT FROM STRUCTURE, especially while we are testing new structures. Generic / wishy-washy / length matter too, but we are already fixing those and they should be baked into a structure at approval time. So this plan is built around structural conformance first.

Read with: Affirmology_FirstReading_Structure_v1.md (the Mirror's spec'd 12-rule gate, doc-only today), Affirmology_ReadingCraft_HeroStory_Synthesis_v1.md (the craft bar), the memory self-healing-agent-loops (the parked accountability workstream this advances).

1. Executive summary (the honest diagnosis)

The pipeline reliably prevents broken and wrong. It has almost nothing preventing drifted or mediocre.

• What works today (enforced in code, every render): technical audio QC (dropouts + loudness), chart-integrity (content must match the person's real chart), and a mechanical script gate (em dashes, the Origin's fixed close, banned HD gate numbers, a very wide length band).
• The hole: there is no structural-conformance layer. A structure is a prose directive; the model interprets it; nothing verifies the render followed the intended beats, in order, carrying the threaded symbol, opening and closing the loop, sealing. The only shape enforcement anywhere is the Origin's fixed last line (enforce_origin_close).
• There is also no accountability: no render gets a quality/conformance score, nothing is HELD for being weak or drifted, and no structure has a measured pass-rate. A drifted 25-minute Mirror ships as easily as a perfect one; the only backstop is a human listening end to end.

This is fixable, and most of it is cheap. The core build is: give every structure a machine-checkable blueprint, add a conformance gate that scores each render against its blueprint and blocks drift, and wrap both in accountable agents that hold weak renders, log verdicts per person and per structure, and never silently ship or drop.

2. The audit - what QC exists, mapped

What the mechanical gate actually does (script_qc.py): replaces em/en dashes with commas; for the Origin, guarantees the exact fixed close (enforce_origin_close); bans raw HD gate/channel numbers; flags word count outside a band of target_minutes * 110 to * 175 (very wide - see §3). It is a pure deterministic pass. It does NOT read for meaning, structure, personalization, or tone.

Structures: 28 registered (api/structures.py:108). Each is a Structure dataclass whose shape lives in two free-text fields, directive (the locked concept passed to the generator, line 62) and overview (line 63). There is no ordered list of required beats, no per-beat lead-system, no threaded-symbol or open-loop requirement in any machine-readable form. The generator is TOLD the shape in prose and trusted to hit it.

The core finding (Jeff's #1 fear, confirmed)

Structural drift is unmeasured and unguarded. Because a structure is prose and the only shape check in the codebase is the Origin's closing line, a render can silently: skip a beat, reorder beats, lead with the wrong system, drop the threaded symbol, leave the loop unopened or unclosed, or omit the seal - and pass every gate we have. When we test a NEW structure, we have no automated signal for whether the output matched the intent; we find out by listening to the whole thing. That is the arduous, scary loop.

Secondary findings (real, but lower priority per Jeff)

• Length calibration is inconsistent across modules, which is why long renders overshoot. The generator prompts for target_minutes * 140..150 words (script_generator.py:646), the QC band is * 110..175 (script_qc.py:121), pacing uses PACE_WPM = deep 118 / standard 132 / activation 145 (pacing.py:39), and the soul-song calibration note is ~2.1 words/sec (~126 wpm). Four different words-per-minute truths. A single source of truth for length would stop the overshoot.
• verify_send.py is orphaned - a send gate nothing calls. Either wire it in or delete it so it is not mistaken for protection.
• The Mirror's 12-rule gate exists only in a doc. It should become the first blueprint (§4).

3. The plan - a conformance-first QC system

Principle: a structure is not "the prose we hope the model follows"; it is a checkable blueprint, and every render is scored against ITS blueprint before it can ship. Craft requirements (definite, personal, hope-dominant, length) get baked into the blueprint at approval time, so conformance to an approved structure implicitly enforces the craft bar - exactly Jeff's model.

3.1 Give every structure a machine-checkable blueprint

Add a blueprint to the Structure dataclass (additive; untagged structures keep behaving exactly as today). A blueprint is an ordered list of beats, each with: - id + human name (e.g. promise, being_seen, three_doors, install, seal) - required (hard) vs optional - lead_systems (which of astrology / HD / gene keys / numerology should drive this beat) - must_contain markers the render must exhibit (e.g. the threaded symbol is introduced in beat 1 and recurs; an open loop is opened and later closed; the seal is the final movement) - share_of_length (rough fraction of total words) so one beat can't eat the whole piece - plus structure-level rules: total length band (from the single source of truth), banned tokens, the fixed close (Origin), hope-dominant / definite / no-catastrophe, personalization floor.

3.2 A two-layer conformance gate (runs after generation, before render/ship)

1. Deterministic layer (free, instant): length vs the single source of truth, banned tokens, fixed close present, threaded-symbol string recurs N times, required section markers present and in order, chart-match (reuse chart_guard). Catches the cheap, obvious drift with zero tokens.
2. Judge layer (one cheap grounded LLM pass, ~pennies): given the blueprint + the script, score each required beat present / in-order / doing-its-job, and score the four craft dimensions Jeff named - structural drift (primary), generic/not-personal, wishy-washy/weak, length/pacing - each 0-100 with the specific offending lines. Returns a ConformanceVerdict.

3.3 Gate policy

• Auto-fix the mechanical (em dashes, close) - already done.
• BLOCK + regenerate on hard drift: a required beat missing or out of order, missing seal, threaded symbol dropped, length wildly off, or any craft dimension below a hard floor. Bounded retries with the verdict fed back as the fix instruction (no blind loops; stop after N and hold for human).
• FLAG (ship but log) on soft drift so we see trends without over-blocking.
• Never silently ship a hard-drifted render; never silently drop a held one.

3.4 A structure-test harness (directly serves "we are testing new ones")

When testing a new structure: generate K renders across K varied charts, score each against the blueprint, and report the drift pattern - which beat drifts most, whether the loop reliably closes, length variance - so you fix the directive OR the blueprint before approving. This turns new-structure testing from "listen to 25 minutes and hope" into "read a one-page conformance report."

4. Accountable agents

The render stops being fire-and-forget and becomes accountable to the structure's blueprint.

• ConformanceVerdict logged per render (person + structure + drift score + per-beat pass/fail + craft sub-scores + offending lines). Durable, queryable, so any chat sees the same truth (same pattern as the self-heal ledger).
• Structure scoreboard: per structure, its conformance pass-rate and mean drift across all renders. This makes the STRUCTURE accountable, not just the render - you can see at a glance which structures are reliable and which drift, which is the signal you need while iterating.
• Approval gate = the mechanism behind "when a structure is approved, that stuff is accounted for." A structure cannot be marked approved / production until it clears a conformance pass-rate threshold across its test renders. Approval literally means "its blueprint is defined and it passes." Until then it stays experimental and its renders are held for human review by default.
• Held, never dropped: a below-bar render is held with its specific failed beats, retried with that feedback (bounded), and surfaced - never silently shipped, never silently lost. Ties into the existing self-heal ledger so a held render is a pending job that gets worked, not forgotten.

5. Build sequence (cheap first, each independently useful)

1. Blueprint schema + the Mirror's blueprint (encode the doc's 12-rule gate as the first real blueprint). Small, no render change.
2. Deterministic conformance layer (free) wired into the compose path after generation. Catches order/beat-marker/threaded-symbol/length drift at zero token cost.
3. Single source of truth for length (kill the 140/150 vs 110/175 vs 118/132/145 vs 126 spread).
4. Judge layer (cheap LLM conformance + 4-dimension craft score). This is the one spending step - budget ~pennies/render; I will ask credit status before any batch of it per the standing rule.
5. ConformanceVerdict logging + structure scoreboard + approval gate.
6. Accountable retry/hold loop wired to the self-heal ledger.
7. Backfill blueprints for the other 27 structures as each is approved (not all at once).

6. Owner split + what needs Jeff

• DATA/engine (this chat): the blueprint schema, deterministic + judge conformance layers, the verdict logging + scoreboard, the test harness.
• Cowork: author each structure's blueprint from its spec (the beats/craft bar), starting with the Mirror; owns the craft-bar definitions the judge scores against.
• Code/apps: surface the scoreboard + held-render queue in the admin/app; the A/B structure distributor already noted in the App Experience masterplan.
• Needs Jeff: approve this direction; confirm the hard-block floors (what drift is unshippable vs just flagged); confirm the conformance pass-rate bar a structure must clear to be "approved."

Nothing here is built yet - this is the audit + plan for review. On Jeff's go, step 1-3 (all free / no-LLM) can start immediately; step 4 (the judge, the only spend) waits for a credit check.

7. BUILD STATUS (updated 2026-07-11, Jeff green-lit all 3 decisions)

Jeff's decisions: (1) direction green-lit; (2) floors "propose smart", plus his key rule: sound beds that STRETCH tolerate length variance, beds that can't stretch need tight length or the ending goes weird; (3) approval bar 90% and tune.

DONE (commit fd46119, push HELD for the mirror render window; log-only, blocks nothing): - Step 1 - blueprint schema + the Mirror blueprint (Promise, Being Seen, Three Doors, Install, Seal; threaded symbol; open loop; seal-last) + the Origin blueprint (fixed bed, tight length). affirmology-agent/.../agents/conformance.py. - Step 2 - deterministic conformance engine: beat presence + order + seal-last + per-beat length share (via beat markers), bed-aware length. Beat markers emit/parse/strip with a no-leak assertion. Wired into council.compose in the existing Apollo gate loop, guarded so QC can never break a render. - Step 3 - single source of truth for length: pacing.word_band(minutes, pace, bed_stretchable). Stretchable = generous band + soft flag; fixed bed = tight band + HARD block (Jeff's rule). - Floors implemented: HARD = missing/out-of-order required beat, seal not last, length out of band only for fixed (non-stretchable) beds. Soft (flag only) = length for stretchable beds, per-beat imbalance. - 16 tests pass (tests/test_conformance.py + standalone).

Modes (env): AFFIRMOLOGY_CONFORMANCE = off | log (default: compute + log, block nothing) | enforce. AFFIRMOLOGY_CONFORMANCE_MARKERS = 0 (default; no markers emitted, so live generation output is unchanged and no marker can leak) | 1.

NEXT (needs Jeff / a later increment): - Jeff signs off the Mirror blueprint (the 5 beats + shares) -> then enable markers on the Mirror and flip that structure to enforce. Cowork authors blueprints for the other structures at approval. - Step 4 - the LLM craft-judge (generic / wishy-washy / hope-dominant + beat-quality), the only spend; waits for a credit check per the standing rule. - Steps 5-6 - ConformanceVerdict logging + the per-structure scoreboard + the 90% approval gate + the accountable hold/retry loop wired to the self-heal ledger.