render-read-fix — verify a generator by reading its real output

render-read-fix — verify a generator by reading its real output

I recently took a practical approach from the overnight build system to verify generator outputs in a more reliable way. This approach is encapsulated in the render-read-fix skill, which focuses on reading the real artifact instead of relying solely on tests or a SUCCEEDED status. This method is particularly effective when the output looks wrong even though all tests have passed.

The Non-negotiable Principle

The essence of render-read-fix is straightforward: read the finished artifact, not the test, not the status. During the illustrated hardening process, every significant defect passed unit tests but was immediately obvious when a human or vision model viewed the rendered page. For example, seeing "6 PAGES TO COLOR" erroneously in a cookbook or recipe text rendered as one unreadable blob. These defects were not caught until someone actually read the artifact.

The Loop

1. Trace the Code That ACTUALLY Ships

It's crucial to understand that push ≠ deploy. Operator and customer flows invoke deployed Lambdas, like book_factory.py which uses boto3 to invoke staging-teneo-*. Local edits and a green PR mean nothing until the code is deployed. Also, beware of forked render copies. Both the Lambda/picture-book-layout/picture_book_layout.py and Lambda/picture-book-finalizer/picture_book_finalizer.py contain the illustrated layout, but only the finalizer actually ships. Confirm which function the live Step Function or finalize path executes before making any edits.

2. Get a REAL Artifact

When possible, use a real finalized book's intermediate document, such as s3://teneo-generated-books/{book_id}/layout/layout_document.json, which includes content and per-page image URLs. For layout-only checks, realistic synthetic content shaped like the generator's real units is acceptable, using the harness --layout-doc.

3. Render Locally and Read It

Run scripts/render_proof.py, which imports the project's real render function, downloads referenced images, and renders to PDF via Playwright, then rasterizes to PNG using PyMuPDF. This allows you to read the page images without a 40-minute deploy per check. The teneo-production working example is scripts/illustrated/local_render_proof.py. Adapt this for fiction/nonfiction finalize. Crucially, open the PNGs and look at them page by page for the families or chapters you changed.

4. Check Per-Type Invariants

After rendering, run invariant checks (--preset/family) and read the report. Ensure there are no wrong-format language leaks, required components are present and non-empty, structure matches the type, and quantities/numbers are preserved. Refer to references/per-type-invariants.md for detailed checks.

5. Fix, Re-render, Re-read

Iterate in the local loop. Only after it reads clean do you push, deploy, and run the live confirmation. This step saves significant time compared to deploying for each check.

6. For a Recurring Defect Class

The final step is dealing with recurring defect classes. This involves prevention, detection, and auto-healing. For instance, a mockup defect needed an anti-mockup prompt (prevent), pixel QA to detect it deterministically (detect), and an auto-heal process.

In my experience, using the render-read-fix method has been invaluable in catching defects that would otherwise slip through automated tests. It ensures the highest quality of the final product by literally seeing the output as it will appear to end-users.