Trial 0020 — long-run degradation: twenty changes on one codebase (ADR 0042)

Question: every earlier trial measured one build plus one change. When fresh agent sessions keep changing the same notes app twenty times, without memory and without any repair, does the Hozu app degrade more slowly than the Nuxt app? The criterion was stated before the runs (ADR 0042 F): over steps 1–20, Hozu has fewer regression failures and a lower slope of cost per step against app size.

Setup

• Task: the notes build spec (step 0), then bench/trial/longrun/changes/01.md … 20.md, from pin + search to removing tags. Each change is behaviour and DOM contracts; later changes touch what earlier ones added (the rename in 04 is observed by the export in 19, the route move in 09 by every later path, tags are removed in 20).
• Acceptance: bench/trial/longrun/accept.mjs, cumulative: after step k it runs every check introduced by steps 0..k (15 build checks, 64 at step 19, 61 at step 20 after the tag checks retire), split into new and regression. Each check signs in its own fresh user.
• Validated before the agents ran, on two reference apps written from the change files (Hozu from examples/notes, Nuxt from the nuxt-0006 scaffold), both replayed from base + patches by verify-reference.sh: every step passes 100 % on both. One check was wrong and fixed before any run (A1 / SH1 counted <input type=hidden> as an input that acts on a note). Four deliberate breaks (duplicate text, rate limit 11, a non-HttpOnly cookie) each turned the expected checks red and left the control checks green.
• Runs: Hozu 0.7.0 (packed tarballs, create-hozu --agent claude) and Nuxt 4.5.2, claude-opus-5-5, one claude -p session per step with the prompts in bench/trial/longrun/prompts/, two runs per framework, Hozu and Nuxt in parallel. No session-limit voids, no 1800 s timeouts, no browser crashes in the trial runs.
• Differences from trials 0016–0019:
  • Node 22.22.1 (22.22.2 is not installed here); pnpm pinned to 10.33.0;
  • --strict-mcp-config, so no local MCP servers are loaded;
  • the trial 0016 prompt text was not kept, so the prompts were rewritten from its description;
  • the organisation's managed instructions still reach claude -p (they are not a setting source): every agent, in both frameworks, ended with a Traditional Chinese summary and three next steps. The output cost is inflated equally on both sides, but absolute numbers are not comparable with earlier trials.
• Data: bench/trial/longrun/results/<fw>/<run>/ (acceptance JSON, hozu check and inspect JSON, metrics.jsonl, run.log). The transcripts (NN.jsonl, 22 MB) stay next to them, untracked. The apps, with a tag per step, are in ~/hozu-trial-0020/<fw>-<run>/app.

Results

Correctness (steps 1–20):

Cost (weighted tokens):

Per step (mean of the two runs):

Codebase (step 20; slopes over steps 1–20):

JS is the uncompressed size of the scripts loaded. /login and the list page load the same Hozu bundle.

Where the calls go (mean per change, anatomy.mjs):

Reading the result

The thesis does not hold on this task. Both pre-stated criteria go to Nuxt:

• Nuxt made twenty changes twice without a single regression or failed new check.
• Hozu's cost per change doubled from the first half to the second, while Nuxt's rose by 15–27 %.

The Hozu regressions come from two mechanisms, both outside the IR, both found in the reference app first:

• Both are silent. In both runs the step-17 agent wrote that the admin table no longer lists the deleted user. It had checked with curl, and without JS the bug does not occur. hozu check stayed clean at every step: neither failure is in anything the validator sees (an endpoint's HTML, resolver state, a client/server race).
• The cost spikes are the framework gaps of the reference app (bench/trial/longrun/reference/hozu-defects.md):
  • step 13, bulk actions, 851 k and 1 668 k (47 edits in one run): D3, forms cannot send several checked values or tell two submit buttons apart, so both agents built a native form and an endpoint;
  • step 18, German, 591 k / 407 k: D8, i18n prefixes every locale, English included; neither run used site.locales, both built their own language switch;
  • step 6, tags, 407 k / 235 k: D1, a search schema on a route changes every navigate link, and HZ018 fires with identical was / now (both runs edited the account machine's navigation in this step, and HZ018 appears 4 times in each transcript);
  • step 10, undo, 285 k / 305 k: 13 and 11 verify calls; the cause was not analysed further.
• The fixed cost of the guide stays: Hozu agents read 3.5 docs topics per change and verify 3.7× as often (hozu get / post / browse), and that context is carried through every later call.

What Hozu does better, and keeps better, as the codebase grows:

• Duplication stays low and flat (1–2.5 % at the end, against 6.5–8.2 %). In both Nuxt runs the jump comes with step 5 (edit) and step 7 (archive); which blocks repeat was not analysed.
• Client JS stays an order of magnitude smaller and grows 10–20× more slowly (+1.5 to +3 KB over twenty changes, against +30 KB).
• The structure stays checkable: 70–71 lock entries, 16 states and 70–71 transitions at the end, no unreachable state, every deciding transition covered by a contract, hozu check clean at every step.
• None of this turned into fewer regressions or a cheaper change: the app is larger (1.2× Nuxt's lines), and the cost per change rises with it.

What the data does not show

• Two runs per framework, one task, one model. The per-step spread between runs is large (Hozu step 13: 851 k and 1 668 k), so single steps are not significant. The direction of every aggregate is the same in both runs.
• The change set is ours. It was written without looking at either framework, but it contains exactly the kinds of change that meet Hozu's current gaps (403, multi-value forms, cross-user data, a second language). A change set that stays inside Hozu's supported surface could show a different result, and would also be a weaker test.
• The agents' prior knowledge is not equal: Nuxt is in the model's training data, and Hozu is learned from the guide each session. That is part of what is measured, not a flaw of the measurement.
• "Degradation" here is behaviour and cost. Readability, reviewability and the effort a human needs to verify a change are not measured. The lock, the contracts and the render plan are aimed at those.
• Silent failures were classified by hand for the failing steps. The automatic heuristic cannot read the summaries' intent (they are in Chinese and always list limitations).
• The managed organisation instructions changed every agent's final message. The effect is equal on both sides, but it adds output tokens.

Conclusion

• Over twenty sequential changes, Nuxt stayed correct and flat in cost. Hozu 0.7 stayed structurally clean, with low duplication and 10× less client JS, but its cost per change doubled, and from step 16–17 on both runs carried regressions that the agents reported as working.
• The claim "AI-generated codebases degrade more slowly under Hozu" is not supported by this trial.
• The measured degradation is concentrated where an app has to leave Hozu's surface: 403 pages, multi-value forms, cross-user invalidation, an unprefixed default locale. In those places the agent writes endpoints and hand-made HTML that no validator checks, and they are where both the regressions and the largest costs are.
• Candidates for the next phase, by measured effect:
  • D9 + D4: cross-session and endpoint invalidation, and live refetch racing a session change (the only regression seen in both runs);
  • D3: multi-value form fields and the submitter (step 13, 8.9× Nuxt);
  • D7: a 403 from a page (a regression in run 1, and an endpoint page outside the IR);
  • D8: a second locale without prefixing the default (step 18, 3.1×);
  • D1 / D6: HZ018 printing identical was / now (steps 6, 10, 14).
• Re-measure after fixes: steps 13–20 on the step-12 commits of these apps (about an hour per run), with the same acceptance.