ArchLang

13. A versioned, fully synthetic, self-verifying repair-trajectory dataset ​

  • Status: Accepted
  • Date: 2026-07 (v1.15 / Tranche 5)
  • Relates to: ADR 0009 (distribution, agent-first), ADR 0011 and ADR 0006 (the fix/repair boundary)

Context ​

The round-2 research (docs/research/2026-07-roadmap-proposal.md, deep-dive H4) asked whether ArchLang should invest in fine-tuning or RLVR to lift what was then believed to be a single-digit one-shot intent number (later shown to be mostly a judge-v1 measurement artifact — the calibrated judge-v2 rate is 54%; see AGENTS.md's honest-eval read). The verdict was no training now: a deployment mismatch (the shipped value is a zero-dependency compiler and its deterministic tools, not a hosted model) plus real cost, against evidence that is subject-mismatched (SFT corpora lift validity; RLVR targets intent/geometry/topology — neither speaks for the other). But the same review found one data asset that is aligned with the project's actual claim — drivability, the property that an ArchLang plan can be driven to physical soundness by deterministic tools: a corpus of repair trajectories. Two independent tracks (RustAssistant, the APR survey) converged on it.

So the decision is not "train a model." It is "publish the one asset a future training effort would need, cheaply and honestly, so a reversal is a matter of running a script — not a research program." For an adoption-driven open-source language, any model that learns ArchLang is a win; publishing the generator recruits community training rather than gatekeeping it. The dataset is the option premium on a future managed-fine-tuning reversal, priced near zero.

The load-bearing risk is contamination. The project measures authorability against a private 26-brief eval holdout. If a public dataset overlaps that holdout, every future number reported against it is worthless. The dataset must be provably disjoint, and that must be machine-enforced forever — not a promise in a README.

Decision ​

Publish a versioned, fully synthetic, self-verifying Hugging Face dataset (ChanMeng666/archlang-repair-trajectories) under CC0-1.0, generated by an in-repo generator (dataset/, npm run dataset:gen) that is repo tooling only — no core change, no new runtime dependency, archlang_version pinned. Two splits: repair (broken source + catalogued diagnostics → deterministically healed source + diff + fix_kind + per-stage steps) and authoring (NL brief + golden + describe() facts + intent contract).

a. The contamination iron law, machine-enforced. The eval's 26 briefs and goldens are a frozen private holdout, never published. The public corpus is generated independently (procedural templates phrased unlike the holdout, geometry/labels its own), then double-deduplicated against the holdout: text (token-set Jaccard ≥ 0.5 or any shared 8-word n-gram → reject) and structure (a describe() fingerprint — normalized room-label multiset with areas rounded to 0.5 m², interior-door adjacency edge set, room count → reject exact or near match). Every row embeds a canary GUID twice (a field and a first-line source comment) for downstream leakage probing. A rejected candidate is counted in report.json, never silently emitted. test/dataset.test.ts enforces leakage-freedom, canary presence, determinism, and self-verification permanently, in CI.

b. fix_kind preserves the ADR 0011 / 0006 boundary in the data. Each repair row is tagged "fix" (a syntactic span edit, the rustc/rustfix model — ADR 0011), "repair" (a geometric solve no span edit could express — ADR 0006), or "both" (the two composed). The hard line between "known text edit" and "searched geometry" is not just a code invariant; it is a labelled dimension of the corpus, so a consumer learns the distinction the tools enforce.

c. Zero coupling — the generator imports only the pure core. dataset/ imports ../src/index.js (compile, applyFixes, repair, lint, describe, diagnosticToJson, validateIntent) and nothing from eval/. The eval judge and JUDGE_VERSION are untouched; the dedup layer reads the holdout files only to reject look-alikes. The trajectory recorder mirrors cmdFix / l1Pipeline byte-for-byte, so a recorded fixed_source is exactly what the shipped tools produce.

d. Prose discipline. No dataset artifact — generator, card, or report — claims a diagnostic-feedback-loop gain or its absence: the T3 live experiment was permanently declined (owner decision 2026-07-12), so the loop-vs-equal-budget-resampling question is permanently unanswered. Baseline citations are judge-v2 only (2026-07-12 recalibration); rates are never compared across judge versions. The card frames the asset as drivability packaging plus reward-harness documentation, explicitly not a fix for one-shot intent.

e. Rows are self-verifying; generation is seed-deterministic. Every row is constructed and checked by the compiler at generation time (broken raises its fault; healed is strict-clean and idempotent; authoring intent validates). Generation uses a seeded PRNG only — no clock, no entropy, no Date.now/Math.random/new Date( anywhere under dataset/ (grep-asserted) — so the corpus is reproducible byte-for-byte from the seed.

Consequences ​

Pros.

  • A reproducible, self-verifying corpus that any downstream training effort can trust and re-check — cheap reversal ammunition for the H4 "not now" decision, priced as an option premium.
  • Publishing the generator and card recruits community training instead of gatekeeping it — adoption over moat, the win condition for an open-source language.
  • The holdout's integrity is machine-enforced, not merely promised: contamination becomes a CI failure, not a silent corruption of every future eval number.
  • The fix/repair boundary (ADR 0011/0006) is carried into the data via fix_kind, so the corpus teaches the same distinction the tools enforce.

Cons.

  • A narrow procedural distribution by design — three template families, six fault classes. It is not a benchmark and not a survey of real floor plans.
  • Card maintenance is a release-checklist item: a core release that changes generated output means a new archlang_version, a regenerate at the pinned seed, and a re-upload (see CONTRIBUTING.md → "Releasing").
  • The canary and holdout rules are permanent obligations: the canary is never regenerated, and the holdout is never published — a future contributor must know both.

Full operational detail lives in dataset/README.md (regeneration, upload, invariants) and dataset/CARD.md (the HF consumer view); the holdout's side of the law is in eval/README.md's contamination warning; the motivation is docs/research/2026-07-roadmap-proposal.md → Tranche 5 (deep-dive H4).