Oregami
Repositories/oxedyne/daimond

oxedyne/daimond/dev/staleguard.mjs

20.8 KiB, 1 run

created by r2519314175:213, which is this file's identity for as long as the history lasts, whatever it is later renamed to

download · who wrote it · its history

1// staleguard.mjs — the one answer to "is the artefact under test this tree's?".
2//
3// ── The defect this closes ──────────────────────────────────────────
4//
5// A verifier that measures an artefact somebody's earlier build left behind
6// reports on code that no longer exists. Both directions are wrong and only one
7// of them is noisy: a false RED wastes a morning, and a false GREEN says a
8// boundary holds when nothing has looked at it. Everything here fences a model
9// off from a user's filesystem, so the silent one is the one that matters.
10//
11// It has happened twice in one day, both times because the artefact was reached
12// by a hardcoded path and taken on trust:
13//
14// * `exec::tests::the_shipping_launcher_fences_a_real_command` execs
15// `hand/target/debug/daimond-hand`, which `cargo test` never builds. With
16// that binary MOVED OFF DISK the suite still reported `193 passed`.
17// * `dev/verify_ptyedge.mjs`, the same shape against `hand/target/release`.
18// * `dev/verify_kitfence.mjs` records the live version of it: with a
19// `CARGO_TARGET_DIR` inherited from the shell, cargo wrote the new binary
20// somewhere else, the test ran the old one, and a security test passed
21// against a binary from before the fix.
22//
23// ── The oracles ─────────────────────────────────────────────────────
24//
25// A cargo binary carries its own: the dep-info file `<bin>.d` written beside it
26// names every source that went into the link — this crate's and every fe2o3
27// crate's — so nothing is hardcoded and a change anywhere upstream counts as
28// staleness. [`whyStaleBinary`] reads it.
29//
30// A wasm bundle does NOT. `www/pkg` is written by wasm-bindgen out of whichever
31// cargo target directory the builder happened to have set, and this tree has
32// been built into a dozen of them, so the dep-info that describes the shipped
33// bundle cannot be found from the bundle. [`whyStaleWasm`] therefore uses the
34// coarser oracle `dev/verify_ptyedge.mjs` settled on: EVERY `.rs` under `src/`,
35// which is a superset of what a hand-picked list would catch and needs no
36// maintenance when a file is added. Its blind spot is a change inside fe2o3,
37// which is named here rather than left to be discovered.
38//
39// Neither returns a boolean. A guard that says only "stale" leaves the reader to
40// work out which file and what to run, so each returns the SENTENCE, or '' when
41// there is nothing wrong. [`refuse`] is how a caller acts on one.
42//
43// ── A CLOCK IS A PROXY; CONTENT IS THE PROPERTY ─────────────────────
44//
45// The property wanted of a bundle is "it was built from THIS source". An mtime
46// answers a different question — "was it written after the source was?" — and
47// the two part company the moment a bundle legitimately moves between trees.
48// `dev/gate.sh` runs the suite in a `git worktree`, and rather than spend a wasm
49// build per bisect step it BORROWS the main tree's bundle when the commit under
50// test has byte-identical Rust. A fresh checkout stamps every `.rs` with the
51// moment it was written, so the borrowed bundle is necessarily older than the
52// source it was in fact built from, and from 488f2d5 (2026-08-11) every
53// wasm-guarded verifier refused under the gate — three of them for the whole of
54// the first full run. Nothing was wrong with either half; the COMPARISON was.
55//
56// So a bundle may carry a record of the source it was built from: [`certifyWasm`]
57// writes `www/pkg/source.json` — a SHA-256 per engine source file, plus one over
58// the wasm itself — and [`whyStaleWasm`] prefers it to the clock. The record is
59// not taken on trust: the guard rehashes the tree in front of it and compares,
60// so a certificate can only vouch for source that is still byte-for-byte what it
61// names, and an edit made after it was written is caught exactly as a rebuild
62// would be. It names the FILE that differs, which an mtime never could.
63//
64// Two properties keep the record from becoming a second thing to go stale:
65//
66// * It records the bundle's own hash, so a bundle rebuilt by any means at all
67// no longer matches the record beside it. A record about some other bundle is
68// IGNORED, and the clock takes over again — a stale certificate can never
69// launder a stale bundle, and can never false-refuse a fresh one.
70// * It is optional. No record, no cost: an ordinary tree has none, the mtime
71// oracle runs exactly as before, and the developer who edited `web.rs` and
72// forgot to rebuild is refused as they always were.
73//
74// `dev/build-wasm.sh` writes one too, since 2026-08-13, so the strong oracle
75// covers the main tree as well as the gate's worktrees. That is HALF a change,
76// and the other half is not optional: `pkg/source.json` is in `EXCLUDE` in
77// `verify/lib.mjs`, beside `pkg/package.json`, so a provenance note never enters
78// a sealed manifest. A record of when and where a bundle was built moves on
79// every build; sealed, it would put a figure nobody else can reproduce inside
80// the one artefact whose whole purpose is that a stranger can reproduce it, and
81// `dev/repro-check.sh` is what catches that.
82//
83// What it does NOT do is make a rebuild rarer than it should be. The record is
84// SHA-256 per file, and a comment added to `src/web.rs` moves that file's hash
85// exactly as a changed fence does. Nothing short of compiling can tell the two
86// apart, so a comment still costs a build; what stops costing a build is source
87// that did not change at all and only looks like it did.
88import fs from 'node:fs';
89import os from 'node:os';
90import path from 'node:path';
91import { fileURLToPath } from 'node:url';
92import { sha256, posix, bundleHash } from '../verify/lib.mjs';
93
94/// The prerequisites named on one line of a cargo dep-info file.
95///
96/// Make's escaping, which is what the format is: a backslash makes the character
97/// after it ordinary, so a path containing a space survives being split on
98/// whitespace. The same reading as `dep_sources` in `hand/src/exec.rs`.
99export function depSources(list) {
100 const out = [];
101 let cur = '', esc = false;
102 for (const c of list) {
103 if (esc) { cur += c; esc = false; }
104 else if (c === '\\') { esc = true; }
105 else if (/\s/.test(c)) { if (cur) { out.push(cur); cur = ''; } }
106 else { cur += c; }
107 }
108 if (cur) out.push(cur);
109 return out;
110}
111
112/// Why the binary at `bin` is not this tree's, or '' when there is no reason.
113///
114/// # Arguments
115/// * `bin` - The artefact under test.
116/// * `subject` - What cannot be verified, as the head of a sentence.
117/// * `rebuild` - The command that makes the refusal go away.
118/// * `what` - What the artefact is, for the sentence. Defaults to `binary`.
119///
120/// # Returns
121/// The refusal, or '' when the binary is newer than every source cargo says
122/// went into it.
123export function whyStaleBinary(bin, { subject, rebuild, what = 'binary' }) {
124 let built;
125 try { built = fs.statSync(bin).mtimeMs; }
126 catch (e) {
127 return `${subject} cannot be verified, because there is no ${what} at ${bin} `
128 + `to verify it against. Run \`${rebuild}\` and try again.`;
129 }
130 const record = `${bin}.d`;
131 let listed;
132 try { listed = fs.readFileSync(record, 'utf8'); }
133 catch (e) {
134 return `${subject} cannot be verified, because ${bin} has no dep-info file at `
135 + `${record}, so there is no record of what went into it and its vintage `
136 + `cannot be established. Run \`${rebuild}\` and try again.`;
137 }
138 let described = false;
139 let newest = null;
140 for (const line of listed.split('\n')) {
141 const at = line.indexOf(':');
142 if (at < 0) continue;
143 if (path.resolve(line.slice(0, at).trim()) !== path.resolve(bin)) continue;
144 described = true;
145 for (const src of depSources(line.slice(at + 1))) {
146 let t;
147 try { t = fs.statSync(src).mtimeMs; }
148 catch (e) {
149 return `${subject} cannot be verified, because ${bin} was built from `
150 + `${src}, which can no longer be read, so what is inside the ${what} `
151 + `cannot be established. Run \`${rebuild}\` and try again.`;
152 }
153 if (!newest || t > newest.t) newest = { src, t };
154 }
155 }
156 if (!described) {
157 return `${subject} cannot be verified, because ${record} says nothing about `
158 + `${bin}, so that ${what} is not the one this build produced. Run `
159 + `\`${rebuild}\` and try again.`;
160 }
161 if (newest && newest.t > built) {
162 const by = Math.round((newest.t - built) / 1000);
163 return `${subject} would have been verified against a stale ${what}, which proves `
164 + `nothing about it in either direction: ${newest.src} was last changed ${by} `
165 + `second(s) after ${bin} was linked, so that ${what} is not this source. Run `
166 + `\`${rebuild}\` and try again.`;
167 }
168 return '';
169}
170
171/// Every Rust source under `dir`, which is everything the bundle is built from.
172export function rustSources(dir) {
173 const out = [];
174 for (const ent of fs.readdirSync(dir, { withFileTypes: true })) {
175 const f = path.join(dir, ent.name);
176 if (ent.isDirectory()) out.push(...rustSources(f));
177 else if (ent.name.endsWith('.rs')) out.push(f);
178 }
179 return out;
180}
181
182/// The command that builds the bundle. Named in every wasm refusal, because a
183/// verifier that stops without saying what to run has only moved the problem.
184export const WASM_REBUILD = 'dev/build-wasm.sh';
185
186/// Where a tree keeps the three things this file needs, relative to its root.
187export const SRC_REL = 'src';
188export const PKG_REL = 'www/pkg';
189export const WASM_REL = 'www/pkg/oxedyne_daimond_bg.wasm';
190
191/// The name of the record a bundle carries about the source it was built from.
192/// It sits in the bundle's own directory so that copying the bundle copies its
193/// provenance with it.
194export const WASM_SOURCE_RECORD = 'source.json';
195
196/// The `{ relpath: sha256 }` map of everything that goes into the bundle.
197///
198/// The same set the mtime oracle walks — every `.rs` under `src/` — plus
199/// `Cargo.toml` and `Cargo.lock`, which decide what is compiled in and which
200/// `dev/gate.sh` already compares before it borrows. Paths are relative to
201/// `root` and POSIX, so two trees in different directories compare equal.
202export function sourceMap(root, srcDir = path.join(root, SRC_REL)) {
203 const map = {};
204 for (const f of rustSources(srcDir)) {
205 try { map[posix(path.relative(root, f))] = sha256(fs.readFileSync(f)); }
206 catch (e) { /* removed under us; the next build will say so */ }
207 }
208 for (const extra of ['Cargo.toml', 'Cargo.lock']) {
209 try { map[extra] = sha256(fs.readFileSync(path.join(root, extra))); }
210 catch (e) { /* a tree without one is described by not naming it */ }
211 }
212 return map;
213}
214
215/// One figure for a whole engine source tree, by the same algorithm
216/// `verify/lib.mjs` fingerprints a bundle with. Two trees agree on it if and
217/// only if every file this bundle is built from is byte-identical.
218export function sourceHash(root, srcDir) {
219 return bundleHash(sourceMap(root, srcDir));
220}
221
222/// A path with this machine's home directory written as `~`.
223///
224/// The record is written into `www/pkg`, and `www/` is what gets rsync'd to the
225/// server, so every field in it is served to the public. Nothing COMPARES this
226/// one -- `files` and `bundle` are what the guard rehashes, and `from` only ever
227/// reaches a sentence a developer reads -- so the tree can be named without
228/// naming whoever owns it. `~/usr/.../daimond` and `~/usr/.../daimond-oss` still
229/// tell the two trees apart, which is the only thing the sentence needs.
230function homeless(p) {
231 const home = os.homedir();
232 return home && p.startsWith(home + path.sep) ? '~' + p.slice(home.length) : p;
233}
234
235/// Record, beside the bundle in `pkgDir`, the source it was built from.
236///
237/// `from` is the root of the tree that BUILT it, which is not necessarily the
238/// tree the bundle now sits in — that is the whole point when `dev/gate.sh`
239/// borrows one. The record also carries the bundle's own hash, so it is silently
240/// disregarded the moment that bundle is replaced.
241///
242/// # Arguments
243/// * `pkgDir` - The bundle's directory, `www/pkg`.
244/// * `from` - Root of the tree whose source went into it.
245/// * `by` - What wrote the record, for the sentence a later refusal prints.
246///
247/// # Returns
248/// The source hash written down.
249export function certifyWasm(pkgDir, from, { by = 'dev/gate.sh' } = {}) {
250 const files = sourceMap(from);
251 const wasm = path.join(pkgDir, path.basename(WASM_REL));
252 const record = {
253 version: 1,
254 source: bundleHash(files),
255 bundle: sha256(fs.readFileSync(wasm)),
256 files,
257 from: homeless(path.resolve(from)),
258 by,
259 when: new Date().toISOString(),
260 };
261 fs.writeFileSync(path.join(pkgDir, WASM_SOURCE_RECORD), JSON.stringify(record, null, '\t') + '\n');
262 return record.source;
263}
264
265/// The record beside `wasm` if there is one that is about `wasm`, else null.
266///
267/// A record naming a different bundle hash is not this bundle's, so it is
268/// dropped rather than believed: whatever rebuilt the bundle — `wasm-pack` by
269/// hand, the mirror, a copy from somewhere else — left the note behind, and a
270/// note about a bundle that no longer exists must not be able to speak for the
271/// one that does.
272export function wasmRecord(wasm) {
273 let rec;
274 try { rec = JSON.parse(fs.readFileSync(path.join(path.dirname(wasm), WASM_SOURCE_RECORD), 'utf8')); }
275 catch (e) { return null; }
276 if (!rec || rec.version !== 1 || !rec.files || typeof rec.bundle !== 'string') return null;
277 try { if (sha256(fs.readFileSync(wasm)) !== rec.bundle) return null; }
278 catch (e) { return null; }
279 return rec;
280}
281
282/// The first way this tree's source differs from what `rec` names, or '' when
283/// it does not differ at all. Sorted, so the same tree always names the same
284/// file rather than whichever the filesystem happened to hand back first.
285function firstDifference(rec, root, srcDir) {
286 const now = sourceMap(root, srcDir);
287 for (const rel of Object.keys(rec.files).sort()) {
288 if (!(rel in now)) return `${rel} is gone from this tree, though it is in the copy`;
289 if (now[rel] !== rec.files[rel]) return `${rel} differs from the copy`;
290 }
291 for (const rel of Object.keys(now).sort()) {
292 if (!(rel in rec.files)) return `${rel} is in this tree and not in the copy`;
293 }
294 return '';
295}
296
297/// Why the wasm bundle at `wasm` is not `srcDir`'s, or '' when there is no
298/// reason.
299///
300/// A hand-picked list of sources is what three of these guards had, and each
301/// missed files that changed the same day: `src/wasm/opfs.rs`,
302/// `src/wasm/diamond.rs`, `src/prompts.rs`, `src/skills.rs`. So the list is not
303/// picked — it is every `.rs` there is.
304///
305/// Two oracles, in order of strength. A bundle carrying a source record is
306/// judged on CONTENT: every file rehashed against what the record names. One
307/// without is judged on the clock, as before, which is all there is to go on
308/// and enough for the case it was written for — a source edited after the last
309/// build, in the tree that build happened in.
310///
311/// # Arguments
312/// * `wasm` - The bundle under test, `www/pkg/oxedyne_daimond_bg.wasm`.
313/// * `srcDir` - The engine's source tree, `src/`.
314/// * `subject` - What cannot be verified, as the head of a sentence.
315/// * `holds` - What lives in the bundle, for the sentence. Optional.
316/// * `quiet` - Suppress the line saying a borrowed bundle was accepted.
317///
318/// # Returns
319/// The refusal, or '' when the bundle is this source's.
320export function whyStaleWasm(wasm, srcDir, { subject, holds = '', quiet = false }) {
321 let built;
322 try { built = fs.statSync(wasm).mtimeMs; }
323 catch (e) {
324 return `${subject} cannot be verified, because there is no wasm bundle at `
325 + `${wasm}${holds ? ` and ${holds} lives in it` : ''}. Run `
326 + `\`${WASM_REBUILD}\` and try again.`;
327 }
328 const root = path.dirname(srcDir);
329 const rec = wasmRecord(wasm);
330 if (rec) {
331 const diff = firstDifference(rec, root, srcDir);
332 if (diff) {
333 return `${subject} would have been verified against a stale engine, which proves `
334 + `nothing about it in either direction: ${diff} ${wasm} was built from `
335 + `(${rec.by} recorded ${rec.from} at ${rec.when}), so that bundle is not this `
336 + `source. Run \`${WASM_REBUILD}\` and try again.`;
337 }
338 // Said out loud, in every log that carries a wasm-guarded verifier: a
339 // bundle that was not built here is exactly the thing a reader wants to
340 // know about before they read anything else in the file.
341 if (!quiet) {
342 console.log(`engine: ${wasm} was built from ${rec.from}, not from this tree, and every `
343 + `file it was built from still hashes the same here (${rec.by}, ${rec.when}).`);
344 }
345 return '';
346 }
347 let newest = null;
348 for (const f of rustSources(srcDir)) {
349 let t;
350 try { t = fs.statSync(f).mtimeMs; }
351 catch (e) { continue; } // removed under us; the next build will say so
352 if (!newest || t > newest.t) newest = { f, t };
353 }
354 if (newest && newest.t > built) {
355 return `${subject} would have been verified against a stale engine, which proves `
356 + `nothing about it in either direction: ${newest.f} was last changed `
357 + `${Math.round((newest.t - built) / 1000)} second(s) after ${wasm} was built, `
358 + `so that bundle is not this source. Run \`${WASM_REBUILD}\` and try again.`;
359 }
360 return '';
361}
362
363/// Whether `wt` can use `main`'s bundle rather than spend a build on its own.
364///
365/// Two questions, and both must be answered before a bundle is copied anywhere.
366/// Is the bundle in `main` this source's at all — because borrowing a stale
367/// bundle would carry the staleness into the gate under a certificate saying it
368/// is fine. And is `wt`'s engine source byte-for-byte `main`'s. Content both
369/// times: what git says about two commits is a good pre-filter and not the
370/// property, since the tree the bundle was actually built in is a WORKING tree
371/// and may hold things no commit does.
372///
373/// # Returns
374/// `''` when the bundle may be borrowed, else the reason it may not, phrased to
375/// be printed after "building this commit's bundle (".
376export function whyNotBorrow(main, wt) {
377 const wasm = path.join(main, WASM_REL);
378 if (!fs.existsSync(path.join(main, PKG_REL, 'oxedyne_daimond.js'))) {
379 return 'the main tree has no bundle to lend';
380 }
381 const stale = whyStaleWasm(wasm, path.join(main, SRC_REL), {
382 subject: 'The bundle about to be lent', quiet: true,
383 });
384 if (stale) return 'the main tree\'s own bundle is not the main tree\'s source';
385 const here = sourceMap(wt);
386 const there = sourceMap(main);
387 for (const rel of new Set([...Object.keys(here), ...Object.keys(there)].sort())) {
388 if (here[rel] !== there[rel]) return `${rel} differs from the main tree`;
389 }
390 return '';
391}
392
393/// Print a refusal and stop, or return where there is nothing to refuse.
394///
395/// Exit 2, never 1: a suite reads 1 as "the code under test is wrong" and this
396/// is "nothing was measured". `dev/run_all.sh` reports it as a failure either
397/// way, which is right — a verifier that did not run is not a pass.
398export function refuse(...reasons) {
399 for (const why of reasons) {
400 if (!why) continue;
401 console.error(why);
402 process.exit(2);
403 }
404}
405
406// ── The same oracle, for a caller that is not JavaScript ────────────
407//
408// `dev/gate.sh` decides whether to borrow a bundle and then has to live with a
409// verifier's opinion of what it decided. Those were two separate pieces of
410// reasoning, one in bash about git and one here about clocks, and they
411// disagreed. There is one now, and bash asks it:
412//
413// node dev/staleguard.mjs hash <root> # one figure for its engine source
414// node dev/staleguard.mjs borrow <main> <wt> # '' to borrow, else why not
415// node dev/staleguard.mjs certify <pkgdir> <builtFrom> [by]
416// node dev/staleguard.mjs why-stale <root> # the refusal a verifier would print
417//
418// `by` is what a later refusal names as the author of the record, and it is
419// worth passing: two things write one now, and "dev/gate.sh recorded this" on a
420// bundle `dev/build-wasm.sh` built sends a reader to the wrong file.
421//
422// Each prints one line or nothing, and exits 0 whatever the answer: the answer
423// is the output, and an exit code would only give a caller a second thing to
424// read. `why-stale` is the exception — 2 when it refuses, so `gate.sh` can stop
425// before a two-hour suite that every wasm-guarded verifier would refuse.
426if (process.argv[1] && path.resolve(process.argv[1]) === path.resolve(fileURLToPath(import.meta.url))) {
427 const [cmd, a, b, c] = process.argv.slice(2);
428 switch (cmd) {
429 case 'hash':
430 console.log(sourceHash(path.resolve(a)));
431 break;
432 case 'borrow':
433 console.log(whyNotBorrow(path.resolve(a), path.resolve(b)));
434 break;
435 case 'certify':
436 console.log(certifyWasm(path.resolve(a), path.resolve(b), c ? { by: c } : {}));
437 break;
438 case 'why-stale': {
439 const root = path.resolve(a);
440 const why = whyStaleWasm(path.join(root, WASM_REL), path.join(root, SRC_REL), {
441 subject: 'Every verifier in this run that loads the app',
442 holds: 'the engine they all measure',
443 quiet: true,
444 });
445 if (why) { console.log(why); process.exit(2); }
446 break;
447 }
448 default:
449 console.error(`staleguard: no such command "${cmd}". `
450 + `Try hash, borrow, certify or why-stale.`);
451 process.exit(2);
452 }
453}