Oregami
Repositories/oxedyne/daimond

oxedyne/daimond/dev/verify_checkreach.mjs

17.3 KiB, 1 run

created by r2519314175:289, which is this file's identity for as long as the history lasts, whatever it is later renamed to

download · who wrote it · its history

1// verify_checkreach -- every runnable file in `dev/` is either in the gate or says why not.
2//
3// WHY THIS EXISTS. `dev/run_all.sh` built its work list as `ls verify_*.mjs`, and a list built
4// from a pattern omits whatever does not match it, silently. That one line cost more than any
5// other fault in this repository:
6//
7// * THE GATE HAD NEVER RUN A SINGLE RUST TEST. Not one, in any release this project has
8// made -- the "269 passed of 277" that seq 150 shipped on was browser verifiers
9// exclusively, and so was every release before it.
10// * `dev/verify_walkbound.sh`, a verifier by name, the only `.sh` among 281 `verify_*`
11// files: never run, confirmed absent from all forty `suite.log` files on disk.
12// * `dev/jscheck.sh`, the ONLY gate on whether the browser JavaScript parses at all -- it
13// exists because `node --check` exits 0 on a `.js` file holding a syntax error -- absent
14// from the same forty logs.
15// * `jscheck.sh`'s own `ls www/js/*.js`, which could not see the eight locale tables, the
16// files most often edited eight at a time. Against a tree with an unescaped apostrophe in
17// `www/i18n/fr.js` it answered `jscheck: 61 file(s) parse.`
18// * `NEEDS_GRANT`, a hand-kept list of two names that could not see `verify_sync`'s
19// identity: 37 failed / 68 passed in every gate, on an account that never held Pro.
20//
21// Six instances in two days, which is what makes it a shape and not six bugs, and Lane T
22// stated the rule in one sentence: **a list built by a glob needs asking what it cannot see,
23// not just what it returns.** This file is that question, made into a check.
24//
25// THE INVERSION IS THE WHOLE FIX. A pattern's default is invisible; this file's default is
26// red. It does not ask "which files look like checks" -- that is a pattern again, one level
27// up, and it would lose the next `verify_walkbound.sh` exactly as `ls verify_*.mjs` lost the
28// last one. It enumerates EVERY file in `dev/`, asks git rather than the filesystem, and
29// requires each one to land in exactly one of two places:
30//
31// 1. REACHABLE from `dev/run_all.sh`, following the files it names and the files those
32// name, or
33// 2. REGISTERED in `dev/CHECKS.md` with a written reason it is not in the gate.
34//
35// A file in neither fails this check by name. A new `assert_thing.mjs` that nobody wired up
36// goes red the day it is written, whatever it is called.
37//
38// AND IT IS CHECKED IN BOTH DIRECTIONS, because a register is a hand-kept list and this
39// repository has learned what those do. A row naming a file that no longer exists fails; a
40// row naming a file that IS in the gate fails, because it is then a false sentence about the
41// gate. The register cannot rot in either direction while this runs.
42//
43// WHAT THIS CANNOT SEE, answered plainly, because an unqualified answer here would be the
44// very fault the file is about:
45//
46// * ONLY `dev/`. A check living anywhere else is outside its reach. The Rust tests are the
47// large case, and they have their own instrument one directory over: `dev/testcount.mjs`
48// asks each harness `--list` for the number of tests compiled into it and refuses to call
49// a run a pass unless that many executed. Between the two, "compiled but never run" is
50// covered for Rust and "written but never run" for `dev/`. Nothing covers `verify/`,
51// `ext/` or `hand/` the same way.
52// * REACHABLE IS NOT EXECUTED. This proves a file is WIRED INTO something the suite runs.
53// It does not prove the suite reaches the code inside it, and it cannot: `verify_conformance`
54// is reachable and skips whenever no forge is up. Whether a check has ever been seen to
55// fail is a different question and `dev/breakcheck.mjs` is the instrument for it.
56// * EDGES ARE STATIC TEXT. A file named through a variable -- `node "dev/$name.mjs"` -- is
57// not followed, except for the one form the suite's own work list uses, which is seeded
58// below. That errs toward calling a file unreachable, which is the safe direction: the
59// cost is a register row, not a hole. It errs the other way only through a transitive
60// edge, where naming a file is taken for running it: `dev/publish.mjs` counts as reachable
61// because `dev/deploy.sh` runs it and `verify_deploy` runs `deploy.sh --check-fresh`,
62// which never gets as far as publishing. The report prints the path it followed for every
63// such file, so the claim can be read rather than trusted.
64// * IGNORED FILES ARE INVISIBLE. `--exclude-standard` means `.gitignore` decides, so
65// `dev/audit_*.mjs` and `dev/*.log` are outside the roster by the same rule that keeps
66// build output out of it. A check hidden under an ignored name would not be seen.
67//
68// node dev/verify_checkreach.mjs # the assertions, and a summary
69// node dev/verify_checkreach.mjs --list # every file, with its bucket and its path in
70import fs from 'node:fs';
71import path from 'node:path';
72import { execFileSync } from 'node:child_process';
73
74const HERE = path.dirname(new URL(import.meta.url).pathname);
75const ROOT = path.resolve(HERE, '..');
76const LIST = process.argv.includes('--list');
77
78// Extensions that hold a program. A file with any of these, or with a `#!`, is runnable.
79const RUNNABLE = new Set(['mjs', 'js', 'sh', 'py']);
80// Extensions that hold data, and are therefore not anybody's check. THIS LIST IS HALF THE
81// POINT: an extension on neither list is a file nobody has classified, and it fails below
82// rather than being skipped, which is what stops the definition of "check file" from being
83// one more pattern with a silent outside.
84const INERT = new Set(['md', 'json', 'jsonl', 'txt', 'bin', 'typ', 'xlsx', 'ttf', 'otf',
85 'svg', 'png', 'jpg', 'webp', 'pdf', 'csv', 'html', 'css', 'jdat', 'toml',
86 'lock', 'yml', 'yaml', 'log', 'zip', 'docx', 'xml', 'wasm',
87 // A diff is data. `git apply` runs it, in the sense that a `.json` is run by
88 // whatever reads it, but nothing in dev/ can be reached THROUGH one, so it is
89 // no more a check than a fixture is.
90 'patch']);
91
92const fails = [];
93const oks = [];
94function ok(msg) { oks.push(msg); console.log(` ok ${msg}`); }
95function bad(msg) { fails.push(msg); console.log(` FAIL ${msg}`); }
96
97// ── The roster: asked of git, and never of a glob ───────────────────────────
98//
99// `--cached` is everything tracked; `--others --exclude-standard` is everything else that is
100// not ignored. A file written this minute and never added is in it. A glob over the working
101// tree would sweep in build output; a glob over `git ls-files` alone would miss the file
102// somebody has just written, which is precisely the file most likely to be an unwired check.
103let roster;
104try {
105 roster = execFileSync('git', ['-C', ROOT, 'ls-files', '--cached', '--others',
106 '--exclude-standard', '--', 'dev'], { encoding: 'utf8' })
107 .split('\n').filter(Boolean);
108} catch (e) {
109 console.log(` FAIL git could not list dev/: ${e.message}`);
110 console.log('\n0 ok, 1 failed — and a fallback glob would be the fault this file exists to catch.');
111 process.exit(1);
112}
113
114/// Does this file hold a program, rather than data? Extension first, then the first two bytes.
115function isRunnable(rel) {
116 const ext = extOf(rel);
117 if (ext && RUNNABLE.has(ext)) return true;
118 try {
119 const fd = fs.openSync(path.join(ROOT, rel), 'r');
120 const b = Buffer.alloc(2);
121 fs.readSync(fd, b, 0, 2, 0);
122 fs.closeSync(fd);
123 return b.toString('latin1') === '#!';
124 } catch { return false; }
125}
126function extOf(rel) {
127 const name = rel.slice(rel.lastIndexOf('/') + 1);
128 const dot = name.lastIndexOf('.');
129 return dot > 0 ? name.slice(dot + 1).toLowerCase() : '';
130}
131
132const runnable = roster.filter(isRunnable);
133const runSet = new Set(runnable);
134const byName = new Map();
135for (const p of runnable) byName.set(p.slice(p.lastIndexOf('/') + 1), p);
136
137// ── The graph ───────────────────────────────────────────────────────────────
138//
139// An edge is one file naming another IN A POSITION WHERE SOMETHING RUNS IT: after `bash`,
140// `sh`, `node` or `source`; inside a `spawn`, `execFile`, `fork`, `execSync` or `path.join`;
141// as the target of an `import`, a `require` or a `from`; or on the right of a shell variable
142// assignment, which is how `run_all.sh` names `dev/smtpd.mjs`.
143//
144// Comment lines are dropped, and so are lines whose business is to PRINT a filename rather
145// than run it -- `say`, `echo`, `console.log`, and the continuation lines of an assembled
146// message. Without that, `run_all.sh`'s own header, which names five other scripts while
147// explaining why lists go stale, would make all five look wired in.
148const CALLS = /(?:^|[\s;&|("'])(?:bash|sh|node|python3?|source|\.)\s|\bspawn|\bexecFile|\bfork\(|\bexecSync|path\.join|\brequire\(|\bimport\(|\bfrom\s+['"]|^[A-Za-z_][A-Za-z0-9_]*=/;
149const PRINTS = /^\s*(say|echo)\b|console\.(log|error|warn)|^\s*\+/;
150const COMMENT = /^\s*(#|\/\/|\*|\/\*)/;
151const NAMED = /([A-Za-z0-9_.-]+\.(?:mjs|js|sh|py))\b/g;
152
153function edgesOf(rel) {
154 const out = new Set();
155 let text;
156 try { text = fs.readFileSync(path.join(ROOT, rel), 'utf8'); } catch { return out; }
157 for (const line of text.split('\n')) {
158 if (COMMENT.test(line) || PRINTS.test(line) || !CALLS.test(line)) continue;
159 for (const m of line.matchAll(NAMED)) {
160 const target = byName.get(m[1]);
161 if (target && target !== rel) out.add(target);
162 }
163 }
164 return out;
165}
166
167// THE SEEDS ARE READ OUT OF `dev/run_all.sh`, NOT RESTATED HERE.
168//
169// The first draft of this file listed phase 0's five static checks by name, and that draft
170// could not have caught its own subject: delete `static_one jscheck` from the gate and the
171// hand-kept list here would have gone on calling jscheck reachable. A checker for hand-kept
172// lists, kept by hand. So the only seed is the gate itself, and everything it runs is found
173// by following what it NAMES -- `static_one jscheck bash dev/jscheck.sh` is an edge like any
174// other, and a phase-0 line that is deleted stops being one.
175//
176// One construction cannot be followed that way, because it is a glob the shell expands rather
177// than a name: `ALL=$(cd dev && ls verify_*.mjs | ...)`, with `refluxduo` appended beside it.
178// That is read as a form and asserted below -- if the work list stops being built this way,
179// this file says so and fails, rather than quietly seeding nothing.
180const GLOB = /^dev\/verify_[A-Za-z0-9_]+\.mjs$/;
181const gateText = fs.readFileSync(path.join(ROOT, 'dev/run_all.sh'), 'utf8');
182const usesGlob = /\bls\s+verify_\*\.mjs\b/.test(gateText);
183const appended = [...gateText.matchAll(/^\s*ALL="\$ALL\s+([A-Za-z0-9_]+)"/gm)].map(m => `dev/${m[1]}.mjs`);
184const SEEDS = [
185 'dev/run_all.sh',
186 ...(usesGlob ? runnable.filter(p => GLOB.test(p)) : []),
187 ...appended,
188];
189
190const via = new Map();
191const seen = new Set();
192const queue = [];
193for (const s of SEEDS) { via.set(s, 'the work list in dev/run_all.sh'); queue.push(s); }
194while (queue.length) {
195 const p = queue.shift();
196 if (seen.has(p) || !runSet.has(p)) continue;
197 seen.add(p);
198 for (const e of edgesOf(p)) {
199 if (seen.has(e) || via.has(e)) continue;
200 via.set(e, p);
201 queue.push(e);
202 }
203}
204
205// ── The register ────────────────────────────────────────────────────────────
206//
207// One row per file, and the path is the first backticked thing on the row. Prose around the
208// rows is for the reader; only the rows are read here.
209const REGISTER = 'dev/CHECKS.md';
210const registered = new Map();
211let regText = '';
212try { regText = fs.readFileSync(path.join(ROOT, REGISTER), 'utf8'); } catch { /* handled below */ }
213for (const line of regText.split('\n')) {
214 if (!line.startsWith('|')) continue;
215 const m = line.match(/^\|\s*`([^`]+)`\s*\|/);
216 if (m) registered.set(m[1], line.split('|')[2]?.trim() ?? '');
217}
218
219// ── The checks ──────────────────────────────────────────────────────────────
220console.log(`dev/ holds ${roster.length} files, of which ${runnable.length} are runnable.`);
221console.log(`${seen.size} are reachable from dev/run_all.sh; ${registered.size} rows in ${REGISTER}.\n`);
222
223// 1. A checker that enumerates nothing must not report a pass. `git ls-files` answers with
224// silence and exit 0 from a directory that is not a repository, and silence read as a
225// green is this whole blocker in one line.
226if (roster.length > 0 && runnable.length > 0 && roster.includes('dev/run_all.sh')) {
227 ok(`the roster came from git and holds dev/run_all.sh — ${roster.length} files`);
228} else {
229 bad('git listed no dev/ files, or not dev/run_all.sh. An empty enumeration is not a pass.');
230}
231
232// 2. The work list is still built the way this file knows how to read. A gate that stopped
233// globbing `verify_*.mjs` would leave every one of them unseeded, and 281 orphans would
234// read as an avalanche rather than as the one thing that actually changed.
235if (usesGlob) {
236 ok(`dev/run_all.sh still builds its work list from \`ls verify_*.mjs\`, plus ${appended.length ? appended.join(', ') : 'nothing'}`);
237} else {
238 bad('dev/run_all.sh no longer builds its work list from `ls verify_*.mjs`. Teach this file the '
239 + 'new form before reading anything below it — with no seeds, every verifier looks orphaned.');
240}
241
242// 3. Every file the suite's OWN glob matches must come out reachable. If it does not, the
243// graph below is broken and every other answer on this page is worthless.
244const globbed = runnable.filter(p => GLOB.test(p));
245const globMiss = globbed.filter(p => !seen.has(p));
246if (globMiss.length === 0) {
247 ok(`all ${globbed.length} verify_*.mjs the gate's own glob matches are reachable`);
248} else {
249 bad(`${globMiss.length} verify_*.mjs are not reachable, so the graph is wrong: ${globMiss.slice(0, 5).join(', ')}`);
250}
251
252// 4. Every extension in dev/ is classified. An unknown one is a file nobody has decided
253// about, and deciding by silence is the fault.
254const unknownExt = new Map();
255for (const p of roster) {
256 const ext = extOf(p);
257 if (RUNNABLE.has(ext) || INERT.has(ext)) continue;
258 if (isRunnable(p)) continue; // a shebang settled it
259 if (!unknownExt.has(ext)) unknownExt.set(ext, []);
260 unknownExt.get(ext).push(p);
261}
262if (unknownExt.size === 0) {
263 const kinds = [...new Set(roster.map(extOf))].sort().join(' ');
264 ok(`every extension in dev/ is classified as a program or as data — ${kinds}`);
265} else {
266 for (const [ext, files] of unknownExt) {
267 bad(`.${ext} is on neither the runnable nor the inert list, so ${files.length} file(s) `
268 + `were skipped without a decision: ${files.slice(0, 3).join(', ')}. Add the extension to `
269 + 'RUNNABLE or INERT in dev/verify_checkreach.mjs.');
270 }
271}
272
273// 5. THE CHECK THIS FILE IS FOR.
274const orphans = runnable.filter(p => !seen.has(p) && !registered.has(p));
275if (orphans.length === 0) {
276 ok(`every runnable file in dev/ is in the gate or in ${REGISTER} — ${seen.size} + ${runnable.length - seen.size}`);
277} else {
278 bad(`${orphans.length} runnable file(s) in dev/ are in neither the gate nor ${REGISTER}:`);
279 for (const p of orphans) console.log(` ${p}`);
280 console.log(` Wire each into dev/run_all.sh, or add a row to ${REGISTER} saying why not.`);
281}
282
283// 6. The register cannot name a file that is gone.
284//
285// ASKED OF THE DISK, not of the roster. `git ls-files --cached` still names a tracked file
286// that has been deleted from the working tree, so testing against the roster made this check
287// inert until somebody ran `git rm` -- which is to say, inert during exactly the window in
288// which a row goes stale. It was proved by moving `dev/shot_badge.mjs` aside, and it stayed
289// green.
290const ghosts = [...registered.keys()].filter(p => !fs.existsSync(path.join(ROOT, p)));
291if (ghosts.length === 0) {
292 ok(`every one of the ${registered.size} rows in ${REGISTER} names a file that is on disk`);
293} else {
294 bad(`${ghosts.length} row(s) in ${REGISTER} name files that are gone: ${ghosts.join(', ')}`);
295}
296
297// 7. Nor a file that is in the gate. A row saying "the gate does not run this" about something
298// the gate runs is a false sentence about the gate, and the next reader believes it.
299const liars = [...registered.keys()].filter(p => seen.has(p));
300if (liars.length === 0) {
301 ok(`no row in ${REGISTER} claims a file is outside the gate that the gate reaches`);
302} else {
303 bad(`${liars.length} row(s) in ${REGISTER} name files the gate DOES reach: ${liars.join(', ')}. Delete the rows.`);
304}
305
306// 8. And the register must not name a file that is not runnable, which would mean the row was
307// written about something this check would never have asked about in the first place.
308const inertRows = [...registered.keys()].filter(p => roster.includes(p) && !runSet.has(p));
309if (inertRows.length === 0) {
310 ok(`every row in ${REGISTER} names a runnable file`);
311} else {
312 bad(`${inertRows.length} row(s) in ${REGISTER} name files that hold no program: ${inertRows.join(', ')}`);
313}
314
315if (LIST) {
316 console.log('\nreachable, and by what path:');
317 for (const p of [...seen].sort()) {
318 if (GLOB.test(p)) continue; // the work list itself; 281 lines of no news
319 console.log(` ${p} <- ${via.get(p)}`);
320 }
321 console.log('\nregistered as outside the gate:');
322 for (const p of [...registered.keys()].sort()) console.log(` ${p}`);
323}
324
325console.log(`\n${oks.length} ok, ${fails.length} failed`);
326process.exit(fails.length ? 1 : 0);