Oregami
Repositories/oxedyne/daimond

oxedyne/daimond/dev/verify_outcome.mjs

29.7 KiB, 1 run

created by r2519314175:557, which is this file's identity for as long as the history lasts, whatever it is later renamed to

download · who wrote it · its history

1// verify_outcome.mjs — a tool's outcome travels as DATA, and four consumers read it.
2//
3// THE DEFECT. `toolFailed` was `/^\s*Error\b/i` over a tool result's TEXT. It missed
4// `Refused:`, which `refusal_line` (src/tools.rs) prefixes unconditionally across
5// some twenty sites — so every fence refusal drew as a completed step, was
6// journalled as a SUCCESS, was never telemetered as `tool.fail`, and told
7// `DaimondSignals.noteTool` the tool had worked. Widening the regex was not the
8// fix: four consumers were each guessing back a fact the tool layer already had.
9// `CallOutcome { Done, Refused, Failed }` and `call_outcome()` have been sitting in
10// src/tools.rs the whole time. `AgentEvent::ToolResult` now carries the answer as a
11// third field, `outcome`, and `toolFailed` is deleted — dev/CONTRACT_OUTCOME.md.
12//
13// FIVE PROPERTIES, and they are the four consumers plus the store:
14//
15// 1. A REFUSAL IS SHOWN AS A REFUSAL — its own state, in the warning colour and
16// NAMED, not the danger red of a broken tool and not a silent tick.
17// 2. AND JOURNALLED AS ONE. `J.toolDone(..., failed)` says the call did not
18// complete, so a recovered turn does not resurrect it as a success.
19// 3. AND TELEMETERED AS ONE. `tel('tool.fail', …)` fires, so which capability is
20// being refused in the field is visible at all.
21// 4. AND REPORTED TO THE OPTIMISER AS ONE. `noteTool(name, ok)` gets `false`.
22// 6. AND AN OUTCOME THE BUILD DOES NOT RECOGNISE SAYS SO — loudly, because a
23// quiet one is a stale engine wearing a tick — WITHOUT rewriting the tool's
24// own words to say it. This is the state the app is in right now, between the
25// two lanes, and it is a property rather than an accident.
26// 5. AND STORED WITH THE TOOL LOG, so the one prose reader left —
27// `outcomeOfStoredText`, for conversations written before the field existed —
28// serves a set that stops growing rather than one that grows for ever.
29// 7. AND THE WRITE-AHEAD JOURNAL STORES IT TOO. Added 2026-08-28. §4's promise
30// that the exception stops growing was not kept: `journal.js`'s `toolDone`
31// took a BOOLEAN, and the crash-recovery path rebuilt the third value out of
32// the result text — the quarantined reader, on the live path, in the journal
33// of the build running today. A refusal recovered after an interrupted turn
34// came back as a FAILURE. The fixture is a refusal whose words read like a
35// success, because a result beginning "Refused" recovers correctly even from
36// the text and would prove nothing.
37//
38// WHY THE FIXTURE LIES ON PURPOSE. Turn A is truthful: three tool calls whose
39// outcomes are what their text says. Every check would pass on turn A alone with
40// the OLD text-sniffing code still in place, so turn A proves nothing about where
41// the fact came from. Turn B runs the same three tools and stamps outcomes that
42// CONTRADICT the prose — a successful write called `refused`, a `Refused:` and an
43// `Error:` both called `done`. A consumer still reading the text is caught there
44// and nowhere else. Contradicting the prose is not a licence the app has: the
45// engine's word is the truth by contract, and turn B is the only way to ask
46// whether the app believes that.
47//
48// WHAT IS SIMULATED, AND SAID OUT LOUD. The producing half is another lane's:
49// until it lands, `ev.outcome` is `undefined` in a live run. So this file stands in
50// for it — `DaimondApp.prototype.run_turn` is wrapped in the served wasm glue, and
51// the sink it passes on stamps each `tool_result` from a plan this file owns. The
52// stamp is not derived from anything the app can see; it is the test's own arranged
53// truth, which is exactly what the engine's field will be. CHECK 0 reports whether
54// the engine ITSELF sent an outcome, and fails until the other lane lands. It is
55// meant to fail today. Nothing else in this file falls back to reading text.
56//
57// EACH PROPERTY PROVED AGAINST BROKEN CODE FIRST:
58//
59// node dev/verify_outcome.mjs --break sniff # turn B's checks fail: the text wins again
60// node dev/verify_outcome.mjs --break flat # 1 fails: a refusal is drawn as a failure
61// node dev/verify_outcome.mjs --break journal # 2 fails: journalled as a success
62// node dev/verify_outcome.mjs --break tel # 3 fails: never telemetered
63// node dev/verify_outcome.mjs --break signals # 4 fails: the Optimiser is told it worked
64// node dev/verify_outcome.mjs --break nostore # 5 fails: the history exception grows
65// node dev/verify_outcome.mjs --break quiet # 6 fails: a missing outcome draws as a tick
66// node dev/verify_outcome.mjs --break mangle # 6 fails: friendlyError rewrites the tool's words
67// node dev/verify_outcome.mjs --break friendly # 1 fails: the refusal's own sentence is prettified away
68// node dev/verify_outcome.mjs --break alarm # 1 fails: a call that completed is drawn as broken
69// node dev/verify_outcome.mjs --break swap # 1 fails: a failure is drawn as a refusal
70// node dev/verify_outcome.mjs --break flatten # 7 fails: the journal takes a boolean again
71// node dev/verify_outcome.mjs # and then, clean but for check 0
72//
73// eval "$(bash dev/world.sh 6 --up)"
74// node dev/verify_outcome.mjs
75//
76// Needs dev/serve.mjs and the mock. No gateway. No wasm rebuild — the wasm is the
77// one already built; only its JS glue is wrapped, and only in this file's browser.
78import fs from 'node:fs';
79import path from 'node:path';
80import { fileURLToPath } from 'node:url';
81import { open, newChat, chat, storedChats, signInAs, scratch, shot, errors } from './harness.mjs';
82
83const HERE = path.dirname(fileURLToPath(import.meta.url));
84const WWW = path.join(HERE, '..', 'www');
85
86const BREAK = (() => {
87 const i = process.argv.indexOf('--break');
88 return i > 0 ? String(process.argv[i + 1] || '') : '';
89})();
90
91// Each break is a list of [find, replace, howManyTimes] over www/js/daimond.js. The
92// count is asserted before the browser opens, so an anchor that has moved stops the
93// run rather than patching nothing and reporting green.
94const BREAKS = {
95 // Every consumer reads the result TEXT again — the state before the seam. Turn A
96 // still passes under this, which is the point of turn B.
97 sniff: [
98 ["if (ev.outcome !== 'done') tel('tool.fail'",
99 "if (outcomeOfStoredText(ev.content || '') !== 'done') tel('tool.fail'", 1],
100 ["ev.content || '', ev.outcome || '');",
101 "ev.content || '', outcomeOfStoredText(ev.content || ''));", 1],
102 ["DaimondSignals.noteTool(ev.name || '', ev.outcome === 'done');",
103 "DaimondSignals.noteTool(ev.name || '', outcomeOfStoredText(ev.content || '') === 'done');", 1],
104 ["renderToolResult(ev.name || '', ev.content || '', ev.outcome);",
105 "renderToolResult(ev.name || '', ev.content || '', outcomeOfStoredText(ev.content || ''));", 2],
106 ],
107 // Refused and failed collapse back into one state, as they were when nothing
108 // could tell them apart.
109 flat: [["var refused = outcome === 'refused';", "var refused = false;", 1]],
110 journal: [["ev.content || '', ev.outcome || '');", "ev.content || '', 'done');", 1]],
111 tel: [["if (ev.outcome !== 'done') tel('tool.fail'", "if (false) tel('tool.fail'", 1]],
112 signals: [["DaimondSignals.noteTool(ev.name || '', ev.outcome === 'done');",
113 "DaimondSignals.noteTool(ev.name || '', true);", 1]],
114 // The outcome is not written down with the log, so history is back to guessing.
115 nostore: [["pendingTool.outcome = ev.outcome || '';", "pendingTool.outcome = '';", 1]],
116 // An outcome the build does not recognise is drawn as a success -- the quiet
117 // failure mode, where a stale engine looks like a working one.
118 quiet: [["var failed = !refused && outcome !== 'done';",
119 "var failed = outcome === 'failed';", 1]],
120 // friendlyError is let loose on anything not drawn as a success again, which is
121 // what turns "Wrote 6 bytes" into "Could not reach that endpoint".
122 mangle: [["resPre.textContent = outcome === 'failed' ? friendlyError(result) : stripAnsi(result);",
123 "resPre.textContent = failed ? friendlyError(result) : stripAnsi(result);", 1]],
124 // The plausible over-correction: a refusal is not a success, so put it through
125 // the error prettifier too -- and lose the sentence the fence wrote for the user.
126 friendly: [["resPre.textContent = outcome === 'failed' ? friendlyError(result) : stripAnsi(result);",
127 "resPre.textContent = outcome !== 'done' ? friendlyError(result) : stripAnsi(result);", 1]],
128 // The other over-correction: everything that is not a refusal is a failure, so a
129 // call that completed is drawn as a broken one.
130 alarm: [["var failed = !refused && outcome !== 'done';", "var failed = !refused;", 1]],
131 // The two non-done states collapse the other way: a failure is drawn as a refusal.
132 swap: [["var refused = outcome === 'refused';", "var refused = outcome !== 'done';", 1]],
133 // THE SEAM AS IT STOOD IN THE WRITE-AHEAD JOURNAL, both halves. `toolDone` took a
134 // BOOLEAN, so a recovered turn could say that a call did not complete but not
135 // which of the two ways -- and the recovery path rebuilt the third value by
136 // reading the result TEXT, through the reader §4 quarantines for conversations
137 // written before the field existed. A refusal whose words do not begin "Refused"
138 // came back as a FAILURE, which is what check 7 reads back. It reddens the two
139 // journal checks with it, and honestly: the recorder in `arm()` keeps the fifth
140 // argument verbatim, so under this break it records `"true"` and `"false"` --
141 // which is the flattening itself, seen at the call site rather than after the
142 // round trip.
143 flatten: [
144 ["result: result, outcome: String(outcome || '') }), true);",
145 "result: result, failed: !!outcome }), true);", 1, 'js/journal.js'],
146 ["c.tools[i].outcome = String(e.outcome || ''); c.tools[i].failed = !!e.failed;",
147 "c.tools[i].outcome = ''; c.tools[i].failed = !!e.failed;", 1, 'js/journal.js'],
148 ["ev.content || '', ev.outcome || '');", "ev.content || '', ev.outcome !== 'done');", 1],
149 ["\t\t\t\t\t: tl.outcome ? tl.outcome\n\t\t\t\t\t: !tl.failed ? 'done'\n",
150 "\t\t\t\t\t: !tl.failed ? 'done'\n", 1],
151 ],
152};
153if (BREAK && !BREAKS[BREAK]) {
154 console.error(`unknown break '${BREAK}'; one of: ${Object.keys(BREAKS).join(', ')}`);
155 process.exit(2);
156}
157
158const GLUE_SRC = fs.readFileSync(path.join(WWW, 'pkg/oxedyne_daimond.js'), 'utf8');
159
160// A break entry is `[find, replace, count]` and patches `js/daimond.js`, which is
161// where all but one of them live. A fourth element names another served file
162// instead: the seam has two halves and `flatten` restores BOTH, because the call
163// site flattening and the journal storing a boolean are one defect and a break
164// that put back half of it would be a fixture nobody wrote.
165const DEFAULT_FILE = 'js/daimond.js';
166const damaged = new Map();
167const sourceOf = (f) => {
168 if (!damaged.has(f)) damaged.set(f, fs.readFileSync(path.join(WWW, f), 'utf8'));
169 return damaged.get(f);
170};
171sourceOf(DEFAULT_FILE);
172if (BREAK) {
173 for (const [find, repl, want, file] of BREAKS[BREAK]) {
174 const f = file || DEFAULT_FILE;
175 const got = sourceOf(f).split(find).length - 1;
176 if (got !== want) {
177 console.error(`--break ${BREAK}: expected ${want} occurrence(s) in ${f} of\n ${find}\nbut found ${got}; `
178 + 'the anchor has moved and this break would patch nothing');
179 process.exit(2);
180 }
181 damaged.set(f, sourceOf(f).split(find).join(repl));
182 }
183}
184
185// The producing half of the seam, stood in for. Appended to the wasm glue, which is
186// where `DaimondApp` is declared, so the wrap is on the class the app really uses.
187// `__plan` is set from this file before each turn; with no plan, nothing is stamped.
188const SHIM = `
189/* ── dev/verify_outcome.mjs: stands in for AgentEvent::ToolResult's outcome field ── */
190const __vo_run_turn = DaimondApp.prototype.run_turn;
191DaimondApp.prototype.run_turn = function (msg, onEvent) {
192 return __vo_run_turn.call(this, msg, function (ev) {
193 if (ev && ev.type === 'tool_result') {
194 try {
195 // What the ENGINE sent, recorded before anything is written over it.
196 (globalThis.__seen = globalThis.__seen || []).push(
197 ev.outcome === undefined ? null : String(ev.outcome));
198 const plan = globalThis.__plan || [];
199 const at = globalThis.__at | 0;
200 if (at < plan.length) { ev.outcome = plan[at]; globalThis.__at = at + 1; }
201 } catch (e) { /* the shim may never break the run it observes */ }
202 }
203 return onEvent(ev);
204 });
205};
206`;
207
208let bad = 0;
209const check = (pass, name, detail) => {
210 if (!pass) bad++;
211 console.log((pass ? ' ok ' : ' FAIL ') + name + (detail ? ' — ' + detail : ''));
212};
213
214const s = await open({
215 name: 'outcome',
216 profile: scratch('pw', 'outcome' + (BREAK ? '-' + BREAK : '')),
217 route: async (page) => {
218 await page.route('**/pkg/oxedyne_daimond.js', (r) => r.fulfill({
219 status: 200, contentType: 'application/javascript', body: GLUE_SRC + SHIM,
220 }));
221 if (BREAK) for (const [file, body] of damaged) {
222 await page.route('**/' + file, (r) => r.fulfill({
223 status: 200, contentType: 'application/javascript', body,
224 }));
225 }
226 },
227});
228const { page: p } = s;
229if (BREAK) console.log(`\n*** RUNNING UNDER --break ${BREAK}: failures below are the point ***\n`);
230
231/// Arm the shim with the outcomes to stamp, in the order the results come back, and
232/// start the recorders that the four consumers write into.
233const arm = (plan) => p.evaluate((plan) => {
234 globalThis.__plan = plan;
235 globalThis.__at = 0;
236 globalThis.__seen = [];
237 globalThis.__tel = [];
238 globalThis.__jrn = [];
239 // The journal and the telemetry client are both read at CALL time by the app
240 // (`var J = window.DaimondJournal` at the top of the turn, `window.DaimondTelemetry`
241 // inside `tel`), so wrapping them here is enough and nothing has to be reloaded.
242 if (window.DaimondJournal && !window.DaimondJournal.__wrapped) {
243 const orig = window.DaimondJournal.toolDone;
244 // THE FIFTH ARGUMENT, verbatim. It was `!!failed` here, which is what the call
245 // site passed and is exactly the flattening check 7 is about: recorded as a
246 // boolean, a check on it could not tell a refusal from a failure either.
247 window.DaimondJournal.toolDone = function (turnId, chatId, callId, result, outcome) {
248 globalThis.__jrn.push(outcome === undefined ? null : String(outcome));
249 return orig.apply(this, arguments);
250 };
251 window.DaimondJournal.__wrapped = true;
252 }
253 if (window.DaimondTelemetry && !window.DaimondTelemetry.__wrapped) {
254 const orig = window.DaimondTelemetry.emit;
255 window.DaimondTelemetry.emit = function (name, n) {
256 globalThis.__tel.push(name + ':' + n);
257 return orig.apply(this, arguments);
258 };
259 window.DaimondTelemetry.__wrapped = true;
260 }
261 if (window.DaimondSignals) window.DaimondSignals.reset();
262 return true;
263}, plan);
264
265/// Every tool block on screen, in order, as the reader sees it.
266const blocks = () => p.evaluate(() => [...document.querySelectorAll('#chat-output .tool-block')]
267 .map((b) => {
268 const tag = b.querySelector('.tool-outcome');
269 const res = b.querySelector('.tool-result');
270 return {
271 cls: b.className,
272 failed: b.classList.contains('failed'),
273 refused: b.classList.contains('refused'),
274 word: tag ? tag.textContent.trim() : '',
275 colour: tag ? tag.style.color : '',
276 border: b.style.borderColor || '',
277 text: res ? res.textContent.slice(0, 120) : '',
278 };
279 }));
280
281/// This fixture's three tool logs, as the STORE holds them: raw result text and
282/// the outcome written down beside it.
283///
284/// Read from storage rather than off the screen, because the screen is what several
285/// of the checks below are about -- a fixture assertion that reads the rendering is
286/// not a fixture assertion, and under `--break flat` it failed for the thing it was
287/// supposed to hold still.
288const toolLogs = async () => {
289 const chats = await storedChats(s);
290 const out = [];
291 chats.forEach((c) => (c.messages || []).forEach((m) => {
292 if (m.role === 'tool_log' && /ok\.txt|must-not-write|not-here/.test(String(m.args || ''))) out.push(m);
293 }));
294 return out.slice(-3);
295};
296
297const recorded = () => p.evaluate(() => ({
298 seen: globalThis.__seen || [],
299 tel: globalThis.__tel || [],
300 jrn: globalThis.__jrn || [],
301 tools: (window.DaimondSignals ? window.DaimondSignals.snapshot().tools : {}) || {},
302}));
303
304/// The three-call fixture, written into the chat now in focus.
305///
306/// A write that lands, a write the FENCE refuses, and a read of a file that is not
307/// there. One turn, one round: `@tools` returns all three calls together, so the
308/// three results arrive in this order and the stamping plan lines up with them.
309const fixture = async () => {
310 const dir = await p.evaluate(() => {
311 const f = window.DaimondAttach.focus();
312 return window.DaimondAttach.chatScratch(f.id);
313 });
314 return '@tools '
315 + `file_write {"path":"${dir}/ok.txt","content":"landed"}`
316 + ' ;; '
317 + 'file_write {"path":"/etc/daimond-must-not-write.txt","content":"nope"}'
318 + ' ;; '
319 + `file_read {"path":"${dir}/not-here.txt"}`;
320};
321
322try {
323 // ── Turn A: the outcomes agree with the prose ────────────────
324 await newChat(s);
325 await arm(['done', 'refused', 'failed']);
326 await chat(s, await fixture(), { timeout: 60000 });
327 const a = await blocks();
328 const ar = await recorded();
329 const al = await toolLogs();
330 await shot(s, 'outcome-truthful');
331
332 // THE FIXTURE IS THE FIXTURE. Every assertion below is about three particular
333 // results; a turn that produced something else would make them meaningless. Asked
334 // of the STORED result text, which no rendering decision can move.
335 check(a.length === 3 && al.length === 3
336 && !/^\s*(?:Error|Refused)\b/i.test(String(al[0].content || ''))
337 && /^\s*Refused\b/i.test(String(al[1].content || ''))
338 && /^\s*Error\b/i.test(String(al[2].content || '')),
339 'THE FIXTURE RAN: a write that landed, a fence refusal, and a read that failed',
340 `${a.length} tool block(s); `
341 + al.map((m) => JSON.stringify(String(m.content || '').slice(0, 26))).join(' / '));
342
343 // ── 0. The seam itself ───────────────────────────────────────
344 //
345 // EXPECTED TO FAIL until the engine sends the field. Everything below runs on
346 // this file's stand-in for it, and this is the check that says so out loud
347 // rather than letting the rest imply the seam is live.
348 const live = ar.seen.filter((o) => o === 'done' || o === 'refused' || o === 'failed').length;
349 check(live === a.length && a.length > 0,
350 'THE ENGINE ITSELF SENDS THE OUTCOME (fails until the Rust lane lands)',
351 `the engine sent ${JSON.stringify(ar.seen)} for ${a.length} result(s)`);
352
353 // ── 1. Shown as a refusal ────────────────────────────────────
354 check(a.length === 3 && a[1].refused && !a[1].failed,
355 'A REFUSAL IS SHOWN AS A REFUSAL — its own state, not the red of a broken tool',
356 a.length === 3 ? `classes ${JSON.stringify(a[1].cls)}` : 'the fixture did not run');
357 check(a.length === 3 && /refused/i.test(a[1].word) && /warn/.test(a[1].border),
358 'and it is NAMED as well as coloured, so it does not rest on amber against red',
359 a.length === 3 ? `word ${JSON.stringify(a[1].word)}, border ${JSON.stringify(a[1].border)}` : '');
360 check(a.length === 3 && a[2].failed && !a[2].refused && /failed/i.test(a[2].word),
361 'while a real failure keeps the failure state, so the two are told apart',
362 a.length === 3 ? `classes ${JSON.stringify(a[2].cls)}, word ${JSON.stringify(a[2].word)}` : '');
363 check(a.length === 3 && !a[0].failed && !a[0].refused && a[0].word === '',
364 'and a call that completed is left alone',
365 a.length === 3 ? `classes ${JSON.stringify(a[0].cls)}` : '');
366 // The refusal's own sentence, unmangled. `friendlyError` collapses paragraphs and
367 // swallows any string naming a number into its status-code arm, so a refusal put
368 // through it would lose the very text this seam exists to start showing.
369 check(a.length === 3 && /^\s*Refused\b/i.test(a[1].text),
370 'and the refusal is shown in its own words, not put through friendlyError',
371 a.length === 3 ? JSON.stringify(a[1].text.slice(0, 70)) : '');
372
373 // ── 2, 3, 4. Journalled, telemetered, reported ───────────────
374 check(JSON.stringify(ar.jrn) === JSON.stringify(['done', 'refused', 'failed']),
375 'AND JOURNALLED AS ONE — the refusal is written down as a refusal, in the engine’s own word',
376 `toolDone(outcome) = ${JSON.stringify(ar.jrn)}`);
377 const failsA = ar.tel.filter((e) => e.indexOf('tool.fail:') === 0);
378 check(failsA.length === 2,
379 'AND TELEMETERED AS ONE — tool.fail fires for the refusal and the failure, not the write',
380 `${failsA.length} tool.fail event(s): ${JSON.stringify(failsA)}`);
381 check((ar.tools.file_write || {}).calls === 2 && (ar.tools.file_write || {}).failed === 1
382 && (ar.tools.file_read || {}).failed === 1,
383 'AND REPORTED TO THE OPTIMISER AS ONE — noteTool is told the refused call did not work',
384 JSON.stringify(ar.tools));
385
386 // ── 5. Stored with the log ───────────────────────────────────
387 check(JSON.stringify(al.map((m) => m.outcome)) === JSON.stringify(['done', 'refused', 'failed']),
388 'AND STORED WITH THE TOOL LOG, so the history exception stops growing',
389 `stored outcomes ${JSON.stringify(al.map((m) => m.outcome))}`);
390
391 // ── Turn B: the outcomes CONTRADICT the prose ────────────────
392 //
393 // The check that makes the six above mean something. Every one of them would
394 // pass with the old text-sniffing code still in place, because on turn A the
395 // text and the truth agree. Here they do not, and only a consumer reading the
396 // FIELD can get this right.
397 await newChat(s);
398 await arm(['refused', 'done', 'done']);
399 await chat(s, await fixture(), { timeout: 60000 });
400 const b = await blocks();
401 const br = await recorded();
402 const bl = await toolLogs();
403 await shot(s, 'outcome-crossed');
404
405 check(b.length === 3 && bl.length === 3
406 && !/^\s*(?:Error|Refused)\b/i.test(String(bl[0].content || ''))
407 && /^\s*Refused\b/i.test(String(bl[1].content || ''))
408 && /^\s*Error\b/i.test(String(bl[2].content || '')),
409 'THE CROSSED FIXTURE RAN: three results whose text says the opposite of their outcome',
410 `${b.length} tool block(s); `
411 + bl.map((m) => JSON.stringify(String(m.content || '').slice(0, 26))).join(' / '));
412 check(b.length === 3 && b[0].refused,
413 'A SUCCESSFUL-LOOKING RESULT THE ENGINE CALLED REFUSED IS DRAWN AS REFUSED',
414 b.length === 3 ? `classes ${JSON.stringify(b[0].cls)}, text ${JSON.stringify(b[0].text.slice(0, 40))}` : '');
415 check(b.length === 3 && !b[1].refused && !b[1].failed && !b[2].refused && !b[2].failed,
416 'and a "Refused:" and an "Error:" the engine called done are drawn as done',
417 b.length === 3 ? `${JSON.stringify(b[1].cls)} / ${JSON.stringify(b[2].cls)}` : '');
418 check(JSON.stringify(br.jrn) === JSON.stringify(['refused', 'done', 'done']),
419 'the journal follows the engine, not the words',
420 `toolDone(outcome) = ${JSON.stringify(br.jrn)}`);
421 const failsB = br.tel.filter((e) => e.indexOf('tool.fail:') === 0);
422 check(failsB.length === 1,
423 'telemetry follows the engine: one failure reported, and it is the one that reads as a success',
424 `${failsB.length} tool.fail event(s)`);
425 check((br.tools.file_write || {}).calls === 2 && (br.tools.file_write || {}).failed === 1
426 && !((br.tools.file_read || {}).failed),
427 'and so does the Optimiser',
428 JSON.stringify(br.tools));
429
430 // ── 5b. And history redraws it from the STORE, not the prose ─
431 //
432 // A reload is the honest way to ask: nothing of the turn is left in memory, so
433 // what comes back is what was written down.
434 const storedB = bl.map((m) => m.outcome);
435 check(JSON.stringify(storedB) === JSON.stringify(['refused', 'done', 'done']),
436 'the crossed outcomes are what got stored, so a reload cannot fall back to reading the text',
437 `stored outcomes ${JSON.stringify(storedB)}`);
438
439 await p.reload({ waitUntil: 'domcontentloaded' });
440 await p.waitForTimeout(1200);
441 await signInAs(s, 'outcome');
442 await p.waitForTimeout(1500);
443 let after = await blocks();
444 if (!after.length) {
445 // The reload did not land back in the conversation; open it the way a user does.
446 await p.evaluate(() => {
447 const box = document.querySelector('#session-list .chat-box');
448 if (box) box.click();
449 });
450 await p.waitForTimeout(1200);
451 after = await blocks();
452 }
453 await shot(s, 'outcome-reloaded');
454 check(after.length === 3 && after[0].refused && !after[1].refused && !after[1].failed,
455 'AND A RELOADED CONVERSATION REDRAWS THE REFUSAL FROM THE STORED FIELD',
456 after.length === 3 ? `${JSON.stringify(after[0].cls)} / ${JSON.stringify(after[1].cls)}`
457 : `${after.length} tool block(s) after reload`);
458
459 // ── 6. An outcome this build does not know ──────────────────
460 //
461 // A WORD THE ENGINE MIGHT SEND THAT THIS PAGE HAS NEVER HEARD OF -- a newer
462 // engine against an older bundle, or a fourth state added upstream. It must be
463 // flagged rather than waved through as a tick, because "I do not understand what
464 // happened" and "it worked" are the two answers that must never look alike.
465 //
466 // THIS TEST USED TO DISARM THE SHIM and read what the engine really sent, which
467 // was nothing at all while the Rust half was unlanded. That case is now
468 // unreachable: the engine always sends one of the three, so disarming produces
469 // two ordinary DONE calls and the check silently stopped testing anything. Which
470 // is the same defect this whole file exists to prevent, arriving in the file
471 // itself the moment its subject shipped.
472 await newChat(s);
473 await arm(['quinquagenarian', 'quinquagenarian']);
474 const dirC = await p.evaluate(() => {
475 const f = window.DaimondAttach.focus();
476 return window.DaimondAttach.chatScratch(f.id);
477 });
478 // A file of TWO LINES, written and then read back. The second call's result is
479 // the instrument: `friendlyError` collapses all whitespace to single spaces, so a
480 // result that keeps its line break is a result that was not put through it.
481 await chat(s, '@tools '
482 + `file_write {"path":"${dirC}/two.txt","content":"line one\\nline two"}`
483 + ' ;; '
484 + `file_read {"path":"${dirC}/two.txt"}`, { timeout: 60000 });
485 const c = await blocks();
486 await shot(s, 'outcome-absent');
487 check(c.length === 2 && c.every((x) => (x.failed || x.refused) && x.word !== ''),
488 'AN OUTCOME THE BUILD DOES NOT RECOGNISE IS FLAGGED, not drawn as a tick',
489 c.map((x) => `${JSON.stringify(x.cls)}/${JSON.stringify(x.word)}`).join(' '));
490 check(c.length === 2 && c[1].text.indexOf('\n') >= 0
491 && /line one/.test(c[1].text) && /line two/.test(c[1].text),
492 "but the tool's own words are left alone — friendlyError is for a STATED failure",
493 c.length === 2 ? JSON.stringify(c[1].text.slice(0, 60)) : `${c.length} tool block(s)`);
494
495 // ── 7. A REFUSAL SURVIVES AN INTERRUPTED TURN AS A REFUSAL ───
496 //
497 // §4 GRANTS ONE EXCEPTION AND SAYS IT STOPS GROWING: *"Going forward the outcome
498 // is STORED with the tool log."* It was not. `journal.js`'s `toolDone` took
499 // `(turnId, chatId, callId, result, failed)` -- a BOOLEAN where the contract
500 // requires the outcome -- and the crash-recovery path in daimond.js then rebuilt
501 // the third value out of the RESULT TEXT, through `outcomeOfStoredText`, whose
502 // own doc says calling it on a live path puts back the defect the field was added
503 // to remove. This is the write-ahead journal of the build running today, so the
504 // exception grew by one every time a tab died mid-turn.
505 //
506 // THE FIXTURE IS A REFUSAL WHOSE WORDS READ LIKE A SUCCESS, and it has to be:
507 // a result beginning "Refused" recovers correctly even from the text, so a
508 // natural-looking fixture would pass against the defect and prove nothing. The
509 // engine called it refused; the prose says a file was written. Only the stored
510 // field can get this right, which is the whole of §3.
511 //
512 // DRIVEN THROUGH THE JOURNAL'S OWN API AND A REAL RELOAD, not by calling the
513 // recovery function: what is under test is the round trip, and half of it is
514 // what `toolDone` chose to write down.
515 await newChat(s);
516 await chat(s, '@text a turn that will be interrupted', { timeout: 60000 });
517 const icid = await p.evaluate(() => {
518 const f = window.DaimondAttach && window.DaimondAttach.focus();
519 return (f && f.kind === 'chat') ? String(f.id) : '';
520 });
521 const iturn = 'vo-interrupted-' + Date.now();
522 // Opened and never closed, which is what an interrupted turn IS: `recover()`
523 // returns a turn only when no `turn_close` landed for it.
524 const wrote = await p.evaluate(async ({ cid, tid }) => {
525 const J = window.DaimondJournal;
526 if (!J) return false;
527 await J.turnOpen(tid, cid, 'write the fixture file', null);
528 await J.toolOpen(tid, cid, 'vo-call-1', 'file_write', '{"path":"vo-interrupted.txt"}');
529 await J.toolDone(tid, cid, 'vo-call-1', 'Wrote 8 bytes to vo-interrupted.txt', 'refused');
530 await J.flush();
531 return true;
532 }, { cid: icid, tid: iturn });
533 check(!!icid && wrote,
534 'a turn is journalled and never closed — an interrupted turn, as the journal records one',
535 `chat=${JSON.stringify(icid)} written=${wrote}`);
536
537 await p.reload({ waitUntil: 'domcontentloaded' });
538 await p.waitForTimeout(1200);
539 await signInAs(s, 'outcome');
540 await p.waitForTimeout(3000);
541 const rec = await (async () => {
542 const chats = await storedChats(s);
543 for (const c of chats) {
544 for (const m of (c.messages || [])) {
545 if (m.role === 'tool_log' && /vo-interrupted\.txt/.test(String(m.args || ''))) return m;
546 }
547 }
548 return null;
549 })();
550 check(!!rec,
551 'and it is recovered into the conversation after the reload',
552 rec ? '' : 'no recovered tool_log for the interrupted turn');
553 check(!!rec && rec.outcome === 'refused',
554 'A REFUSED TOOL RECOVERED FROM THE JOURNAL COMES BACK AS REFUSED, not as a failure',
555 `outcome=${JSON.stringify(rec && rec.outcome)} result=${JSON.stringify(String((rec && rec.content) || '').slice(0, 44))}`);
556
557 const errs = errors(s);
558 if (errs.length) console.log(` note ${errs.length} console error(s): ${errs.slice(0, 3).join(' | ')}`);
559} finally {
560 await s.close();
561}
562
563console.log(bad ? `\n${bad} check(s) FAILED` : '\nall checks passed');
564process.exit(bad ? 1 : 0);