Oregami
Repositories/oxedyne/daimond

oxedyne/daimond/dev/verify_dropped.mjs

16.0 KiB, 1 run

created by r2519314175:389, which is this file's identity for as long as the history lasts, whatever it is later renamed to

download · who wrote it · its history

1// verify_dropped.mjs — a turn the ROAD killed is handed back; a turn the PROVIDER
2// refused is not.
3//
4// THE DEFECT, reported live on 2026-08-19 by a user working from a laptop:
5// *"I am moving around during some days like today on my laptop, closing it and
6// moving between locations, so there can be frequent and long interruptions."*
7// Each interruption killed the turn outright. The prompt they had typed, the part
8// of the answer that had arrived, and the tools that had already run were all
9// deleted, and the only thing left was a red line.
10//
11// WHY IT WAS NOT A RETRY PROBLEM, which is what it looked like and what was first
12// tried. `LlmClient::stream_turn` does not retry once tokens have started —
13// `if started || !e.retryable` — and that is correct: re-sending a request whose
14// answer was half delivered bills the answer twice. So the commonest shape of
15// this, a break part way through a reply, was never retryable at any budget. A
16// wider retry policy buys a couple of minutes against a gap that lasts as long as
17// the walk between two buildings.
18//
19// WHAT WAS ACTUALLY MISSING was a judgement, not a mechanism. Everything needed to
20// hand a turn back had been built for the tab-crash case: the write-ahead journal
21// holds the prompt, the partial reply and the tools that ran, and `markInterrupted`
22// draws the badge and the Continue button. But `runTurn`'s catch split the world in
23// two — `_unloading`, which kept the turn, and everything else, which threw it away
24// — and a suspended laptop is neither. It is not unloading; it is asleep.
25//
26// FOUR PROPERTIES:
27//
28// 1. A DROPPED CONNECTION LEAVES AN INTERRUPTED TURN, not an error. The badge is
29// on screen and the Continue button with it.
30// 2. AND THE WORDS THAT ARRIVED SURVIVE. A turn handed back empty is barely
31// better than one thrown away; the partial answer is read off the message.
32// 3. AND CONTINUE CONTINUES IT. The words that arrived STAY, and the model is asked
33// to carry on from them.
34//
35// IT USED TO RE-RUN THE PROMPT, tombstoning the partial reply first. Two costs, and
36// the second is the one that was reported: output tokens already billed were thrown
37// away and bought again, and THE ANSWER THE USER WAS READING WAS REPLACED BY A
38// DIFFERENT ONE. They watched text appear, pressed the button the app offered them,
39// and the text became something else with no way back to it. A tester described
40// exactly that shape — "I see some response text from a model, then later it is
41// superceded by a final answer and I can't see what was there before".
42//
43// A turn that died BEFORE THE FIRST TOKEN still re-runs the prompt, and that is
44// right: there is nothing to carry on from and nothing was billed. That case is
45// dev/verify_predrop.mjs.
46// 4. BUT A PROVIDER REFUSAL IS STILL TERMINAL. This is the check that makes the
47// other three mean something: if everything became recoverable then nothing
48// was classified, and a 500 from the provider would sit there offering a
49// Continue that will fail the same way for ever.
50//
51// EACH PROVED AGAINST BROKEN CODE FIRST:
52//
53// node dev/verify_dropped.mjs --break terminal # 1-3 fail: every death is terminal again
54// node dev/verify_dropped.mjs --break everything # 4 fails: a 500 is offered back too
55// node dev/verify_dropped.mjs --break rerun # 3 fails: Continue throws the partial away
56// node dev/verify_dropped.mjs # and then, clean
57//
58// `--break terminal` restores the old condition — `_unloading` alone — which is
59// the state the defect was reported from. `--break everything` widens it to every
60// error, which is the plausible over-correction and the reason check 4 exists.
61// `--break rerun` sends every Continue down the path meant for a turn that produced
62// nothing, which is `continueTurn` exactly as it stood on 2026-08-27: tombstone the
63// turn and ask the question again.
64//
65// WAITING FOR THE TURN TO END, RATHER THAN FOR A FIXED FIVE SECONDS. Check 4 sends
66// `@err 500` and used to look at the thread 5,000 ms later. A 500 is RETRYABLE —
67// `RetryPolicy::default()` in `src/llm.rs` is eight attempts over up to 120 s of
68// backoff, widened from four attempts over 20 s on 2026-08-19 so a laptop that
69// wakes in another building does not lose its turn — so at five seconds the turn
70// is on attempt 4 of 8 and has not ended. Measured in world 7 on 2026-08-21: the
71// first request goes out at once, the retry notices land at 1.1, 1.7, 2.8, 4.8,
72// 12.1, 20.9 and 40.8 s, and the error line is written at 69.2 s. Nothing was
73// misclassified; the file was reading the thread mid-turn.
74//
75// That made check 4a pass for the wrong reason as well: NOTHING is marked
76// interrupted five seconds in, because nothing has ended, so "a 500 is not offered
77// back" was true of a turn that was still running. Both checks now wait on the
78// composer — the app's own "this turn is over" — and the wait is bounded well
79// above the retry budget, so a turn that really hangs still fails rather than
80// being waited out.
81//
82// eval "$(bash dev/world.sh 4 --up)"
83// node dev/verify_dropped.mjs
84//
85// Needs dev/serve.mjs and the mock. No gateway, no wasm rebuild — the branch under
86// test is JavaScript.
87import fs from 'node:fs';
88import path from 'node:path';
89import { fileURLToPath } from 'node:url';
90import { open, newChat, scratch, shot, storedChats } from './harness.mjs';
91
92const HERE = path.dirname(fileURLToPath(import.meta.url));
93const WWW = path.join(HERE, '..', 'www');
94
95const BREAK = (() => {
96 const i = process.argv.indexOf('--break');
97 return i > 0 ? String(process.argv[i + 1] || '') : '';
98})();
99
100// The two lines that decide it, and the three ways of getting them wrong. Each break
101// names the line it patches, so an anchor that has moved is caught rather than skipped.
102const COND = '\t\t\t\t} else if (offline(e)) {';
103const PART = '\t\tif (!partial.trim()) {';
104const BREAKS = {
105 // The state before the fix: only a closing page keeps its turn.
106 terminal: [COND, '\t\t\t\t} else if (false) {'],
107 // The over-correction: everything is recoverable, so nothing is classified.
108 everything: [COND, '\t\t\t\t} else if (true) {'],
109 // Continue as it was before it continued anything.
110 rerun: [PART, '\t\tif (true) {'],
111};
112if (BREAK && !BREAKS[BREAK]) {
113 console.error(`unknown break '${BREAK}'; one of: ${Object.keys(BREAKS).join(', ')}`);
114 process.exit(2);
115}
116
117let bad = 0;
118const check = (pass, name, detail) => {
119 if (!pass) bad++;
120 console.log((pass ? ' ok ' : ' FAIL ') + name + (detail ? ' — ' + detail : ''));
121};
122
123const SRC = fs.readFileSync(path.join(WWW, 'js/daimond.js'), 'utf8');
124for (const anchor of [COND, PART]) {
125 if (SRC.split(anchor).length !== 2) {
126 console.error(`the line ${JSON.stringify(anchor.trim())} is not in js/daimond.js `
127 + 'exactly once; the anchor has moved and the breaks below would patch nothing');
128 process.exit(2);
129 }
130}
131
132const s = await open({
133 name: 'dropped',
134 profile: scratch('pw', 'dropped' + (BREAK ? '-' + BREAK : '')),
135 // Serve the damaged file in place of the real one, so the page under test is
136 // the shipped page with one line changed and not a copy of it. Registered
137 // before `goto`, which is what `route` is for.
138 route: BREAK ? (async (page) => {
139 const body = SRC.replace(BREAKS[BREAK][0], BREAKS[BREAK][1]);
140 await page.route('**/js/daimond.js', (r) => r.fulfill({
141 status: 200, contentType: 'application/javascript', body,
142 }));
143 }) : null,
144});
145const { page: p } = s;
146if (BREAK) console.log(`\n*** RUNNING UNDER --break ${BREAK}: failures below are the point ***\n`);
147
148/// What the last assistant message in the thread is, and what is offered about it.
149const lastTurn = () => p.evaluate(() => {
150 const msgs = [...document.querySelectorAll('#chat-output .chat-msg')];
151 const last = msgs[msgs.length - 1] || null;
152 const inter = [...document.querySelectorAll('#chat-output .chat-msg.interrupted')].pop() || null;
153 return {
154 interrupted: !!inter,
155 // The button by its role rather than its words, so a translation does not
156 // decide whether this file passes.
157 continues: !!(inter && inter.querySelector('.turn-interrupted button')),
158 text: inter ? (inter.querySelector('.chat-msg-content') || {}).textContent || '' : '',
159 errors: [...document.querySelectorAll('#chat-output .chat-msg-error, #chat-output .error-log')].length,
160 lastClass: last ? last.className : '',
161 };
162});
163
164/// Wait for the composer to offer Send again — the app's own "the turn is over".
165///
166/// The bound is above `RetryPolicy::default()`'s whole budget (eight attempts,
167/// `max_total_wait_ms` 120 s, `max_backoff_ms` 30 s) plus the requests between
168/// them, so a turn that ends fails nothing and a turn that HANGS still fails.
169/// Answers how long it waited, because that number is the retry budget and a
170/// change to it should be visible here rather than as a mystery timeout.
171const settle = async (label, timeout = 180000) => {
172 const t0 = Date.now();
173 while (Date.now() - t0 < timeout) {
174 const busy = await p.evaluate(() => {
175 const b = document.getElementById('chat-send');
176 return !!b && (b.classList.contains('stop') || b.disabled);
177 });
178 if (!busy) return { ended: true, ms: Date.now() - t0 };
179 await p.waitForTimeout(200);
180 }
181 console.log(` note the ${label} turn was still running after ${timeout} ms`);
182 return { ended: false, ms: Date.now() - t0 };
183};
184
185try {
186 await newChat(s);
187
188 // ── 1 and 2. The road fails part way through an answer ───────
189 //
190 // Three words arrive and then the socket is destroyed: no `[DONE]`, no finish
191 // reason, exactly what a lid closing does to a connection.
192 await p.fill('#chat-input', '@drop 3');
193 await p.click('#chat-send');
194 const dropEnd = await settle('dropped');
195 check(dropEnd.ended, 'THE DROPPED TURN ENDS rather than hanging on the road',
196 `${dropEnd.ms} ms`);
197 const dropped = await lastTurn();
198 await shot(s, 'dropped-interrupted');
199
200 check(dropped.interrupted,
201 'A DROPPED CONNECTION LEAVES AN INTERRUPTED TURN, not a dead one',
202 `last message class ${JSON.stringify(dropped.lastClass)}`);
203 check(dropped.continues,
204 'and the turn is offered back with a Continue button',
205 dropped.continues ? '' : (dropped.interrupted ? 'no button in .turn-interrupted'
206 : 'nothing was marked interrupted'));
207 // The words that DID arrive. Asserted on a word the mock only sends in this
208 // mode, so a generic reply cannot satisfy it.
209 check(/word-1/.test(dropped.text),
210 'AND THE WORDS THAT ARRIVED SURVIVE — the partial answer is still there',
211 JSON.stringify(dropped.text.slice(0, 80)));
212
213 // ── 3. Continue CONTINUES it ───────────────────────────────
214 //
215 // The words the user was reading must still be there afterwards. Pressing Continue
216 // keeps the partial as an ordinary answer and asks the model to carry on from it, so
217 // what must be true afterwards is: the same question once, `word-1` still on screen,
218 // nothing badged interrupted any more, and MORE messages than before rather than the
219 // same ones rewritten.
220 //
221 // THE ASSERTION THAT MATTERS IS `word-1` SURVIVING. The count and the badge are
222 // bookkeeping; the text is the complaint.
223 const before = await p.evaluate(() =>
224 document.querySelectorAll('#chat-output .chat-msg').length);
225 // THE PARTIAL'S OWN ID, off the disk. Text alone cannot answer this question: the
226 // prompt is `@drop 3`, so a Continue that RE-RUNS it produces `word-1 word-2` a
227 // second time and a check on the words passes while the user's answer has in fact
228 // been thrown away and bought again. The `mid` is the only thing that tells a kept
229 // message from an identical new one.
230 const midOf = async () => {
231 const chats = await storedChats(s);
232 for (const c of chats) {
233 for (const m of (c.messages || [])) {
234 if (m.role === 'assistant' && m.interrupted && /word-1/.test(m.content || '')) {
235 return m.mid;
236 }
237 }
238 }
239 return '';
240 };
241 const partialMid = await midOf();
242 if (dropped.continues) {
243 await p.evaluate(() => {
244 const inter = [...document.querySelectorAll('#chat-output .chat-msg.interrupted')].pop();
245 inter.querySelector('.turn-interrupted button').click();
246 });
247 await settle('continued');
248 }
249 const after = await p.evaluate(() => ({
250 count: document.querySelectorAll('#chat-output .chat-msg').length,
251 // Nothing is left badged: the partial has become an ordinary answer, and the
252 // continuation finished cleanly.
253 interruptedCount: document.querySelectorAll('#chat-output .chat-msg.interrupted').length,
254 user: [...document.querySelectorAll('#chat-output .chat-msg-user .chat-msg-content')]
255 .map(e => e.textContent).join('|'),
256 // Every assistant word on screen, so "is what I was reading still here?" is asked
257 // of the thread rather than of one element.
258 said: [...document.querySelectorAll('#chat-output .chat-msg-assistant .chat-msg-content')]
259 .map(e => e.textContent).join(' '),
260 }));
261 // The same record, still there, still holding the same words, no longer badged.
262 const kept = await (async () => {
263 const chats = await storedChats(s);
264 for (const c of chats) {
265 for (const m of (c.messages || [])) {
266 if (m.mid && m.mid === partialMid) {
267 return { there: true, said: /word-1/.test(m.content || ''),
268 badged: !!m.interrupted };
269 }
270 }
271 }
272 return { there: false, said: false, badged: false };
273 })();
274 check(!!partialMid && kept.there && kept.said && !kept.badged && /word-1/.test(after.said),
275 'CONTINUE KEEPS WHAT THE USER WAS READING — the same message, not a fresh copy',
276 kept.there ? (kept.badged
277 ? 'the message with that id is still badged interrupted, so what is on screen '
278 + 'is a fresh turn beside it: the turn was re-run, not continued'
279 : (kept.said ? '' : 'the id survived but the words did not'))
280 : `no message ${JSON.stringify(partialMid)} survives: the answer the user was `
281 + 'reading was thrown away and generated again');
282 check(dropped.continues && after.interruptedCount === 0 && after.count > before,
283 'and the turn carries on from it rather than starting again',
284 `${before} messages before, ${after.count} after, ${after.interruptedCount} interrupted`);
285 // One question, not two. This is the assertion that caught the first version of
286 // the offline branch: the prompt was left untagged, so Continue removed the answer,
287 // sent the question again, and the thread held it twice. It still holds, and it is
288 // what tells a continuation apart from a re-run: the app's own "carry on" line is
289 // the second user message, and the user's question is never sent twice.
290 const asked = after.user.split('|').filter(Boolean);
291 check(asked.filter(u => /@drop 3/.test(u)).length === 1,
292 'and the prompt the user typed was never sent again',
293 JSON.stringify(after.user.slice(0, 110)));
294
295 // ── 4. A provider refusal is still terminal ──────────────────
296 //
297 // THE CHECK THAT MAKES THE OTHERS MEAN SOMETHING. A 500 is the provider
298 // answering: it is not interrupted work, and offering Continue on it would
299 // offer a button that fails identically every time it is pressed.
300 await newChat(s);
301 await p.fill('#chat-input', '@err 500');
302 await p.click('#chat-send');
303 // THE TURN MUST HAVE ENDED BEFORE EITHER CHECK BELOW MEANS ANYTHING. A 500 is
304 // retryable, so this is not a five-second wait: it is the retry ladder running
305 // to the end of its budget, about seventy seconds against the mock. Asked as a
306 // check of its own, because "a 500 is not offered back" is trivially true of a
307 // turn still in flight and was passing that way.
308 const errEnd = await settle('500');
309 check(errEnd.ended,
310 'A PROVIDER 500 IS RETRIED TO THE END OF THE BUDGET AND THEN THE TURN ENDS',
311 `${errEnd.ms} ms`);
312 const refused = await lastTurn();
313 await shot(s, 'dropped-refusal');
314 check(!refused.interrupted,
315 'BUT A PROVIDER REFUSAL IS STILL TERMINAL — a 500 is not offered back',
316 refused.interrupted ? 'a 500 was marked interrupted and offers a Continue that cannot work'
317 : 'no interrupted badge');
318 check(refused.errors > 0,
319 'and is reported as the error it is',
320 `${refused.errors} error lines`);
321} finally {
322 await s.close();
323}
324
325console.log(bad ? `\n${bad} check(s) FAILED` : '\nall checks passed');
326process.exit(bad ? 1 : 0);