Oregami
Repositories/oxedyne/daimond

oxedyne/daimond/dev/verify_ooxml.mjs

26.4 KiB, 1 run

created by r2519314175:555, which is this file's identity for as long as the history lasts, whatever it is later renamed to

download · who wrote it · its history

1// verify_ooxml.mjs — a THIRD instrument over the Office writers, because two
2// were not enough and one of the two has never read a Microsoft format.
3//
4// WHAT WAS ACTUALLY BEING RELIED ON. Every claim about `.docx`, `.xlsx` and
5// `.pptx` rested on LibreOffice: `soffice --headless --convert-to` round-trips a
6// file and a real reader says whether it agrees. It earned that place at once —
7// it found a `.ods` string cell written without `<text:p>` that showed as EMPTY
8// in LibreOffice while a numeric cell showed fine, so the cells hardest to notice
9// losing were exactly the ones lost. But the reader the Microsoft formats are FOR
10// is Excel, and there is no Excel and no Windows on this machine. Leading with
11// `.docx` because "most other users use MS formats, I do not, I want to cater to
12// them" put the one reader the work targets outside the test.
13//
14// An oracle also checks your OUTPUT and is silent about your INPUT. LibreOffice
15// could never have found the OpenFormula bracketing bug, because there the file
16// was right and our READER was wrong. Two instruments, two classes of defect.
17// This is a third: the SPECIFICATION as the judge, plus four readers written by
18// strangers.
19//
20// 1. the ECMA-376 4th edition TRANSITIONAL schemas — `wml.xsd`, `sml.xsd`,
21// `pml.xsd`, `dml-main.xsd` and the OPC pair — through libxml2, which is
22// the closest thing to an authority a Linux box holds;
23// 2. the OPC package graph, from ECMA-376 Part 2, checked directly: every
24// `r:id` resolves, every part has a content type, no relationship dangles,
25// no part is stranded. Excel's "we found a problem with some content" is
26// very often exactly one of these and names none of them;
27// 3. `openpyxl`, `python-docx`, `python-pptx` and `odfpy` — four codebases that
28// have never seen ours or LibreOffice's — opening the files and giving back
29// the words that went in;
30// 4. the four things ECMA-376 says about `xl/calcChain.xml`, which is the part
31// the `.xlsx` editor DELETES and which no reader on this machine reads;
32// 5. the OASIS OpenDocument RELAX NG grammars, 1.2 and 1.3, for the other three
33// formats — OpenDocument is normatively a grammar and not a schema, and
34// libxml2 validates against it, so the `.odt` / `.ods` / `.odp` side gets an
35// authority too rather than only a reader;
36// 6. three things Excel enforces that NO schema states: two tabs may not share
37// a name, `<dimension>` must cover the cells, and a `count` on the shared
38// string table means the number of references and not the number of strings.
39//
40// The name says OOXML because that was the hole. It grew the OpenDocument half
41// because the same instrument reached, and (5) found two defects in ten minutes.
42//
43// THE CONTROL SET IS THE POINT. Every rule is also run over `rich.docx`,
44// `foreign.xlsx` and `foreign.pptx`, which LibreOffice wrote and we did not, so
45// the instrument's own false positives are visible instead of being argued about.
46// Two survive after ECMA-376 Part 3 markup-compatibility preprocessing, both
47// places where the ECMA schemas disagree with every real file, and both named in
48// `ooxml_check.py`. A finding that the control set also produces is reported as
49// noise. Anything our output produces and the control does not is ours.
50//
51// AND THE INSTRUMENT IS PROVED TO GO RED. `--selftest` damages a good package in
52// five known ways — a part loses its content type, an `r:id` points at nothing, a
53// relationship target leaves the package, a cell a chain names loses its formula,
54// an attribute takes a value its type forbids — and reports whether the rule that
55// should catch each one did. A rule that cannot fail is not a check.
56//
57// WHAT THIS CANNOT DO. It cannot tell you Excel opens the file. Schema validity
58// is necessary and not sufficient: Excel refuses things the schema permits — the
59// `styles.xml` fill table is in the writer for exactly that reason — and repairs
60// things the schema forbids. The one honest answer to "does Excel open it" is to
61// open one in Excel.
62//
63// Run: node dev/verify_ooxml.mjs [--selftest] [--keep]
64
65import fs from 'node:fs';
66import os from 'node:os';
67import path from 'node:path';
68import { spawnSync } from 'node:child_process';
69
70const HERE = path.dirname(new URL(import.meta.url).pathname);
71const ROOT = path.resolve(HERE, '..');
72const CACHE = path.join(os.homedir(), '.cache', 'daimond', 'laneE');
73const OUT = path.join(CACHE, 'out');
74const XSD = path.join(CACHE, 'schemas', 'xsd');
75const PYLIB = path.join(CACHE, 'pylibs');
76const CHECK = path.join(HERE, 'fixtures', 'ooxml', 'ooxml_check.py');
77const DATA = path.resolve(ROOT, '../../../../rust/fe2o3/fe2o3_file/tests/data');
78
79const args = process.argv.slice(2);
80
81let pass = 0, fail = 0, noted = 0;
82const check = (ok, name, detail) => {
83 if (ok) { pass++; console.log(' ok ' + name); }
84 else { fail++; console.log(' FAIL ' + name + (detail ? '\n ' + detail : '')); }
85};
86const note = (name, detail) => {
87 noted++;
88 console.log(' note ' + name + (detail ? '\n ' + detail : ''));
89};
90
91// ---------------------------------------------------------------------------
92// The instruments have to BE there, and the run says so before anything else
93// ---------------------------------------------------------------------------
94
95// NEITHER OF THESE LIVES IN THE REPO. The ECMA schemas are a 46 MB download and
96// the Python readers are four packages, so both sit in `~/.cache/daimond/laneE/`
97// and `--fetch` puts them there. That makes a missing instrument the most
98// dangerous state this file can be in: a run that silently skipped the schemas
99// and the second readers would print a page of `ok` and mean nothing, which is
100// how a green suite comes to certify a format nobody validated. So a missing
101// instrument is a FAILURE with the exact command to fix it, never a skip.
102//
103// WHAT `--fetch` DOWNLOADS, EXACTLY:
104//
105// * https://ecma-international.org/wp-content/uploads/ECMA-376_4th_edition_december_2012.zip
106// (46,635,356 bytes) — ECMA's own publication. Part 4 of it holds
107// `OfficeOpenXML-XMLSchema-Transitional.zip`, which is the schema set for the
108// `schemas.openxmlformats.org/...2006/...` namespaces every file here uses.
109// The STRICT set shipped with Part 1 is the wrong one: it carries the
110// `purl.oclc.org` namespaces and matches nothing we write.
111// * https://ecma-international.org/wp-content/uploads/ECMA-376-2_5th_edition_december_2021.zip
112// (1,899,204 bytes) — Part 2, for `opc-contentTypes.xsd` and
113// `opc-relationships.xsd`.
114// * https://www.w3.org/2001/xml.xsd — because ECMA's `wml.xsd` declares
115// `<xsd:import namespace=".../XML/1998/namespace"/>` with NO `schemaLocation`,
116// and libxml2 will not resolve `xml:space` without one. `--fetch` adds the
117// location to a copy. That is the only edit made to any ECMA file, and it adds
118// nothing to the schema's meaning.
119// * the OASIS OpenDocument RELAX NG grammars, 1.2 and 1.3, from
120// docs.oasis-open.org. BOTH, because OpenDocument is normatively a RELAX NG
121// grammar rather than an XML Schema and the version is chosen by what the file
122// declares: LibreOffice still writes 1.2, and validating a 1.2 file against the
123// 1.3 grammar reports the version attribute and then cascades through the
124// interleave — dozens of findings about nothing. Picking by `office:version`
125// took the control set from 40 errors to none.
126// * openpyxl, python-docx, python-pptx and odfpy from PyPI, into
127// `~/.cache/daimond/laneE/pylibs` with `pip3 install --target`. Nothing is
128// installed into the system or into the project.
129
130const NEEDED = ['transitional/sml.xsd', 'transitional/wml.xsd',
131 'transitional/pml.xsd', 'transitional/dml-main.xsd', 'transitional/xml.xsd',
132 'opc/opc-contentTypes.xsd', 'opc/opc-relationships.xsd',
133 '../rng/OpenDocument-v1.2-schema.rng', '../rng/OpenDocument-v1.3-schema.rng',
134 '../rng/OpenDocument-v1.2-manifest-schema.rng',
135 '../rng/OpenDocument-v1.3-manifest-schema.rng'];
136const missing = NEEDED.filter((f) => !fs.existsSync(path.join(XSD, f)));
137const readers = spawnSync('python3', ['-c',
138 'import openpyxl, docx, pptx, odf, lxml; print("all four")'],
139 { env: { ...process.env, PYTHONPATH: PYLIB }, encoding: 'utf8' });
140
141if (args.includes('--fetch')) {
142 const sh = (cmd) => {
143 console.log(' $ ' + cmd.replace(/\s+/g, ' ').slice(0, 110));
144 const r = spawnSync('bash', ['-c', cmd], { stdio: 'inherit' });
145 if (r.status !== 0) { console.log(' FAILED'); process.exit(1); }
146 };
147 const S = path.join(CACHE, 'schemas');
148 sh(`mkdir -p ${S}/x ${XSD}`);
149 sh(`cd ${S} && curl -sSf --max-time 600 -o ecma376-4e.zip `
150 + `https://ecma-international.org/wp-content/uploads/ECMA-376_4th_edition_december_2012.zip`);
151 sh(`cd ${S} && curl -sSf --max-time 300 -o ecma376-2-5e.zip `
152 + `https://ecma-international.org/wp-content/uploads/ECMA-376-2_5th_edition_december_2021.zip`);
153 sh(`cd ${S}/x && unzip -oq ../ecma376-4e.zip && unzip -oq ../ecma376-2-5e.zip `
154 + `&& unzip -oq "ECMA-376, Fourth Edition, Part 4 - Transitional Migration Features.zip" `
155 + `&& unzip -oq OfficeOpenXML-XMLSchema-Transitional.zip -d ${XSD}/transitional `
156 + `&& unzip -oq OpenPackagingConventions-XMLSchema.zip -d ${XSD}/opc`);
157 sh(`chmod -R u+w ${XSD}`);
158 sh(`curl -sSf --max-time 120 -o ${XSD}/transitional/xml.xsd https://www.w3.org/2001/xml.xsd`);
159 sh(`python3 - <<'PY'
160p = '${XSD}/transitional/wml.xsd'
161s = open(p, encoding='utf-8').read()
162old = '<xsd:import namespace="http://www.w3.org/XML/1998/namespace"/>'
163new = '<xsd:import namespace="http://www.w3.org/XML/1998/namespace" schemaLocation="xml.xsd"/>'
164open(p, 'w', encoding='utf-8').write(s.replace(old, new))
165print('wml.xsd: xml namespace import given a schemaLocation')
166PY`);
167 sh(`mkdir -p ${S}/rng`);
168 for (const [v, leaf] of [
169 ['v1.3/os', 'OpenDocument-v1.3-schema.rng'],
170 ['v1.3/os', 'OpenDocument-v1.3-manifest-schema.rng']]) {
171 sh(`curl -sSf --max-time 240 -o ${S}/rng/${leaf} `
172 + `https://docs.oasis-open.org/office/OpenDocument/${v}/schemas/${leaf}`);
173 }
174 for (const [remote, leaf] of [
175 ['OpenDocument-v1.2-os-schema.rng', 'OpenDocument-v1.2-schema.rng'],
176 ['OpenDocument-v1.2-os-manifest-schema.rng',
177 'OpenDocument-v1.2-manifest-schema.rng']]) {
178 sh(`curl -sSf --max-time 240 -o ${S}/rng/${leaf} `
179 + `https://docs.oasis-open.org/office/v1.2/os/${remote}`);
180 }
181 sh(`pip3 install -q --disable-pip-version-check --target=${PYLIB} `
182 + `openpyxl python-docx python-pptx odfpy`);
183 console.log('\ninstruments fetched; run again without --fetch');
184 process.exit(0);
185}
186
187if (missing.length || readers.status !== 0) {
188 console.log('\n FAIL the instruments are not installed, so this run would '
189 + 'mean nothing');
190 if (missing.length) {
191 console.log(' no schemas at ' + XSD + ': missing ' + missing.join(', '));
192 }
193 if (readers.status !== 0) {
194 console.log(' no independent readers at ' + PYLIB + ': '
195 + (readers.stderr || '').trim().split('\n').pop());
196 }
197 console.log('\n node dev/verify_ooxml.mjs --fetch\n');
198 console.log(' downloads the ECMA-376 schemas and the four Python readers '
199 + 'into\n ~/.cache/daimond/laneE/. See the header for exactly what and '
200 + 'from where.');
201 process.exit(1);
202}
203console.log('instruments: ECMA-376 4th edition transitional schemas, '
204 + readers.stdout.trim() + ' independent readers, LibreOffice not consulted');
205
206// ---------------------------------------------------------------------------
207// The wasm, loaded straight into node
208// ---------------------------------------------------------------------------
209
210// The bundle the app ships, not a rebuild of it: `wasm-bindgen`'s glue takes the
211// module bytes as an argument, so nothing here needs a browser, a profile or a
212// server. It is also the reason this verifier is seconds rather than minutes —
213// and the reason it must NOT build: another lane owns `www/pkg`.
214const PKG = path.join(ROOT, 'www', 'pkg');
215if (!fs.existsSync(path.join(PKG, 'oxedyne_daimond_bg.wasm'))) {
216 console.log(' FAIL www/pkg holds no wasm bundle; nothing to test');
217 process.exit(1);
218}
219const wasm = await import(path.join(PKG, 'oxedyne_daimond.js'));
220await wasm.default({ module_or_path: fs.readFileSync(
221 path.join(PKG, 'oxedyne_daimond_bg.wasm')) });
222
223// ---------------------------------------------------------------------------
224// The specimens
225// ---------------------------------------------------------------------------
226
227fs.mkdirSync(OUT, { recursive: true });
228for (const f of fs.readdirSync(OUT)) fs.rmSync(path.join(OUT, f), { force: true });
229
230/// Markdown with one of everything a writer has to carry: two heading levels,
231/// emphasis, a link, both kinds of list, a quotation, code, and a table — which
232/// is what `Book::from_doc` turns into a sheet and `Deck::from_doc` into slides.
233const MD = [
234 '# Quarterly Review',
235 '',
236 'The **margin** held at *11 per cent*, and the [note](https://example.org/n)',
237 'says why. A string of deliberate spaces: and a value of 3.50 that is not a',
238 'number.',
239 '',
240 '## Findings',
241 '',
242 '- Sales rose',
243 '- Costs held',
244 ' - Freight fell',
245 '',
246 '1. Reprice',
247 '2. Restock',
248 '',
249 '> A quotation, kept as one.',
250 '',
251 '`inline code` and a block:',
252 '',
253 '```',
254 'let x = 1;',
255 '```',
256 '',
257 '| Region | Units | Price | Total |',
258 '| --- | --- | --- | --- |',
259 '| North | 120 | 3.4 | =B2*C2 |',
260 '| South | 85 | 11 | =B3*C3 |',
261 '',
262 '## Notes',
263 '',
264 'Nothing further.',
265 '',
266].join('\n');
267
268const specimens = [];
269const put = (name, bytes) => {
270 const p = path.join(OUT, name);
271 fs.writeFileSync(p, Buffer.from(bytes));
272 specimens.push(p);
273 return p;
274};
275
276console.log('\nwritten from Markdown');
277for (const [media, ext] of [['Docx', 'docx'], ['Xlsx', 'xlsx'], ['Pptx', 'pptx'],
278 ['Odt', 'odt'], ['Ods', 'ods'], ['Odp', 'odp']]) {
279 try {
280 const bytes = wasm.office_write(MD, media);
281 put('written.' + ext, bytes);
282 check(bytes.length > 0, `office_write(${media}) produced ${bytes.length} bytes`);
283 } catch (e) {
284 check(false, `office_write(${media})`, String(e && e.message ? e.message : e));
285 }
286}
287try {
288 const bytes = wasm.office_write_docx(MD);
289 put('legacy.docx', bytes);
290 check(bytes.length > 0, `office_write_docx produced ${bytes.length} bytes`);
291} catch (e) {
292 check(false, 'office_write_docx', String(e));
293}
294
295/// Two tables under headings that reduce to ONE tab name. `Book::from_doc` names a
296/// sheet by the heading above its table and `xlsx::write::sheet_name` strips the
297/// characters Excel refuses and truncates at 31, and neither step looks at the
298/// names already handed out. Excel refuses a workbook with two tabs of one name,
299/// and refuses the FILE rather than the name.
300const CLASH = [
301 '## Q1/Q2',
302 '',
303 '| A | B |',
304 '| --- | --- |',
305 '| 1 | 2 |',
306 '',
307 '## Q1:Q2',
308 '',
309 '| A | B |',
310 '| --- | --- |',
311 '| 3 | 4 |',
312 '',
313 '## A very long heading that runs past the thirty-one character limit, one',
314 '',
315 '| A | B |',
316 '| --- | --- |',
317 '| 5 | 6 |',
318 '',
319 '## A very long heading that runs past the thirty-one character limit, two',
320 '',
321 '| A | B |',
322 '| --- | --- |',
323 '| 7 | 8 |',
324 '',
325].join('\n');
326try {
327 const bytes = wasm.office_write(CLASH, 'Xlsx');
328 put('clash.xlsx', bytes);
329 check(bytes.length > 0, `a workbook from four clashing headings — ${bytes.length} bytes`);
330} catch (e) {
331 check(false, 'office_write(Xlsx) on clashing headings', String(e));
332}
333
334// ---------------------------------------------------------------------------
335// The edits, on files this project did not write
336// ---------------------------------------------------------------------------
337
338const read = (p) => new Uint8Array(fs.readFileSync(p));
339
340console.log('\nedited, on somebody else\'s bytes');
341
342const edits = [
343 // A surgical document edit. The rest of the archive must survive it.
344 ['edited.docx', () => wasm.office_edit_doc(read(path.join(DATA, 'rich.docx')),
345 'Docx', JSON.stringify([{ find: 'Findings', replace: 'Conclusions' }]))],
346 // A plain value into an empty cell of a sheet that has no calculation chain.
347 ['cell.xlsx', () => wasm.office_edit_sheet(read(path.join(DATA, 'foreign.xlsx')),
348 'Xlsx', JSON.stringify([{ sheet: 'Sales', ref: 'B4', value: '42' }]))],
349 // A FORMULA into a workbook that HAS a chain. The chain must go.
350 ['chain-formula.xlsx', () => wasm.office_edit_sheet(
351 read(path.join(HERE, 'fixtures', 'ooxml', 'withchain.xlsx')),
352 'Xlsx', JSON.stringify([{ sheet: 'Sales', ref: 'D4', formula: '=B2+B3' }]))],
353 // A PLAIN VALUE over a cell that HELD a formula and IS in the chain. The
354 // chain is now wrong in the other direction, and this is the case the
355 // writer's own reasoning does not cover.
356 ['chain-plainover.xlsx', () => wasm.office_edit_sheet(
357 read(path.join(HERE, 'fixtures', 'ooxml', 'withchain.xlsx')),
358 'Xlsx', JSON.stringify([{ sheet: 'Sales', ref: 'D2', value: '999' }]))],
359 // A cell far outside the declared dimension, which must widen.
360 ['grown.xlsx', () => wasm.office_edit_sheet(read(path.join(DATA, 'foreign.xlsx')),
361 'Xlsx', JSON.stringify([{ sheet: 'Sales', ref: 'K40', value: 'edge' }]))],
362 // A string that is nothing but spaces, and one that looks like a number.
363 ['spaces.xlsx', () => wasm.office_edit_sheet(read(path.join(DATA, 'foreign.xlsx')),
364 'Xlsx', JSON.stringify([
365 { sheet: 'Sales', ref: 'A6', value: ' ' },
366 { sheet: 'Sales', ref: 'B6', value: '3.50' },
367 { sheet: 'Notes', ref: 'A1', value: 'moved' }]))],
368];
369
370for (const [name, run] of edits) {
371 try {
372 const bytes = run();
373 put(name, bytes);
374 check(bytes.length > 0, `${name} — ${bytes.length} bytes`);
375 } catch (e) {
376 check(false, name, String(e && e.message ? e.message : e));
377 }
378}
379
380// ---------------------------------------------------------------------------
381// The control set: the same rules over files nobody here wrote
382// ---------------------------------------------------------------------------
383
384const controls = ['rich.docx', 'loffice.docx', 'withpic.docx', 'foreign.xlsx',
385 'foreign.pptx', 'foreign.odt', 'foreign.ods', 'foreign.odp']
386 .map((n) => path.join(DATA, n))
387 .filter((p) => fs.existsSync(p));
388
389const WORDS = {
390 'written.docx': ['Quarterly Review', 'Findings', 'Sales rose', 'Reprice'],
391 'legacy.docx': ['Quarterly Review', 'Findings'],
392 'written.xlsx': ['North', 'South'],
393 'written.pptx': ['Quarterly Review', 'Findings'],
394 'written.odt': ['Quarterly Review', 'Sales rose'],
395 'written.ods': ['North'],
396 'written.odp': ['Findings'],
397 'edited.docx': ['Conclusions'],
398 'cell.xlsx': ['42'],
399 'grown.xlsx': ['edge'],
400 'spaces.xlsx': ['3.50', 'moved'],
401};
402
403const run = (files, extra = []) => {
404 const r = spawnSync('python3', [CHECK, '--schemas', XSD,
405 '--expect', JSON.stringify(WORDS), ...extra, ...files],
406 { env: { ...process.env, PYTHONPATH: PYLIB },
407 maxBuffer: 64 * 1024 * 1024, encoding: 'utf8' });
408 if (r.status !== 0) {
409 console.log(' FAIL the checker did not run\n' + (r.stderr || '').slice(0, 2000));
410 process.exit(1);
411 }
412 return JSON.parse(r.stdout);
413};
414
415console.log('\nthe instrument, over the control set (LibreOffice\'s output, not ours)');
416const ctl = run(controls);
417/// A finding reduced to what it is ABOUT, so the same divergence in two files is
418/// one entry: the message with every number, quoted value and part name taken out.
419const shape = (f) => f.rule + ' :: ' + f.detail
420 .replace(/line \d+: /, '')
421 .replace(/'[^']*'/g, "'…'")
422 .replace(/\d+/g, 'N')
423 .slice(0, 160);
424
425const noise = new Set();
426for (const file of ctl.files) {
427 for (const f of file.findings) {
428 if (f.severity === 'error' || f.severity === 'known') noise.add(shape(f));
429 }
430}
431let ctlErrors = 0, ctlKnown = 0;
432for (const file of ctl.files) {
433 for (const f of file.findings) {
434 if (f.severity === 'error') ctlErrors++;
435 if (f.severity === 'known') ctlKnown++;
436 }
437}
438check(ctlErrors === 0,
439 `the control set produces ${ctlErrors} unexplained errors and ${ctlKnown} `
440 + `known schema divergences over ${ctl.files.length} files`,
441 ctlErrors === 0 ? '' : 'every one of these is the INSTRUMENT, not the code — '
442 + 'triage them before trusting a finding below');
443if (ctlErrors > 0) {
444 for (const file of ctl.files) {
445 for (const f of file.findings) {
446 if (f.severity === 'error') {
447 note('control ' + file.name + ' ' + f.rule, f.part + ': ' + f.detail.slice(0, 220));
448 }
449 }
450 }
451}
452
453// ---------------------------------------------------------------------------
454// And over ours
455// ---------------------------------------------------------------------------
456
457console.log('\nthe instrument, over ours');
458const ours = run(specimens);
459
460let realErrors = 0;
461for (const file of ours.files) {
462 const errs = file.findings.filter((f) => f.severity === 'error');
463 const shared = errs.filter((f) => noise.has(shape(f)));
464 const mine = errs.filter((f) => !noise.has(shape(f)));
465 const warns = file.findings.filter((f) => f.severity === 'warn');
466 const known = file.findings.filter((f) => f.severity === 'known');
467 realErrors += mine.length;
468 check(mine.length === 0,
469 `${file.name} — ${file.parts.length} parts, ${mine.length} findings `
470 + `(${shared.length} also in the control set, ${known.length} known `
471 + `schema divergences, ${warns.length} warnings)`);
472 for (const f of mine) {
473 console.log(' ' + f.rule + ' | ' + (f.part || '(package)')
474 + '\n ' + f.detail);
475 }
476 for (const f of warns) note(file.name + ' ' + f.rule, f.part + ': ' + f.detail);
477}
478
479// The readers' own verdicts, which are worth printing whether or not they failed:
480// "openpyxl opened it" is the sentence this lane exists to be able to say.
481console.log('\nwhat the independent readers said');
482for (const file of ours.files) {
483 for (const f of file.findings) {
484 if (f.rule.startsWith('reader.') && f.severity === 'info') {
485 console.log(' ' + file.name.padEnd(24) + f.rule.replace('reader.', '')
486 + ': ' + f.detail);
487 }
488 }
489}
490
491// ---------------------------------------------------------------------------
492// "Surgical means the original bytes survive" — checked rather than believed
493// ---------------------------------------------------------------------------
494
495// The contract says an edit "rewrites the runs it was asked to change and leaves
496// every other part of the archive byte-identical". That is a claim about bytes,
497// and bytes can be compared. Nothing else in the suite compares them: LibreOffice
498// re-serialises everything it opens, so an oracle that round-trips a file cannot
499// tell a surgical edit from a full rewrite that happens to land on the same words.
500console.log('\nthe surgical claim, part by part');
501const digests = (p) => {
502 const r = spawnSync('python3', ['-c', `
503import hashlib,json,sys,zipfile
504z=zipfile.ZipFile(sys.argv[1])
505print(json.dumps([[n, hashlib.sha256(z.read(n)).hexdigest()] for n in z.namelist()]))
506`, p], { encoding: 'utf8' });
507 return JSON.parse(r.stdout);
508};
509
510for (const [src, out, expect] of [
511 [path.join(DATA, 'rich.docx'), 'edited.docx', ['word/document.xml']],
512 [path.join(DATA, 'foreign.xlsx'), 'cell.xlsx', ['xl/worksheets/sheet1.xml']],
513 [path.join(DATA, 'foreign.xlsx'), 'grown.xlsx', ['xl/worksheets/sheet1.xml']],
514 [path.join(DATA, 'foreign.xlsx'), 'spaces.xlsx',
515 ['xl/worksheets/sheet1.xml', 'xl/worksheets/sheet2.xml']]]) {
516 const p = path.join(OUT, out);
517 if (!fs.existsSync(p)) continue;
518 const before = digests(src), after = digests(p);
519 const bn = before.map(([n]) => n), an = after.map(([n]) => n);
520 check(JSON.stringify(bn) === JSON.stringify(an),
521 `${out} keeps every part of the original, in the original order`,
522 'added: ' + an.filter((n) => !bn.includes(n)).join(', ')
523 + ' | lost: ' + bn.filter((n) => !an.includes(n)).join(', '));
524 const changed = before
525 .filter(([n, h], i) => after[i] && after[i][0] === n && after[i][1] !== h)
526 .map(([n]) => n);
527 check(JSON.stringify(changed.sort()) === JSON.stringify([...expect].sort()),
528 `${out} changed exactly ${expect.join(', ')} and nothing else`,
529 'changed: ' + (changed.join(', ') || '(nothing)'));
530}
531
532// ---------------------------------------------------------------------------
533// The calculation chain, by itself, because it is the untested decision
534// ---------------------------------------------------------------------------
535
536console.log('\nthe calculation chain');
537const held = (p, name) => {
538 const r = spawnSync('unzip', ['-l', p], { encoding: 'utf8' });
539 return (r.stdout || '').includes(name);
540};
541const declares = (p, needle) => {
542 const r = spawnSync('unzip', ['-p', p, '[Content_Types].xml'], { encoding: 'utf8' });
543 const s = spawnSync('unzip', ['-p', p, 'xl/_rels/workbook.xml.rels'], { encoding: 'utf8' });
544 return (r.stdout || '').includes(needle) || (s.stdout || '').includes(needle);
545};
546
547const FIX = path.join(HERE, 'fixtures', 'ooxml', 'withchain.xlsx');
548check(held(FIX, 'calcChain.xml'),
549 'the fixture HAS a calculation chain, so the removal path is reachable at all',
550 'without this the whole calcChain decision is untested by construction — no '
551 + 'file in fe2o3_file/tests/data has one, because LibreOffice never writes it');
552
553const formulaOut = path.join(OUT, 'chain-formula.xlsx');
554if (fs.existsSync(formulaOut)) {
555 check(!held(formulaOut, 'calcChain.xml'),
556 'writing a FORMULA removed xl/calcChain.xml');
557 check(!declares(formulaOut, 'calcChain'),
558 'and removed the content-type override and the relationship with it',
559 'a part gone while [Content_Types].xml still names it is a package Excel '
560 + 'refuses outright');
561}
562
563const plainOut = path.join(OUT, 'chain-plainover.xlsx');
564if (fs.existsSync(plainOut)) {
565 const gone = !held(plainOut, 'calcChain.xml');
566 check(gone,
567 'writing a PLAIN VALUE over a cell that HELD a formula also removed the chain',
568 'it did not. D2 held <f>B2*C2</f> and is named by xl/calcChain.xml; the '
569 + 'edit replaced the cell, so the formula is gone and the chain still '
570 + 'names it. ECMA-376 §18.6.1: a c in the chain is "a single cell, which '
571 + 'shall contain a formula". fe2o3_file/src/office/xlsx/edit.rs:155 sets '
572 + 'the drop flag from `s.formula.is_some()`, which is the formulas being '
573 + 'WRITTEN and not the formulas being DESTROYED.');
574}
575
576// ---------------------------------------------------------------------------
577// Prove the instrument can fail
578// ---------------------------------------------------------------------------
579
580// Always, not behind a flag. A verifier that only proves it can go red when asked
581// is one whose rules quietly stop working between the days somebody asks.
582console.log('\nthe instrument, damaged on purpose');
583const st = run([], ['--selftest', FIX]);
584for (const c of st.selftest) {
585 check(c.went_red, `breaking a package the ${c.expected} way turns ${c.expected} red`,
586 'it stayed green; the rule cannot fail and is therefore not a check. '
587 + 'What did fire: ' + (c.errors.join(', ') || 'nothing'));
588}
589
590// ---------------------------------------------------------------------------
591
592console.log(`\n${pass} passed, ${fail} failed, ${noted} noted`);
593console.log(`specimens in ${OUT}`);
594process.exit(fail === 0 ? 0 : 1);