oxedyne/fe2o3 — fe2o3_austenite/tools/bench/README.md

oxedyne/fe2o3/fe2o3_austenite/tools/bench/README.md

7.4 KiB, 1 run

created by r1870400018:60930, which is this file's identity for as long as the history lasts, whatever it is later renamed to

download · who wrote it · its history

1	# S0: Austenite vs Typst speed bench
2
3	The harness behind unit S0 of
4	`~/usr/code/ai/claude/notes/austenite_typesetter_tail_plan_20260923.md`. It compares
5	native Austenite against `typst compile` (`-j1` and `-j16`), and -- where Daimond's
6	vendored wasm copies exist -- Austenite wasm against typst.ts wasm in node, including
7	incremental edit latency. It writes a JSON report plus a markdown summary and refuses
8	to present numbers taken while the host was loaded unless told to anyway.
9
10	## Quick start
11
12	```bash
13	# 1. Build the austenite binary (only through this command -- see the top-level plan):
14	flock ~/.cache/cargo-targets/claude-rc-3/evslot1.lock ~/usr/code/bash/rc-build 4G -- bash -c \
15	'cd ~/.cache/austenite-bench-wt && CARGO_BUILD_JOBS=4 \
16	CARGO_TARGET_DIR=~/.cache/cargo-targets/claude-rc-3/bench \
17	cargo build --release -p oxedyne_fe2o3_austenite --bin austenite'
18
19	# 2. Run the harness (smoke mode by default -- small, fast, proves the plumbing):
20	tools/bench/run_bench.sh \
21	--austenite-bin ~/.cache/cargo-targets/claude-rc-3/bench/release/austenite
22
23	# 3. Read the report:
24	cat tools/bench/out/<timestamp>/bench_report.md
25	```
26
27	For the real baseline, on an idle host:
28
29	```bash
30	tools/bench/run_bench.sh --austenite-bin <path> --full
31	```
32
33	`--full` runs the S0 protocol as specified: 2 warm-ups, 7 measured runs, the 300-page
34	synthetic document, 50 edits at 5 positions. Smoke mode (the default) uses 1 warm-up, 3
35	runs, a 20-page synthetic document and 6 edits at 2 positions -- enough to prove every
36	leg of the harness runs and produces sane numbers, not enough to mean anything as a
37	baseline.
38
39	## What it measures, and how
40
41	- Engines: `typst compile -j 16` (default), `typst compile -j 1`, native `austenite`,
42	Austenite wasm in node, and typst.ts wasm in node -- Daimond's real compile path.
43	- Corpora: `fe2o3_austenite/samples/*.typ` plus one generated synthetic document
44	(`gen_synthetic.py`; seeded, no book content, no cetz). This is a narrower corpus than
45	the full plan's four-kind/three-size U12 set, which S0 does not depend on and did not
46	wait for -- see the brief this harness was built from.
47	- Protocol: for each (doc, engine), `WARMUPS` untimed passes then `RUNS` measured
48	passes, run round-robin across every engine for that doc (engine A, B, C, A, B, C,
49	...) rather than all of one engine then all of another, so host load hits every engine
50	equally over the course of the run. Each run is capped with `systemd-run --user
51	--scope --quiet -p MemoryMax=3G --slice=claude-rc.slice` and timed with
52	`/usr/bin/time -v`, which gives wall clock and peak RSS in one pass.
53	- Load flagging: before and after every run, the harness reads `/proc/loadavg` (the
54	1-minute average) and `some avg10` from `/proc/pressure/cpu` and `/proc/pressure/
55	memory`. A run is flagged (`flagged_under_load: true`) when the load average was
56	above `BENCH_LOAD_THRESHOLD` (default 2) on either side. `aggregate.py` refuses to
57	print a report's numbers -- it writes a "REFUSED" json/md pair instead -- when any run
58	in the batch was flagged, unless `--force` is passed, in which case the report is
59	produced but headed "TAKEN UNDER LOAD" in bold, in the markdown and as `under_load:
60	true` in the JSON.
61	- Incremental edit latency: `edit_latency.mjs` keeps one compiler instance alive
62	across a scripted run of one-character edits to five positions in the document (the
63	synthetic generator tags every paragraph with a stable `EDITTOKn` marker so a
64	position can be found and mutated by string search), timing each edit's recompile.
65	Austenite's leg calls `compileProjectDelta` with the `known`-id cache carried forward
66	edit to edit, exactly as the real watch loop does. **typst.ts's leg is a full
67	recompile to the `vector` format on every edit, not a call to typst.ts's
68	`incr_compile`/`IncrServer` API** -- checked against `www/js/typstwatch.js` and
69	`www/js/typst.js` in the Daimond app, neither of which calls that API anywhere; the
70	live-view path Daimond actually ships is `compileProjectVector`, a full recompile to
71	typst.ts's intermediate format, rendered to SVG by a second wasm module
72	(`typst_ts_renderer_bg.wasm`) through `render_svg(session, domElement)`, which needs a
73	live DOM. Timing the API Daimond does not call would not be the number a Daimond user
74	feels, so this harness times what Daimond actually does and excludes the DOM-render
75	step, which needs a browser; see "What is not measured" below.
76	- Phase attribution: `austenite --timings` is a U9 addendum that has not landed
77	(checked by `grep -q -- '--timings' src/bin/austenite.rs` in `speed_bench.sh`, not by
78	probing the binary -- `austenite`'s argument parser treats an unrecognised flag as a
79	positional path, so guessing wrong would silently corrupt the invocation rather than
80	fail loudly). Until it lands, `speed_bench.sh` runs without it and says so on stderr;
81	once it lands, the same check turns it on with no harness change needed. Typst's own
82	`--timings` and a `perf record -g` cross-check are follow-on work this unit does not
83	attempt.
84	- Peak RSS: `/usr/bin/time -v`'s "Maximum resident set size", per run for the native
85	and wasm-compile legs; for edit latency, once per whole edit session (a per-edit
86	figure inside one warm process is not a meaningful peak).
87
88	## What is not measured
89
90	- The final DOM-diff/SVG-string step of typst.ts's live view (`render_svg`), because it
91	needs an `HTMLElement`, i.e. a browser, not node. It runs on typst.ts's SIR output
92	after the compile this harness times, and by the numbers in
93	`www/js/typstwatch.js`'s own comments is the cheaper of the two steps, but it is not
94	zero and this harness does not claim to have measured it. A follow-on that drives a
95	real (headless) browser closes this gap.
96	- Typst's own `--timings` trace and the `perf record -g` cross-check the plan's §S0
97	names for phase attribution.
98	- Anything beyond `fe2o3_austenite/samples/*.typ` plus the one generated document --
99	no oxeweb TechSpec/Overview, no cetz examples, no 50/1000-page variants. Point
100	`--docs` at a wider glob once U12's corpora exist.
101
102	## Files
103
104	\| File \| Role \|
105	\|---\|---\|
106	\| `run_bench.sh` \| top-level driver: generates the synthetic doc, runs every leg, aggregates \|
107	\| `speed_bench.sh` \| native leg: typst -j16/-j1 vs austenite, interleaved, capped, timed \|
108	\| `wasm_bench.mjs` / `wasm_bench_runner.sh` \| wasm full-compile leg (node process per engine per doc, capped by the runner) \|
109	\| `edit_latency.mjs` / `edit_latency_runner.sh` \| incremental edit-latency leg \|
110	\| `gen_synthetic.py` \| deterministic synthetic `.typ` generator, any page count \|
111	\| `aggregate.py` \| merges the three legs' JSONL into `bench_report.json` + `bench_report.md`, and the load gate \|
112	\| `lib/host.sh` \| bash: `/proc/loadavg` + PSI reading, the capped-and-timed single-run helper \|
113	\| `lib/wasm_common.mjs` \| node: loads both vendored wasm compilers the way `www/js/typst.js` does, minus its use of `fetch` (Node does not resolve `file://` through `fetch`; confirmed on this host, Node v20.20.2 -- everything here reads with `fs` instead) \|
114
115	## Wasm legs are best-effort
116
117	The wasm legs run only when Daimond's vendored copies are present:
118	`~/usr/code/web/apps/oxedyne/daimond/www/vendor/austenite/` and `.../vendor/typst/`.
119	Both are hand-shipped (not built by any `cargo`/`npm` step in this repo -- see the
120	Daimond integration plan's decision on `vendor/austenite`), so their absence on a
121	fresh checkout is normal, not an error: `run_bench.sh` says so on stderr and reports
122	the native leg alone. Pass `--skip-wasm` to skip them even when present.