ore import held a complete copy of the repository for every commit it had read
proposal 19, opened by backlog
State done — finished
No mark closes this yet, and nothing in the forge sets that field. The review lane is what will. A mark in Ore is a point in the log, so naming one here will name a state anybody can go and render rather than a sentence somebody wrote.
Importing fe2o3 took 5.54 GB of anonymous memory and was killed by the kernel's OOM killer, taking a fourteen-hour session with it. The process was `ore`, pid 2224499; the Claude process it took down held 0.30 GB.
The cause was in `~/usr/code/rust/ore/cli/src/gitimport.rs`. `commits: HashMap<u64, Done>` kept a `Done` for every commit, and `Done.state: State` carries `seq: Sequence` -- the entire CRDT -- plus `repo: Render`, the whole rendered tree. It was kept because a merge needs both parents' states and nothing in the importer knew when the last commit that could name a given parent had gone past. Worse, `state = done.state.clone()` deep-copied the whole thing once per commit.
A wrong culprit was rejected on evidence before the right one was found. `blobs: HashMap<u64, Vec<u8>>` is the obvious accumulator and looks exactly like a leak, but fe2o3's entire uncompressed blob set is 0.11 GB, measured with `git cat-file --batch-all-objects`. Fixing it would have been the wrong fix and would have left the real one in place.
The fix is a census. The fast-export stream is read twice with identical arguments, so the marks are identical between passes; pass one counts how many commits name each mark as a parent; pass two counts back down and drops a state the moment nothing can build on it, moving rather than cloning where a parent is named for the last time. `Done.state` became `Option<State>`. `head` is kept for ever, because a tag may name a commit long after.
Measured after the fix: fe2o3 went from 5.54 GB and killed to 982 MB and completing -- 534 commits, 21,863 operations, a 66 MB log. The `ore` repository's own import was unchanged at 46 commits and 1,190 operations, identical before and after, and its peak went from 78 MB to 21 MB. Committed as `36018ce`.
The general lesson recorded alongside it is worth keeping: run heavy Ore commands under their own memory cap with `systemd-run --user --scope -p MemoryMax=4G`, because a clean exit 137 you can diagnose beats a death you cannot see. This defect was only found because the kill was visible.
Transcribed from the project record on 2026-08-14. The words quoted in the discussion below are their named authors' own; the `backlog` voice carried them here and wrote none of them.
1 reply
-
backlog
Ore session: The OOM kill turned out to be the most valuable finding of the day, because it was an Ore defect and not a memory nuisance. A wrong culprit was rejected on evidence first: blobs looks like the obvious accumulator, but fe2o3's entire uncompressed blob set is 0.11 GB, measured. Fixing it would have been the wrong fix.