[tamnd]

Second box of #200, moved up the list by what #201 measured.

The timing line says which phase of the pool held the time, and that is the right first question. It stops being enough as soon as the answer is a phase rather than a knob.

A pool spending 55 percent of itself extracting says extraction is the cost and does not say which of the passes inside it is. A run holding 4.5 GB resident says nothing at all about where, because the timing line does not measure memory. That second one is live right now: #201 bounded the write queue in bytes, the queue peaked at 32.9 MB of its 134.2 MB bound on a real run, and the same binary at 288 workers still climbed to 4.5 GB. The memory is somewhere the run does not report, and a heap profile is the only thing that will say where.

What this adds

--profile-dir DIR writes NAME.cpu.pprof and NAME.heap.pprof, and --profile-name NAME picks the name.

The name is the whole point rather than a convenience. go tool pprof -base first.cpu.pprof second.cpu.pprof diffs two profiles, and a diff is what a change is worth judging on, so the two runs being compared have to be told apart on disk. A single file that every run overwrites is a profile of the last thing anybody tried.

Profiling wraps the run and nothing else. Setting up the work list is a few HuggingFace requests and a Parquet open, which is not what anybody profiling a crawler is asking about, and leaving it in makes two runs differ by however long the hub took that morning.

The heap profile is written as soon as the run stops and before anything is closed, and this is not a detail: the encoders hold their buffers until a shard is sealed, so a profile taken after the sink is closed has already lost the largest thing in the run. It runs a GC first, because a heap profile is of what was reachable at the last collection and the question is what the run was holding rather than what it had not got round to freeing.

Off unless asked for. It costs a few percent and writes tens of megabytes, and no fleet unit sets it, because neither is something a unit that is up for months should be paying by default.

Tests

No directory asked for means no profile started, no files written, and an empty summary line rather than a nil dereference on the stop path.

Both files land under the name given, both are non empty, and the summary line names both so a reader of a log knows where to look.

The name is slugged, since it is going to be something like 288w-6writers typed on a command line and a stray slash would write the profile somewhere nobody is looking for it. ../../etc becomes etc and an empty name becomes run.

Next

The heap profile from a 288 worker run goes on #200 as a comment, along with whatever it says about the encoder buffers, which is the box this was moved up to unblock.