[tamnd]

Fourth box of #200. The heap profile in #203 left bytes.growSlice as the largest single line at 210 MB and said it was response bodies, so this is that line.

Two things were holding them.

A writer's batch was batchRows = 500 and nothing else, which is the same mistake the write queue was making one layer further out.
A count cannot bound bytes when a body runs from a kilobyte to the 10 MB cap, and 500 rows of the URL corpus is about 72 MB, so three writers sat on 217 MB of bodies waiting to reach a count.
A batch now hands over on whichever of the count or 16 MB fills first.
The count stays as the other half of the pair, because a corpus of tiny pages wants to hand over on the count rather than sit on ten thousand rows waiting to reach a byte figure.
flushBatch also clears the slice before truncating it, since the backing array was keeping a Capture alive after the row had been written, and a buffer that lets go of the rows but not the bodies flushes on schedule and frees nothing.

The bodies carried their own slack on top of that.
A bytes.Buffer grows by doubling, so a body that stops just past a doubling keeps an array up to twice its length, and that array travels into the row, into the queue and into the batch.
readBody now reserves from Content-Length where there is one, capped at 512 KB, since Content-Length is a claim by the far end and a host saying 10 MB should not be able to make 96 workers each hold ten megabytes of nothing.
A decoded body skips the hint entirely, because the number on the wire is for the compressed bytes.
A finished body is copied to its own length when the slack is at least 32 KB, and left alone below that, where the copy costs more than the slack is worth.

Measurement

server2, 96 workers, 3 writers, 15000 pages, no extraction, run A B B A so that drift on a shared box cancels.

run	build	rate	peak resident
a1	merged	55.0 pages/s	1.64 GB
b1	this branch	68.1 pages/s	0.93 GB
b2	this branch	70.0 pages/s	0.85 GB
a2	merged	67.7 pages/s	1.69 GB

a1 is the low reading of the four and the box was still settling into the run, so the honest reading of the rate is the same or a little better, not a 25 percent win.
The memory is not ambiguous: both A runs sat near 1.7 GB and both B runs under 1 GB, which is 46 percent off the peak.

Heap diff of a2 against b2, in use space:

line	a2	b2
total in use	493 MB	136 MB
bytes.growSlice	222 MB	1 MB
Parquet column buffers	65 MB	13 MB
zstd.ensureHist	64 MB	16 MB

Worker time spent queueing to write fell from 9 and 7 percent to 2 percent across the pair, which is the batch letting go of its rows sooner and the gate therefore holding less.

Tests

ccrawl/capture_batch_test.go covers the batch flushing on bytes with eight 4 MB pages, still flushing on the row count when the pages are small, and the flushed buffer letting go of the bodies in the slots past its length.
ccrawl/crawl_body_test.go covers the hint against a Content-Length, no length, a zero length, a host claiming the whole cap and a decoded body, the right sizing threshold in both directions, and readBody coming back byte for byte both with an explicit length and chunked, plus still decoding and still truncating.

Docs

docs/content/guides/recrawl-fleet.md gains a paragraph after the zstd finding, since it is the same shape of bug found by the same profile and somebody reading about CCRAWL_WRITERS should see both.