[tamnd]

This is what the profiler from #202 found the first time it was pointed at a real run, and it is the memory box of #200.

Where the memory was

A heap profile of a 96 worker run on server2, taken while the encoders still held their buffers, says 750 MB in use:

zstd.(*fastBase).ensureHist	304 MB, 41%
bytes.growSlice	136 MB, 18%
zstd.encoderOptions.encoder	98 MB, 13%
Parquet column buffers	124 MB, 16%

It is not the queue. #201 bounds that in bytes and the same run reported it peaking at 25.7 MB of its 134.2 MB bound. It is not the rows either. More than half the run's memory is zstd encoder state.

zstd holds that state per concurrent encoder. The codec asked for four, and every writer has its own codec, so the encoder count is writers times four rather than four in total. At six writers that is twenty four of them holding somewhere over a gigabyte before a single page is in a buffer, which is most of what killed the 288 worker run.

What changed

The concurrency is divided by the writer count. A run with one writer keeps exactly the four it had, since that is the only parallelism a single encoder has and nothing about that case has changed. A run with more writers already compresses in parallel across the parts, so the constant per writer was multiplying against them rather than adding to them.

The division is on the product rather than a cap, because the cost is the product: four each across six writers is the same twenty four states whatever any single writer thinks it is doing, and the machine has six cores. It floors at one, so six writers still get six encoders going at once.

Measured

server2, 96 workers, 3 writers, no extraction, 15000 pages, live URL list, run A B B A so linear drift in a shared box cancels rather than being read as a result:

run	build	rate	peak resident
a1	four each	54.3 pages a second	2.25 GB
b1	divided	52.7 pages a second	1.74 GB
b2	divided	57.2 pages a second	1.77 GB
a2	four each	53.8 pages a second	2.48 GB

26 percent less memory for the same rate. The two A peaks bracket the two B peaks with no overlap, so the ordering is not the box moving. The rate difference between the means is 1.7 percent, which is smaller than the 8 percent spread inside B on its own, so I am reading it as no change rather than as a small win.

The heap profiles of a2 and b2 say the same thing about where it went. In use 993 MB against 462 MB. ensureHist 416 MB against 64 MB. encoderOptions.encoder 130 MB against 16 MB. Both of those are the ratio of twelve encoders to three, which is what the change does and nothing else.

The run was at 3 writers rather than the 6 that OOMed. I started the comparison at 6, watched the old build reach 4.0 GB with 1.2 GB free on the box, and stopped it: I already took two of another tenant's processes down once measuring this and I am not doing it twice to confirm a number the profile already explains.

Tests

The arithmetic, including the property that matters rather than any single row of it: the total encoder count across n writers never grows past what one writer used to cost on its own.

A Parquet fanout of four opens one encoder per part, checked on the parts themselves rather than on the helper.

A single writer still opens four, so a run that was never split does not quietly get slower because of a change made for runs that were.

Not fixed by this

bytes.growSlice at 210 MB in the B profile is now the largest single line, and it is response bodies being read. That is the next thing to look at and it is not this PR.