Milestone: recrawl-fleet

Recrawl the two corpora we already publish, on a three server fleet, and publish what comes back as datasets that are usable while the run is still going.

The goal

Three times the current domain corpus and three times one month of URLs, at 250 pages a second per server.

	source	rows	3x
domains	`open-index/ccrawl-domains`, cc-main-2026-apr-may-jun	121,091,933	363,275,799
urls	`open-index/ccrawl-urls`, CC-MAIN-2026-25	2,098,491,742	6,295,475,226

6.66B pages. At 750 a second across the fleet that is 64.8M a day and about 103 days, so the run is measured in months and the publish has to be incremental from the first shard.

Output goes to two repos, open-index/ccrawl-recrawl-domains and open-index/ccrawl-recrawl-urls, each with its own card, ledger and schedule.

Checklist

What I measured before writing this

Numbers first, because the interesting result is that the obvious bottleneck is not the bottleneck.

crawl run against a local server, all production defaults, 200 distinct hosts: 140 pages a second. My first read of that was an engine ceiling and it was wrong. Every URL in my first benchmark was on one host, so the one second per host delay was serialising the whole thing and doing exactly its job. With 200 hosts and the delay out of the way the same binary sustained 8234 pages a second at 32 workers, and 140 a second with the delay on is just 200 hosts divided by one second. Throughput on a real crawl is set by how many distinct hosts are eligible at once, and the engine has around thirty times the headroom the 250 per server target needs.

Worker count past that point hurts rather than helps, 8234 a second at 32 workers falling to 4333 at 128, which is contention on the frontier's single lock and only shows up because loopback latency is a millisecond. On the open web it will not.

TestFrontierScale at 5M URLs is where the architecture turns:

admit  2000000  117520/s
admit  4000000   70292/s
admit  5000000   32684/s
database 0.63 GB for 5000000 URLs
pop    5000000    6181/s

135 bytes per URL means one server's share of the URL recrawl, 2.1B URLs, is about 283 GB of SQLite before anything is fetched. server2 has 2.8 GB free and server3 has 6.6 GB. The admit rate also falls by a factor of four between 2M and 5M, so even given the disk the load alone would take most of a day and keep slowing down.

That is not a frontier bug. A discovery crawl needs a persistent queue because the work list is not known in advance, outlinks arrive as you go, and a crash has to be recoverable. A recrawl has none of those properties, since the work list is already written down and deduplicated in the datasets we publish. So the recrawl streams from Parquet and keeps a checkpoint, and crawl run keeps its frontier unchanged. That is #161, and it is the decision the rest of the milestone hangs off.

On reusing ami

ami solves the concurrency problem already and there is no reason to solve it twice. Worth taking: the transport shard pools, the worker pool shape, the per host caps, the rotated Parquet capture writer with inline zstd bodies, conditional refetch through digest and etag, and shard partitioning for distributed runs.

Worth not taking: ami does not read robots.txt at all, and it shards on a hash of the URL. Those two are fine together and neither is fine here. If we shard by URL then one host lands on all three servers, each server waits its own second, and the site sees three requests a second while every server believes it is being polite. So the partition key is the registered domain, which keeps a host and its politeness clock on one machine. That is #160, and #164 is the rest of it.

Bugs found while measuring

--delay 0 cannot mean zero. cli/crawl_cmd.go only copies the flag when it is positive and NewCrawler puts the default back when it is not, so no flag and an explicit zero are the same input. The crawler also borrows App.HTTP, the Common Crawl client, so an open web crawl draws on a rate budget that exists to be polite to data.commoncrawl.org. Both in #162.

Process

Every PR that lands against this milestone gets a comment on this issue with the PR number and one line on what changed, and the matching checklist box is ticked when the linked issue closes. Automated by .github/workflows/milestone-tracker.yml.

Merged #169, Give the crawler its own transport and let the delay be zero.

Closes:

Let the crawl delay be zero and stop borrowing the Common Crawl transport #162 Let the crawl delay be zero and stop borrowing the Common Crawl transport

#169 is in, and #170 went in ahead of it to unblock CI.

What #169 changed

--delay 0 now means zero. The flag carries its own default of one second, so unset and explicit zero stopped being the same input, and NewCrawler no longer puts the default back over a zero. Workers and RetryDelay still get re-defaulted, since nobody can have meant zero workers.

The bigger half is that a crawl no longer borrows the Common Crawl client. Page fetches went through CrawlURL and never touched it, which is why this looked harmless, but robots.txt did, and that client's delay is per process rather than per host. Robots was fetched at five hosts a second whatever the worker count.

That was the entire cost of a crawl rather than a share of it:

2000 pages, 200 local hosts
robots on	40.3 s
robots off	under 1 s

200 hosts times 200ms is exactly the 40 seconds. Against the 121,091,933 domains in the recrawl corpus that is 280 days of reading robots.txt before a single page counts, so #164 would have had nothing to stand on until this landed.

Crawls now get NewCrawlClient, with no process-wide delay, no shared budget, and a transport shaped for visiting many hosts once each rather than one host many times. Politeness stays per host, in the frontier and in waitForHost.

A detour worth recording

TestBatchFetchResumes failed on macOS CI on a branch that cannot reach it, reporting 5 requests against 4 wanted while emitting exactly the right 4 records. The count was wrong, not the resume. Stopping the first run cancels a fetch that is already in flight, and while RunBatchFetch waits for its own workers, nothing waits for the server: the handler that request already reached can increment the counter after the test has reset it for the second run.

#170 counts distinct spans instead, so a straggler folds into a span the next run asks for anyway while a genuine refetch still shows up as two spans nobody else asks for. Confirmed by breaking the resume on purpose, which reports 6 spans against the 4 wanted. Using a WaitGroup to wait the handlers out is the same bug again, since the Add lands inside the handler and a request arriving during the Wait is a positive delta from zero.

Next

#160, sharding by registered domain, since #161 and #164 both need the partition key settled before they mean anything.

Merged #171, Shard a crawl by registered domain so a fleet can split one seed list.

Closes:

Shard a work list by registered domain #160 Shard a work list by registered domain

R1 is in, #171 merged, #160 closed.

crawl run --shard N --shards M splits one seed list across machines that never talk to each other. All three servers read the same file, each keeps the partition it was given, and nothing coordinates.

The key is the registered domain, not the URL. This is the decision the rest of the milestone rests on so it is worth restating now that it is real code. A crawler keeps one politeness clock per host and that clock only means anything if the host belongs to one process. Hash the URL, which is what ami does, and a busy site's pages scatter across all three servers, each waiting its own second while the site sees three requests a second and every server believes it is behaving. ami can do that because it never reads robots.txt and keeps no per host clock, so splitting a host costs it nothing. It would cost us the only guarantee we make.

The filter runs as the seed file is read, in loadSeeds, not at fetch time. A URL a machine does not own never reaches its frontier, so three servers keep a third of the state each rather than three full copies. Given server2 has 2.8 GB free and server3 has 6.6 GB, a third of a frontier is the difference between fitting and not, and that is the same disk pressure that pushed R2 towards streaming in the first place.

Evenness, measured on the real release rather than synthetic hosts. Synthetic names would have proved nothing here: h0.example through h2000000.example are uniform by construction, so any hash at all splits them evenly, and the failure mode worth ruling out is exactly what they cannot show, which is real domains clumping under a handful of public suffixes. TestShardEvennessOnRealDomains pulls the released domain table part by part, counts through the same ShardKey the seed loader calls, and deletes each part as soon as it is counted so the whole thing costs one 105 MB part of disk at a time.

Over all 121 091 933 domains:

shards	worst shard off even
3	0.005%
7	0.030%
32	0.159%

Three shards is 40 365 223, 40 361 867 and 40 364 843 domains. A spread of 3356 across forty million each is even enough that the fleet plan can stop thinking about it.

Stability is pinned with fixed vectors, not a property test. A hash swapped for another hash is still a function and a property test would pass straight through the swap. Nine URLs are pinned, and if the assignment moves, the servers stop agreeing with each other and the test says so.

What this unblocks. R2 and R5 both needed the partition key settled before they meant anything. R2 can now checkpoint a shard by row offset knowing which rows the shard owns, and R5 can size a robots cache against a third of the corpus rather than all of it, since a registered domain and its robots.txt now live on the same machine for the life of the run.

Next is #161, streaming the work list. The frontier numbers from earlier in this milestone are the argument for it: 2.1 billion URLs would be about 283 GB of frontier against 2.8 GB free on server2, and the admit rate already falls fourfold between 2 million and 5 million URLs. Sharding cuts that by three, which is not remotely enough, so the URL recrawl has to stream from published Parquet with a shard and row offset checkpoint instead of materialising a queue. crawl run keeps its frontier for discovery work, where the queue is the point.

Merged #172, Stream the recrawl work list instead of building a frontier.

Closes:

Stream the recrawl work list instead of materialising a frontier #161 Stream the recrawl work list instead of materialising a frontier

#161 is in, #172 merged. ccrawl recrawl run streams a published dataset and fetches it, and the frontier is out of the recrawl path.

The case for it was made earlier in this milestone and it held up. A frontier is worth having on a discovery crawl, where the queue is not known ahead of time and outlinks arrive as you go. A recrawl has none of that: the list is published, deduplicated and sorted, and writing it into SQLite in order to read it back is copying a list in order to walk it. At about 135 bytes a URL that copy is roughly 283 GB for one server's third of a threefold recrawl of a monthly crawl, against 2.8 GB free on server2 and 6.6 GB on server3.

Streaming the live domain release, twenty million rows in 61 seconds at 327 000 rows a second, with the heap sawtoothing between 41 and 81 MB inside each part and resetting at every part boundary. The twentieth million holds what the first million held. What replaces the frontier's resume story is a checkpoint of a part number and a row offset, a few hundred bytes whether the list has a thousand rows or six billion, written after the WARC bytes it accounts for are flushed and put in place by a rename.

A real kill against the live dataset: the first run took a SIGINT after 109 seconds having emitted 224 pages with its checkpoint at row 200, the resume emitted 120 more, and the overlap between them was 67 URLs. That is exactly the partial batch the signal threw away, under the batch size of 100, and nothing was skipped. That property comes from refusing to checkpoint a batch that was cut short, because a checkpoint past rows that were never fetched skips them silently and nobody ever finds out.

Three things only showed up because the done-when says real data. robots.txt was going through the bulk client, which is the same bug #162 fixed for crawl run and costs the same here, since the Common Crawl delay is per process and would queue a million unrelated hosts five a second. --max-pages 5 fetched ten pages, because thirty two workers reading a counter all read 4 at once. And one host that accepted a connection and then said nothing stalled a whole twenty page run for over a minute while thirty one workers sat idle, which a goroutine dump caught in Transport.getConn. The crawl default of a two minute timeout is right for one large file off a bulk host and wrong for a million sites visited once each, so the recrawl takes thirty seconds for a page and ten for a robots.txt. That run went from 209 seconds to 18.

Worth noting for #164 that the third one is a preview of it. Bounding the timeout keeps a dead host from holding a batch, but the batch still ends no sooner than its slowest item, and at 250 pages a second a 2000 item batch is eight seconds of work that can wait forty for one straggler. R5 is where that gets solved properly rather than bounded.

Next is #163, writing recrawl captures as Parquet with inline bodies. WARC is what the crawler writes today and it is the right archival format, but it is not what gets published, and the two open-index repos want rows. That is also the last piece before #165 can publish anything.

Merged #173, R4: write recrawl captures as Parquet with inline bodies.

Closes:

Write recrawl captures as Parquet with inline bodies #163 Write recrawl captures as Parquet with inline bodies

#163 is merged as #173, so the recrawl writes rows now.

recrawl run --format parquet is the default and every fetch becomes one row with the body, the request head and the response head inline, in ami's captures schema column for column. --format warc still does what it always did. The schema match is pinned by a test rather than left as an intention, because a schema promise that is not checked breaks as an empty result somewhere downstream rather than as an error here.

Two things I got wrong on the way, both worth writing down because both are the kind of thing that would have looked fine for a week and then eaten a hundred day run.

The first was the sink contract, and I caught this one by thinking rather than by testing. A Parquet file is unreadable until its footer is written. So fsync on one promises nothing, and a checkpoint written after an fsync would point at rows that nobody can open. Sync now answers whether it managed durability, not just whether it errored, and the recrawl holds its checkpoint when the answer is no. A WARC is durable wherever it is fsynced, so a WARC run checkpoints every batch exactly as before.

The second I did not catch by thinking, and it took a live run against open-index/ccrawl-domains to find. I had rotation in Write, checking the accumulated size as rows went in, which reads as the obvious place to put it. It is the wrong place. With rotation inside Write, a run syncing between batches essentially never finds its writer sitting at a shard boundary, so Sync always answers "not durable" and the checkpoint never moves at all. Not rarely. Never. Rotation belongs in Sync, which makes the end of a shard and the end of a batch the same place, and then the checkpoint advances once per shard. The cost is that a shard overshoots --shard-size by up to a batch of pages. That is a fine price for a checkpoint that exists. There is a regression test on it now.

The numbers, from a real run against the published domain list, 400 pages, 32 MB shards, three shards, checkpoint advancing every batch through to row 400:


captures	398
payload, bodies and heads	139.34 MB
gzip per record, which is what a WARC does	25.51 MB
zstd columnar, one Parquet shard	22.79 MB

Eleven percent. I had written "far tighter" in the docs before measuring and had to go back and fix it, so here is the honest version: compression is a bonus, not the argument. The argument is that a reader can count status codes or pull one column without touching the bodies, which a WARC cannot offer at any ratio.

The same run turned up something the publish step needs to know. A shard holding a handful of rows gets almost none of the columnar gain, because the gain comes from having many similar pages in the same file. My first test used a 1 MB shard size, got shards of one to ten rows, and produced 42 MB of files from 24.6 MB of payload. The same captures in one shard are 4 MB. So a very small --shard-size is a false economy and the reference says so.

Next is #164, keeping robots.txt correct at fleet speed. #160 put the partition key on the registered domain so a host and its politeness clock stay on one machine, which is the half of the problem that sharding solves. The other half is what happens inside one machine at 250 pages a second: how long a robots.txt is trusted, what a fetch that times out means, and what happens to the several thousand hosts a shard is working through at once. The 10 second robots timeout that came out of #161's hung run is a floor, not an answer.

Merged #174, Keep robots.txt correct at fleet speed.

Closes:

Keep robots.txt correct at fleet speed #164 Keep robots.txt correct at fleet speed

Merged #194, fix(deploy): stop the crawl unit logging a line per page for months.

#194 is merged and deployed. It came out of chasing a disk number that did not add up.

server3 lost 3 GB in an hour while its capture directory held one megabyte, which ruled out the thing everybody suspects first. The crawl was logging one JSON line per page to stdout and systemd was writing every one of them to the journal. That is about 390 MB an hour, so each machine was spending half a gigabyte of disk a day recording lines nobody reads, on boxes at 91 and 97 percent full.

The crawl unit now sends stdout to /dev/null. Nothing an operator uses is lost, because the release the run picked, the failure breakdown by class, the timing line and every error are on stderr, so systemctl status and journalctl answer what they are asked. The publisher keeps its stdout, since a line per commit is a handful an hour and is the only record of what reached the hub, and the test checks both halves so one does not pick up the other's setting by being edited next to it. Confirmed after deploying: zero page lines in five minutes and the journal holding at 196 MB.

The rest of the disk on these boxes is not ours. server2 is running gao-crawl-bin and gao-bin.clean from another project, and that is what moves its free space by gigabytes an hour. Our footprint on both machines is a one megabyte capture directory, because the publisher deletes each shard once it is committed, plus a capped journal. Worth writing down so the next person watching df on a fleet machine does not go looking for a leak in this crawler.

Cleanup while I was there: go build caches, my own benchmark leftovers in /tmp and oversized journals, about 2.8 GB back on server2 and 2.5 GB on server3. Nothing under /root/data was touched, since that is other projects' data and not mine to delete. server1 is too loaded to complete a du inside a three minute ssh and has 156 GB free anyway.

Current state: both machines crawling the domain list, server3 also crawling the URL list, all four units active, both hub repos publishing.

#195 is merged. It fixes a bug I introduced in #193, and it is the kind that does not announce itself.

The reorder buffer in #193 got the URL crawl off one page a second, and it did that by holding a site's rows back while the rotation works through other sites. The flight set that names the resume point was written on the assumption that rows are handed out in work list order, which is exactly what the buffer stopped being true. So it picked its oldest item by sequence number rather than by row, and when the set emptied it reported one past the last row it had retired, which knows nothing about rows still sitting in the buffer unread by the pool.

Sampling the checkpoint file on the live URL run on server3 every thirty seconds, same run, no restarts:

13:08:43 row 126213
13:09:13 row 489026
13:09:43 row 489026
13:10:14 row 126287

The two positions were 362739 rows apart. A site at row 126287 was draining at the one page a second its politeness clock allows while the reader had reached row 489026. Whichever number a kill left behind is where the next run started, so half the time it started past rows that had been read and never fetched, and nothing would ever have gone back for them.

The resume point is now the earlier of the two positions, because a row can be unfinished in either place, and the flight set picks its oldest by row. Both tests fail against the old code with the right symptom, row 900 instead of 100 and row 81 with row 1 still in hand.

One claim in the docs was wrong and is now corrected. The buffer caps how many rows it holds at sixteen batches, and I had written that as though it also caps how far back the checkpoint sits. It does not. A big site drains at one page a second whatever the pool is doing, so its rows stay in hand and the gap grows for as long as that site has rows left. Nothing is lost by that, the pages in between are fetched and written and committed, but a kill refetches them. The way out is a work list that is not sorted by host, which is a change to how the URL list is published rather than to how it is read.

Two operational things from the same session.

server3's domains run was OOM killed at 12:14 and restarted. Its four writers were each buffering a 512 MB shard in memory, which is 2 GB of shard buffer on a 23 GB box that is also running the URL crawl and other people's work. Shard size there is now 128 MB, which is the buffer knob rather than the throughput knob, and it lands checkpoints more often as a side effect.

Disk on server3 is done. The 159 GB arctic directory was the same thing that was cleared on server2, nothing had touched it in two weeks and no process held it. The box went from 96 percent full with 19 GB free to 55 percent with 178 GB free, and all four units stayed up through it. server1 is at 58 percent with 157 GB free and is not in the fleet, so nothing there is urgent.

#196 and #197 are merged and both machines are on v0.10.2-27.

Resuming used to read every row before the checkpoint and throw it away. The comment in the code said that was the only option, because a row offset is not a byte offset and parquet cannot turn one into the other without an index we do not publish. That is not right. A parquet file records how many rows are in each row group, so a seek steps over whole groups on metadata alone and only the group the row lands in has to be decoded.

The cost was never decode time, it was bandwidth. Parts are read over ranged HTTP, so reading through to the offset pulls every byte before it across the network in order to fetch nothing. On a test part, reading one row twenty rows from the end cost 142069 bytes, exactly what reading the first row costs, because the whole prefix goes over the wire either way. It is 13980 now.

Measured on server2 against part 0 of the live URL list, which holds 6890137 rows, timing a one page run from a fixed row. Same row for both binaries so the page fetched is the same one, and the two alternated so drift on a loaded box cancels.

resume row	before	after
540000	13s, 9s	7s, 6s
6000000	39s, 43s	12s, 6s

So a few seconds where the fleet sits today and most of a minute near the end of a part. It matters because it is paid on every restart, and server3 has been restarting several times a day.

Two things about how I got there that are worth writing down.

My first attempt at this measurement was wrong and I nearly reported it. The probe wrote the resume row into a state file through a shell function, and the row was being expanded by my own shell before it ever reached the server, so every run got invalid JSON and failed in about two seconds. Old and new looked identical at every depth I tried, which I took at face value for three rounds before checking whether the runs were doing any work at all. They were not. The numbers above come from a script written on the server so nothing expands locally.

The first version of the unit test was also too weak to catch anything. It asserted the resume read less than four times the bytes of reading the first row, which the old code passed comfortably. The real contrast is 142069 against 13980, and the test now asserts half, which fails against the old code with the two numbers equal.

#198 is merged. It does not make the crawl faster, it measures the one part of the run that was never measured.

The rate work stalled because the run reports where its worker time goes and reports nothing about the feeder. On server2 against the live URL list at 96 workers, the line reads robots 0%, host clock 0%, fetching 8%, extracting 3%, queueing to write 0%, idle 90% of 96 workers at 2.6 pages a second. The host clock is zero, so politeness is not the ceiling. Dropping to 8 workers cost only 18 percent of the rate, so the pool is not the ceiling either. That leaves the feeder, and there was nothing to look at.

I want to be straight about what happened next, because I nearly shipped the wrong thing. My guess was that the reorder buffer's row cap was too small and was starving host variety, and I built the change and deployed a test binary before measuring it. The measurement says otherwise: 32000 rows and 400000 rows both gave 131 distinct hosts and 2.6 pages a second, for 124 MB more resident memory. The hypothesis is dead and the change is not in the tree.

What is in the tree is the two numbers that would have settled it in ten minutes. The feeder has exactly two states, reading the next row off the work list or waiting for a worker to take the row it is holding, and it now times both and counts the rows it hands out. Reading large means the work list cannot be read fast enough to fill the pool. Waiting large means the pool is the constraint, which is the healthy case and the only case where the worker phase shares are worth acting on. The rows a second is the ceiling on the run, since no run fetches more pages a second than its feeder hands out rows.

It prints as its own line rather than as more fields on the existing one. The phase shares are shares of the pool, which is workers times wall clock, and there is one feeder, so its shares are against one goroutine's wall clock. Mixing the two in a sentence would invite adding numbers that do not add.

Next is deploying this to server2 and server3 and reading what the feeder says on the live URL list. The 250 pages a second box stays open and I am not going to guess at it again.

#199 is merged and both servers are on v0.10.2-30-g4bbeade. This is the measurement the feeder line was built for, and it is the answer to why the 250 pages a second box is still open.

First, #199 exists because the first probe with #198 on it caught a hole in #198. A bounded run reported the feeder reading 12 percent and waiting 24 percent, which leaves two thirds of the run in no state at all. It was in no state, it had returned: the feeder stops the moment the page limit fires and the run carries on until the pool drains, and on that probe one straggler sat out the whole 30 second timeout at the end. Sharing the feeder's time against the run's wall clock charged that drain to a goroutine that had already finished and quartered the row rate with it, 8 a second reported against 22 actually handed out. The feeder now times its own life and the line leads with how much of the run that was.

With the numbers honest, here is what the URL list actually costs on server2 at 96 workers.

The feeder is not the bottleneck. It waits for a free worker 71 to 92 percent of its life and reads the work list 8 to 13 percent. Every theory I had about the work list is dead, including the buffer one I dropped in the last comment.

Extraction is the bottleneck and it is worth about two and a half times the rate. Palindrome ordered so the box's own load drift cancels, 2000 pages, same resume row:

extraction	rate
on	17.4, 14.0 pages a second
off	38.1, 39.4 pages a second

Benchmarked against a real captures shard off that same run, one page costs about 33 ms of CPU at best, on a 144 KB average page. Markdown is about 25 ms of it and text about 7 ms.

Writers make no difference, and I want to record how nearly I got this wrong. One pair of runs read 11.5 pages a second at 2 writers and 28.5 at 6, which is a two and a half times win and I was ready to call it one. Alternated 2, 6, 6, 2 the same probe reads 14.1, 16.0, 15.3, 17.3. That is the shared box swinging by 40 percent between runs and nothing else. The unalternated pair would have gone into a PR body as a result.

So the arithmetic on the goal. 250 pages a second at 33 ms of CPU a page is 8.25 cores of extraction alone, before fetching, TLS, Parquet encoding and compression. server2 has 6 cores at a load average of 24 to 30. server3 has 8 cores at 33 to 40. Most of that load is not ours.

250 pages a second per server is not reachable on this hardware with text and Markdown extraction on, and no amount of tuning in this repo changes that. The ways out are all scope decisions rather than code: bigger machines, a lower target per server, or extraction moved off the crawl into a pass of its own over the published bodies. I am not going to keep grinding at the crawler for a number the cores cannot produce. The box stays open and unticked until one of those is picked.

Two things to report, one of them a measurement mistake of mine that was making every number in this issue look worse than it is.

Short probes were halving the rate. A bounded run stops feeding when --max-pages fires and then waits for the pool to drain, and on a 2000 page run that drain is about half the wall clock, because a straggler sits out the full 30 second timeout. The rate line divides pages by the whole wall clock, so it was charging the drain against the crawl. Everything I posted from short probes understates by roughly two times. Re-measured at 15000 pages, where the drain is a small share, on server2 against the live URL list:

workers	writers	extraction	rate	where the pool went
96	2	on	32.7/s	fetch 44%, extract 27%, queueing to write 20%, idle 8%
96	2	off	58.4/s	fetch 48%, host clock 16%, queueing to write 16%, idle 19%
288	2	off	55.6/s	fetch 32%, queueing to write 41%, idle 21%, writers 75% busy
288	6	off	killed	3.6 GB resident, OOM

So the fleet is at 33 a second and not 14, and raw HTML with no extraction at all is 58 and not 39. The 250 target is still four times away before extraction is in the picture, but the starting point was wrong and I would rather correct it than leave it standing.

Raw HTML alone does not get to 250. That was worth asking and the answer is 58 a second on server2, so extraction is worth about 1.8 times the rate and is not the only thing in the way.

Tripling the pool buys nothing, and that is the interesting part. 96 workers to 288 moved the rate from 58.4 to 55.6 and moved the cost rather than removing it: fetching fell from 48 percent of the pool to 32 and queueing to write rose from 16 to 41 with the writers 75 percent busy. Widening the sink to match is what produced the OOM, at 3.6 GB on a box with 2 GB free, and it took two of another tenant's Chrome processes with it. That is on me for pushing a shared box that hard and I have stopped probing at that width.

That failure is the argument for #200, which I have opened and added to the checklist. One worker pool is doing two jobs with different limits, fetching which wants to be hundreds wide and extraction which wants to be about as wide as the box has cores, and the sink is reached through a queue whose depth is counted in items while what fills it is bodies that run from a kilobyte to several megabytes. writeQueueDepth is 256 items with a comment saying a page averages 300 KB, which is a way of admitting it means to bound bytes and cannot. Three stages, three widths, queues bounded in bytes, and every stage reporting what came in, what went out, how long it was busy and how long it was blocked pushing downstream, so the constraint names itself the way the feeder line does now.

#200 does not promise 250. It promises that widening one stage stops breaking another, that memory is a setting rather than a surprise, and that the remaining gap is a number rather than an argument.

Merged #201, perf(recrawl): bound the write queue in bytes and report it.

#201 landed. The queue between the pool and the sink is bounded in bytes now rather than in items, and the run reports the bound next to the high water mark it reached.

The bound used to be 256 items with a comment beside it saying that a page averages 300 KB, which is a way of admitting that what it meant to bound was bytes. A body on this corpus runs from a kilobyte to the 10 MB cap, so the same 256 items is 76 MB on an average stretch and 2.5 GB on a bad one. --queue-mb sets it and the default is 128 MB.

Measured on server2 straight after, at 96 workers and 2 writers with no extraction, the queue peaked at 32.9 MB against the 134.2 MB bound, so it was never the reason a worker waited. The run held 44.7 pages a second and 1.93 GB resident.

The honest part of that result is on #200: bounding the queue did not fix the kill at 288 workers and 6 writers, because the memory is not in the queue. It is in the Parquet encoder buffers, which hold rows until a shard seals and which nothing bounds or reports. That is now its own box.

R9, split the recrawl into stages with their own pools, queues and numbers #200 first box, every queue bounded by bytes with the bound and the peak reported

Merged #202, feat(recrawl): write a CPU and a heap profile per run, by name.

Merged #203, perf(recrawl): divide the zstd encoders across the writers.

#202 and #203 landed, and together they answer the question #201 opened.

#202 adds --profile-dir and --profile-name to recrawl run, so a run writes a CPU and a heap profile under a name of its own and two runs can be diffed with go tool pprof -base. The heap profile is taken as soon as the run stops and before anything is closed, since the encoders hold their buffers until a shard seals.

Pointed at a real run it found the memory straight away, and it was not the queue. More than half of a 96 worker run's 750 MB was zstd encoder state: the codec asked for four concurrent encoders and every writer has its own codec, so the count was writers times four rather than four in total. At six writers that is twenty four encoders holding over a gigabyte before a single page is in a buffer, which is most of what killed the 288 worker run last week.

#203 divides the concurrency across the writers. Measured on server2 at 96 workers and 3 writers, run A B B A so drift in a shared box cancels, peak resident went from 2.25 and 2.48 GB to 1.74 and 1.77 GB for the same rate. The two profiles agree on where it went: ensureHist 416 MB against 64 MB, which is twelve encoders against three.

Both servers are on v0.10.2-33-g65dda64 and every unit is up.

R9, split the recrawl into stages with their own pools, queues and numbers #200 second box, a CPU and a heap profile per run, by name
R9, split the recrawl into stages with their own pools, queues and numbers #200 third box, the encoder buffers counted in the run's memory rather than only the queue

#204 merged and both servers are on v0.10.2-34-gfcff680.

That is the last of the memory work under #200. Three bounds are written down now, all three found by pointing the profiler from #202 at a live run rather than by reasoning about the code:

what was holding it	before	after
the write queue, an item count	unbounded in bytes	128 MB bound, reported at the end of every run
zstd encoder state, four per writer	writers times four	four in total, divided across the writers
the writer's batch, a row count	217 MB across three writers	16 MB per writer, whichever of count or bytes fills first
response body slack, buffer doubling	up to twice the body	sized from Content-Length, right sized above 32 KB of slack

Peak resident on server2 at 96 workers and 3 writers has gone from 2.48 GB to 0.85 GB over the three PRs, which is 66 percent, and the in use heap from 993 MB to 136 MB. The rate is unchanged by the first two and the same or a little better after the third. Every one of those figures is from a run on server2 against the live URL list, A B B A, 15000 pages a run.

The reason to care is #166 rather than the memory itself. The run that killed two of another tenant's processes was 288 workers and 6 writers, and the whole point of writing these bounds down is that a width like that becomes something a box can be checked against beforehand instead of something the kernel decides. Whether the fleet can actually be run that wide is the next measurement.

What is left on #200 is the shape rather than the memory: fetching, extraction and the sink are still one pool with two knobs, and the timing line on the current build says fetching is 80 percent of the pool at 70 pages a second with extraction off. Three pools with three widths is the part the rate needs, and it is what #200 is now down to.

#205 merged and both servers are on v0.10.2-35-gc33338a.

That is the second half of #200. The memory work bounded what the run holds, and this one fixed the shape: fetching, rendering and encoding are three pools with three widths now, each with its own byte bound and each reporting what it did with its own clock. --extractors is the new flag and the summary gained a line for the pool it sets.

The reason to care about the shape is that it is what made the next answer findable. The extract pool was defaulted to the core count, on an argument that reads perfectly well and is wrong on these machines, and the first sweep said so: 32.5 pages a second at 6 extractors, 35.2 at 12, 40.3 at 24. These are shared boxes with a load average around twelve from other tenants, and our runnable goroutines are what win a share of the cores. A pool sized from nproc is sized for a machine we do not have. The default is four times the cores now.

40.3 pages a second with extraction on, against 33 when #200 was opened. That is a fifth of the way to 250 and the remaining gap is not a knob any more, which is the useful thing about it. At 24 extractors the run reports fetching at 64 percent of 96 workers and the extract pool 65 percent busy on six cores, so both stages want more machine rather than more configuration.

So #166 is now a decision rather than a measurement, and it is the same three options it always was: bigger machines, a lower target per server, or moving extraction into its own pass. The difference is that all three can be priced now, which they could not be a week ago.

#200 is merged, measured and closed, so the checklist has one box left and it is #166.

What #200 shipped, across #201 through #205: every queue bounded in bytes with the bound and the high water mark printed, a CPU profile of a live run rather than of a benchmark, body buffers that stop carrying their own slack into the row and the batch, and extraction moved out of the fetch worker into its own pool with its own width and its own byte budget.
The resume promise is unchanged, and there is a test that holds it to that: a row is still not retired until it is with the sink, and the extract pool only added slots to the replay bound rather than moving the checkpoint.

The numbers at the end of it, 288 workers, 6 writers, 15000 pages, each run inside a 3 GB hard cap so a bad bound kills our run and nobody else's:

server	extraction	rate	peak resident
server2	off	78.3/s	1.44 GB
server2	on	40.4/s	1.59 GB
server3	off	80.2/s	1.34 GB
server3	on	37.2/s	1.85 GB

That same config reached 3.6 GB and was OOM killed when this work started, so the memory half of the issue is settled.
The rate half is not, and now it says why: on both boxes the fetch pool spends 68 and 74 percent of its time handing work on, and both extract queues sit near their bounds, so the constraint is the extract pool and not the network.

Which leaves #166 as a scope decision rather than another round of tuning.
Fetching alone is 78 to 80 pages a second a server, which is up from 58 but is still a third of the target, and with extraction on it is 37 to 40.
The three ways out are bigger machines, a lower target per server, or extraction as its own pass off the crawl path, and that choice is what I want to settle next rather than another sweep.

Navigation Menu

Repository navigation

Milestone: recrawl-fleet #168

24 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Milestone: recrawl-fleet #168

Description

Activity

github-actions commented on Aug 20, 2026

tamnd commented on Aug 20, 2026

github-actions commented on Aug 20, 2026

tamnd commented on Aug 20, 2026

github-actions commented on Aug 20, 2026

tamnd commented on Aug 20, 2026

github-actions commented on Aug 20, 2026

tamnd commented on Aug 20, 2026

github-actions commented on Aug 20, 2026

24 remaining items

github-actions commented on Aug 22, 2026

tamnd commented on Aug 22, 2026

tamnd commented on Aug 22, 2026

tamnd commented on Aug 22, 2026

tamnd commented on Aug 22, 2026

tamnd commented on Aug 23, 2026

tamnd commented on Aug 23, 2026

github-actions commented on Aug 23, 2026

tamnd commented on Aug 23, 2026

github-actions commented on Aug 23, 2026

github-actions commented on Aug 23, 2026

tamnd commented on Aug 23, 2026

tamnd commented on Aug 24, 2026

tamnd commented on Aug 24, 2026

tamnd commented on Aug 24, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions