Large sites

The NDJSON format, the resume journal, list and sitemap modes, the 200 000-page benchmark and the crawl guard-rails added in 3.16 to 3.21.

What was measured

Versions 3.16 to 3.21 were a six-step scale programme, each step measured on a synthetic site generated locally by bench/site.mjs (N pages generated on the fly, about 60 template links and 40 deterministic links per page, 404s, redirects, near duplicates; no real site is ever hit).

Measured with npm run bench -- --pages 200000 on a local synthetic site generated by bench/site.mjs (no real site is hit). Your figures depend on your machine, the site and the options (JavaScript rendering is much slower).

The default budget is deliberately low. Raise Options → Limits → Crawl total for a large site, keep 2 to 5 threads on a client server, and remember that JavaScript rendering is much slower than raw HTML.

The six steps

1. 3.16, no more crash at the end: a Math.min(...) on one value per URL raised Maximum call stack size exceeded beyond about 125 000 URLs, after hours of crawl; fourteen small fixes removed it and the quadratic loops (fix jobs, AI examples, links to redirects, closest URL for 404s, table sort, folder counts). Crawl files are written atomically (temporary file then rename).
2. 3.17, a compact link graph: links live in typed-array columns (URLs and anchor texts interned once, 14 bytes per link) and the analysis runs on a CSR graph. Measured on 50 000 pages × 100 links: link storage 690 MB → 55 MB, analysis 29 s / 2.5 GB → 5.5 s / 647 MB, identical results.
3. 3.18, a crawl written as it goes: results are saved as gzipped NDJSON (.jsonl.gz, one JSON object per line, links by chunks), written and read as streams; the history reads only the first line. A resume journal records every URL queued and every page done; --resume (or the home screen) picks an interrupted crawl up where it stopped. Old .json crawls still open.
4. 3.19, an analysis that scales: every page is compared for near duplicates (the old check stopped at 5 000 pages without saying so) with an exact banded search on the 64-bit simhash; cannibalisation runs in linear time; segment link counters are computed in one pass.
5. 3.20, the browser never receives the whole crawl: the server keeps the crawl and its graph; the browser receives a light result as a stream and asks for the rest on demand (a page's links paginated by 200, the full page record on click, map edges for the displayed nodes only, the comparison computed server-side). Import and export go through the server.
6. 3.21, crawl guard-rails: the page budget counts pages actually downloaded; URL parameters are sorted and tracking parameters removed by default; facet traps are reported; an unreachable robots.txt disallows everything as RFC 9309 requires, with an explicit way out; 5xx and network cuts get a deferred retry; response bodies are bounded; a Chrome that dies is restarted and reported; a runaway regex is interrupted in a worker; list mode goes up to 500 000 URLs and --sitemap crawls exactly a sitemap.

Formats

• Saved crawl (.jsonl.gz): line 1 is the head (meta and summary), then one line per issue, one per page, URLs, anchors and links by chunks of 100 000 (columnar), then images, external links, redirects, broken links. No line is larger than a few megabytes whatever the site.
• Resume journal (.ndjson, append-only): a start line (URL, options), a q line per URL queued, a p line per page processed with its links and images. A brutal stop loses the last seconds only.
• Segments edited after the crawl go to a small side file <name>.segments.json; the crawl is never rewritten.
• Old .json exports (format 1 and 2) are detected and still open.

List and sitemap modes

List mode crawls exactly the URLs you give (pasted in the app, --list file.txt in the CLI, urls for agents), up to 500 000, without following links; internal linking is then not measured. Sitemap mode (--sitemap URL, sitemapUrl, sitemap_url) reads a sitemap or sitemap index and crawls exactly its URLs, deduplicated and normalised. In both modes robots.txt is read once per host.

Guard-rails to know about

• Budget: maxPages counts downloaded pages; URLs blocked by robots.txt are recorded but do not consume it. Discovery stays bounded in memory (4 × the budget).
• Parameters: ?a=1&b=2 and ?b=2&a=1 are one page; utm_*, gclid, fbclid, msclkid, mc_cid, srsltid, _ga… are removed. --keep-params restores the raw behaviour.
• Facets: a path producing 200 parameter variants is reported once (up to 20 paths).
• robots.txt in 5xx or unreachable: everything is disallowed, with a warning; --robots-unavailable allow or the report / ignore mode let the crawl through.
• Rendering: Chrome restarted every 500 rendered pages and once if it dies; a second death is reported instead of silent unrendered pages.
• Responses: bodies read as streams and abandoned beyond maxPageSizeKb; sitemaps and robots.txt bounded by maxSitemapKb (50 MB).
• Security: custom headers are no longer remembered in the browser, like the HTTP password.

Known gaps

Issue URL lists still travel whole to the browser (the issues page, the fix engine, the segments and the map colour by issue); paginating them is the next refinement, not needed below 200 000 pages. Opening a file from the Import button uploads it to the server, which keeps the whole crawl in memory, the same as after a crawl. Reproduce the benchmark yourself with npm run bench -- --pages 200000 (it needs about 8 GB of heap, set by the script).