Command line
The complete reference of node src/cli.mjs - every option with its default, outputs (NDJSON, CSV, HTML report), comparison, segments, resume, sitemap and extraction.
Usage
node src/cli.mjs <url> [options]
node src/cli.mjs --list urls.txt [options]The CLI prints the score and the issues in your language, saves the crawl as gzipped NDJSON (which the interface opens with Import or from the history), and with --csv exports pages.csv and issues.csv, with --report writes a branded HTML audit report that you can print to PDF. JSON, CSV and report are each written as a stream and independently: one failure does not block the others, and the exit code is 1 if any failed.
Examples
node src/cli.mjs https://www.example.com --max 2000 --csv --report --lang en
node src/cli.mjs https://www.example.com --render --ua googlebotMobile --report --client "ACME" --author "Your name"
node src/cli.mjs --list urls.txt --lang de --csv
node src/cli.mjs --sitemap https://www.example.com/sitemap.xml --extract "price=.price" --extract "sku=xpath://meta[@itemprop='sku']/@content" --csv
node src/cli.mjs https://www.example.com --report --compare crawls/previous.jsonl.gz
node src/cli.mjs --resume crawls/.en-cours/www.example.com_2026-10-21T09-00-00.ndjson --csv
node src/cli.mjs https://www.example.com --max 20000 --segments segments.json --csv --report
node src/cli.mjs https://www.example.com --ga4-key ga4-key.json --ga4-property 123456789 --csv
node src/cli.mjs https://staging.example.com --auth user:pass --rate 2 --robots report --path-limit "/blog/=200" --extract "price=.price" --remove-params "utm_*,gclid"
node src/cli.mjs --helpOptions
Generated from node src/cli.mjs --help in the repository (3.21.0); defaults come from the code.
| Option | Description | Default |
|---|---|---|
--max N | Max URLs (500) | 500 |
--depth N | Max depth (20) | 20 |
--concurrency N | Concurrent requests (5) | 5 |
--delay MS | Delay between requests per worker (0) | 0 |
--ua NAME | arachnide | googlebot | googlebotMobile | chrome | arachnide |
--render | JavaScript rendering with Chrome / Edge (installed browser) | |
--list FILE | List mode: crawl only the URLs of a text file (one per line) | |
--external | Check external links | |
--images | Check image size and status | |
--rate N | Max URLs per second (global cap) | |
--robots MODE | respect | report (ignore but report) | ignore | respect |
--custom-robots F | Use this robots.txt file instead of the site's one | |
--per-depth N | Max URLs per crawl depth | |
--folder-depth N | Max folder depth | |
--max-params N | Max number of query strings | |
--max-redirects N | Max redirects to follow (10) | 10 |
--path-limit P=N | Limit by URL path, e.g. "/blog/=200" (repeatable) | |
--remove-params L | Remove URL parameters, e.g. "utm_*,gclid" | |
--auth USER:PASS | HTTP authentication (staging sites) | |
--header "K: V" | Custom HTTP header (repeatable) | |
--extract N=SEL | Custom extraction (repeatable): N=SEL (CSS, text), N=/regex/, N=xpath:EXPR, N=attr:href:SEL, N=html:SEL, N=count:SEL — one column per rule in pages.csv | |
--sitemap URL | List mode from a sitemap or sitemap index: crawl exactly the URLs it lists | |
--keep-params | Keep URL parameters as they are (by default: sorted, tracking params utm_*/gclid/fbclid… removed) | |
--robots-unavailable allow|disallow | robots.txt in 5xx or unreachable: disallow all (RFC 9309, default) or allow | disallow |
--psi N | PageSpeed Insights + Core Web Vitals (CrUX) for the N most important pages | |
--psi-key KEY | Google API key (or env ARACHNIDE_PSI_KEY) --psi-strategy mobile|desktop | |
--gsc-key FILE | Search Console: service account JSON key file | |
--gsc-property P | Search Console property (sc-domain:example.com), auto-detected if omitted | |
--gsc-days N | Search Console period in days (90) | 90 |
--gsc-csv FILE | Import a Search Console "Pages" CSV export instead of the API | |
--ga4-key FILE | Google Analytics 4: service account JSON key file (the Search Console one works if it has access) | |
--ga4-property ID | GA4 property ID (123456789), auto-detected from the domain if omitted | |
--ga4-days N | GA4 period in days (90) | 90 |
--ga4-csv FILE | Import a GA4 "Pages and screens" (or "Landing page") CSV export instead of the API | |
--alert-email A,B | Email an alert if the crawl got worse than the previous one (for cron jobs) | |
--alert-score-drop N | Alert threshold in score points (5) | 5 |
--ignore-robots | Ignore robots.txt (same as --robots ignore) | |
--subdomains | Crawl subdomains too | |
--no-sitemap | Do not read XML sitemaps | |
--include REGEX | Only crawl matching URLs (repeatable: one regex per flag) | |
--exclude REGEX | Skip matching URLs (repeatable) | |
--stay-in-folder | Do not crawl outside the start URL's folder (https://x.com/blog/ → /blog/… only) | |
--search TEXT | Custom search in the source code (/regex/ supported) | |
--segments FILE | Segments (JSON exported from the app, or [{ "name", "match", "rules": [{ "field", "op", "value" }] }]): score, errors and averages per site section, a segment column in the CSV, a section in the report | |
--segment-folders | One segment per top-level directory (automatic) | |
--segment ID | Report and CSV on this segment only (e.g. --segment blog) | |
--lang CODE | en | fr | de | es | it | pt (labels, CSV, report) | |
--csv | Also export pages.csv and issues.csv | |
--report | Also generate a professional HTML audit report (print to PDF) · --report-title, --client, --author, --company, --contact, --website, --color "#5b8c1e" | |
--compare FILE | Previous crawl .json to show the evolution in the report | |
--out DIR | Output folder (./crawls) | ./crawls |
--resume FILE | Resume an interrupted crawl from its journal (<out>/.en-cours/*.ndjson) | |
--no-journal | Do not write the resume journal during the crawl |
Extraction rules
--extract NAME=RULE, repeatable, one column per rule in pages.csv:
| Rule | Meaning |
|---|---|
price=.price | CSS selector, text content |
author=attr:content:meta[name=author] | CSS selector, attribute value |
menu=html:nav.main | CSS selector, inner HTML |
links=count:a[href] | CSS selector, number of matches |
sku=xpath://meta[@itemprop='sku']/@content | XPath (Chrome in rendering mode, jsdom otherwise) |
phone=/\+34 \d{3} \d{3} \d{3}/ | Regular expression (2-second timeout per page) |
Outputs
Files are written in --out (default ./crawls), named <host>_<date>:
<host>_<date>.jsonl.gz: the full crawl (NDJSON, one JSON object per line, gzipped);<host>_<date>_pages.csvand_issues.csvwith--csv(semicolon separated, UTF-8 with BOM, one column per extraction, a Segment column when segments are defined);<host>_<date>_report_<lang>.htmlwith--report;crawls/.en-cours/<host>_<date>.ndjson: the resume journal while a crawl runs (kept after Ctrl+C, deleted on completion).
Press Ctrl+C to stop a crawl cleanly; the CLI then prints the --resume command to pick it up later.
Cron and alerts
--alert-email you@example.com --alert-score-drop 5 compares the crawl with the previous crawl of the same host in the output folder and e-mails you only when something got worse (SMTP settings from the app, or ARACHNIDE_SMTP_URL). See Google data, scheduled crawls and alerts.
Other commands
| Command | Purpose |
|---|---|
npm start | The web interface on http://localhost:4000 (PORT and HOST to change; ARACHNIDE_NO_SCHEDULER=1 disables scheduled crawls) |
npm run mcp | The MCP server over stdio (also arachnide-mcp as a bin) |
npm run scheduler | Scheduled crawls without the interface |
npm run bench -- --pages 200000 | The synthetic-site benchmark |
npm test | The test suite (Node 18, 20 and 22) |