FAQ
Installation, Chrome and JavaScript rendering, robots.txt in 5xx, old .json files, privacy, responsible use, and where to ask.
Is Arachnide really free?
Yes. Arachnide SEO Spider is published under the MIT license: you are free to use, modify and share it, including for commercial work. There is no paid tier, no licence key and no account. The only costs you may choose to pay are those of third-party services you connect yourself: an Anthropic or OpenAI API key for AI fixes, or a Google Cloud project for PageSpeed and Search Console APIs (free within their quotas).
Which systems and Node.js versions are supported?
macOS, Windows and Linux, with Node.js 18.17 or newer; the test suite runs on Node 18, 20 and 22. Install the current LTS from nodejs.org and check with node -v.
The launcher does not open on macOS
The first time, right-click start-mac.command and choose Open, because macOS blocks unsigned scripts. The launcher then installs the dependencies and opens your browser on http://localhost:4000.
I get “cd: no such file or directory” or “npm ERR! enoent … package.json”
You are not in the Arachnide folder. Use cd followed by the path of the unzipped folder, for example cd ~/Downloads/arachnide-seo-spider, then npm install and npm start.
“--disable-warning= is not allowed in NODE_OPTIONS”
Your shell sets an option your Node version does not support. Run unset NODE_OPTIONS (macOS, Linux) or update Node.js to the latest LTS. The launchers already unset it.
JavaScript rendering is unavailable
Install Google Chrome, Microsoft Edge, Chromium or Brave. If the browser is installed in an unusual location, start Arachnide with CHROME_PATH="/path/to/chrome" npm start. Nothing is downloaded by Arachnide itself: it uses the browser already on your computer through puppeteer-core.
Some pages show no H1 or no content
The site probably builds its content with JavaScript. Enable JavaScript rendering in Settings → Rendering and crawl again. The JavaScript tab then shows what changed between the raw and the rendered HTML.
The crawl stops at once: “robots.txt unavailable, everything disallowed”
The site's robots.txt answered a 5xx error or could not be reached. RFC 9309 requires a crawler to treat this as disallow all, and Arachnide does, with an explicit warning. If you are sure you are allowed to crawl the site, set robots.txt unavailable → allow (--robots-unavailable allow in the CLI) or use the report or ignore robots mode. A 4xx robots.txt, on the other hand, means everything is allowed.
My old crawls are .json files; can I still open them?
Yes. Since 3.18 crawls are saved as gzipped NDJSON (.jsonl.gz), but old .json exports (formats 1 and 2) are detected and still open from the history or the Import button. Crawls from 1.x open with fewer details.
Does Arachnide send my data anywhere?
No. The application has no account, no telemetry and no cloud; the interface listens only on 127.0.0.1. It contacts the sites you crawl, and only the services you configure yourself: Anthropic or OpenAI for AI fixes (the page excerpt needed to write a title or a description), your CMS through its connector, Google APIs, your SMTP server for alerts. API keys and credentials are stored encrypted on your disk (AES-256-GCM) and never returned to the browser, logged or exported.
Can the AI change my site without me?
No. The AI proposes, you review before / after and approve, a connector applies, Arachnide re-crawls to verify, and every change can be undone from the history. Simulation mode is on by default and an explicit warning precedes the first live write. For AI agents (MCP, REST), write tools are flagged as consequential so that Claude Code, Claude Desktop and ChatGPT ask for permission, and each site has a policy (allowed fields, batch size, read-only).
How many pages can it crawl?
The default budget is 500 pages: raise it in Settings → Limits. Arachnide was tested on a local synthetic site of 200 000 pages and 19.9 million links; see Large sites for the measured figures and what to expect on your machine.
Is it a Screaming Frog replacement?
It covers the same job, a desktop crawl turned into a technical audit, and adds fixes applied to the CMS and tools for AI agents, but it does not do everything a mature commercial product does (log file analysis, spelling checks, forms-based authentication…). See the honest comparison.
What is responsible use?
Only crawl websites you own or are allowed to audit. On client websites, keep 2 to 5 simultaneous requests and add a pause on shared hosting; respect robots.txt unless you have a reason and the right to do otherwise.
Where do I report a bug or ask a question?
Open an issue on GitHub with your operating system and Node.js version, the steps to reproduce, the URL you crawled if it is public and the error shown in the terminal or the browser console. Pull requests are welcome; see About.