Google data, scheduled crawls and alerts

PageSpeed Insights and Core Web Vitals, Search Console by API or CSV, Google Analytics 4, scheduled crawls with e-mail alerts.

PageSpeed Insights and Core Web Vitals

Settings → Google → Pages to test (for example 20). After the crawl, Arachnide sends the most important pages to PageSpeed Insights: the most-clicked pages when Search Console data is available, otherwise the home page and the pages with the most internal links. For each page you get the Lighthouse performance score and field data from real Chrome users (CrUX, 75th percentile): LCP, INP, CLS, FCP and TTFB, each rated good, needs improvement or poor. These are the values Google uses. Small pages often have not enough traffic; the origin-level values are then shown in the crawl summary.

New Core Web Vitals tab, and three checks: Poor Core Web Vitals, Core Web Vitals need improvement, Low PageSpeed score.

An API key is optional but PageSpeed Insights limits anonymous requests. It is free and takes two minutes: create a project at console.cloud.google.com, enable PageSpeed Insights API, Credentials → Create credentials → API key, then paste it in Settings → Google or set ARACHNIDE_PSI_KEY.

Google Search Console

Two ways.

CSV import, no setup. In Search Console, Performance → Search results, pick the period, Export → Download CSV, unzip, then in Arachnide Settings → Google → Choose a CSV file and select Pages.csv. It works with every interface language.

API, automatic and needed for scheduled crawls.

1. console.cloud.google.com → enable Google Search Console API.
2. IAM → Service accounts → Create, then Keys → Add key → JSON. Save the file, for example ~/arachnide-gsc.json.
3. Search Console → Settings → Users and permissions → Add user: the service account e-mail (…@….iam.gserviceaccount.com) with Restricted permission.
4. In Arachnide, Settings → Google, enter the path of the JSON file. The property is detected automatically (domain property sc-domain:example.com or URL-prefix property).

The key never leaves your computer: Arachnide signs a short-lived token with it and only reads data (webmasters.readonly). The Search Console tab shows clicks, impressions, CTR and position per crawled URL; URLs Google knows but the crawl did not find are listed as Not in crawl. Four checks: pages seen by Google but not found by the crawl (orphan pages, often with traffic), indexable pages with no impressions, low CTR for the position (page 1 but fewer than half the expected clicks: rework the title and description), non-indexable pages getting Google traffic.

Google Analytics 4

Since 3.15. Search Console only sees pages that show up in Google; GA4 sees every visit (direct, e-mail, social, ads, search).

• Connection: the GA4 Data API with a service account (the Search Console one works once it is a Viewer of the property), the property auto-detected from the site's web data stream or given as an id (123456789, properties/… or a GA4 URL); or import of a GA4 CSV export (Pages and screens or Landing page, in any of the six interface languages, with , ; or tab separators, paths or full URLs). Period up to 730 days.
• Matching: host + path rows are tied to crawled URLs (trailing slash and query variants added up); other hosts and (not set) are ignored. Pages seen only in GA4 are listed too.
• Four checks: Orphan pages that get traffic (sessions but no internal link), Pages in error that still get traffic (4xx / 5xx with sessions), Non-indexable pages that get traffic, Indexable pages with no sessions.
• Google Analytics tab (sessions, views, users, engagement rate, in crawl, inlinks, status, indexability) with filters; GA4 sessions as a segment field and column.

Command line and agents

--psi 20 --psi-key KEY --psi-strategy mobile|desktop, --gsc-key ~/arachnide-gsc.json --gsc-property sc-domain:example.com --gsc-days 90 or --gsc-csv Pages.csv, --ga4-key key.json --ga4-property 123456789 --ga4-days 90 or --ga4-csv export.csv. Agents: psi_pages, gsc_key_file, gsc_property, gsc_csv, ga4_key_file, ga4_property, ga4_csv on the crawl tools; arachnide_search_pages can sort by sessions.

Scheduled crawls and e-mail alerts

Click the calendar button in the header, then Schedule a crawl.

• Frequency: daily, weekly or monthly, at the time you choose. A run missed because the computer was off is caught up at the next start. The current crawler configuration is saved with the schedule; passwords are not.
• Alerts: an e-mail is sent when a rule you tick is met: the score drops by N points or more; errors appear or increase (4xx, 5xx, broken links…); a new type of issue appears; or always, for a report after every crawl. The e-mail shows the alerts, a before / now table and how many issues were fixed, with the full HTML report attached (open it in a browser, print to PDF).
• SMTP settings are in the same window. For Gmail: smtp.gmail.com, port 465, Secure connection, your address and an app password. Send a test checks them. Or one variable: ARACHNIDE_SMTP_URL=smtps://user:password@smtp.example.com:465. The password is stored only in ~/.arachnide/smtp.json, readable by your user only.

When do scheduled crawls run? While Arachnide is open (npm start). To run them without the interface, start npm run scheduler when your computer starts: a LaunchAgent or pm2 start npm --name arachnide -- run scheduler on macOS, Task Scheduler → At log on on Windows, a systemd user service or @reboot in crontab on Linux. Already using cron? node src/cli.mjs https://example.com --alert-email you@example.com --alert-score-drop 5 crawls, compares with the previous crawl and e-mails you only when something got worse.