AI agents - MCP, REST and OpenAPI

Connect Claude Desktop, Claude Code, claude.ai, ChatGPT or your own agent to Arachnide - three doors, 25 tools, the suggested system prompt and the security model.

Three doors, one set of tools

Since 3.13, Arachnide is built to be driven by an AI agent: yours, or any assistant that speaks MCP (Claude) or OpenAPI (ChatGPT Actions, LangChain, n8n, Make…). The agent reads the audit you have open, starts crawls that appear live in the app, explores pages, gets the prioritised action plan and the GEO readiness report, and, with your approval, fixes the site through the CMS connectors. Everything runs on your machine or your server; nothing is sent to a third party except what your agent's own model receives.

The token is created the first time Arachnide starts and lives in ~/.arachnide/agent.json (mode 0600). Read it, copy it or renew it in Settings → AI agents; on a server, set ARACHNIDE_AGENT_TOKEN instead. The same screen shows the latest calls received from agents, the MCP and OpenAPI URLs, a public-URL field for tunnels and ready-to-paste snippets. The tools are identical on the three doors: they come from one registry, src/agent/tools.mjs. crawl_id: "current" always means the crawl open in the Arachnide app.

Claude Desktop and Claude Code (local, stdio)

claude mcp add arachnide -- node /path/to/arachnide-seo-spider/src/mcp.mjs
# with a secret for a site:
claude mcp add arachnide -e ARACHNIDE_WP_APP_PASSWORD="xxxx xxxx xxxx xxxx" -- node /path/to/arachnide-seo-spider/src/mcp.mjs

Claude Desktop: Settings → Developer → Edit Config, then add the server and restart Claude Desktop:

{
  "mcpServers": {
    "arachnide": {
      "command": "node",
      "args": ["C:\\Users\\you\\arachnide-seo-spider\\src\\mcp.mjs"],
      "env": { "ARACHNIDE_WP_APP_PASSWORD": "xxxx xxxx xxxx xxxx", "ARACHNIDE_LANG": "en" }
    }
  }
}

Then simply ask: “Audit www.my-site.com and fix the titles, descriptions and missing alt texts of the most important pages.”, or use the optimize_site prompt. With the app running you can also use the HTTP door from Claude Desktop: Settings → AI agents → Copy the Claude Desktop config gives { "mcpServers": { "arachnide": { "url": "http://127.0.0.1:4000/mcp", "headers": { "Authorization": "Bearer ara_…" } } } }.

Environment variables of the MCP server: ARACHNIDE_LANG (labels: en, fr, de, es, it, pt), ARACHNIDE_HOME (config and journal, default ~/.arachnide), ARACHNIDE_CONFIG (sites file), ARACHNIDE_CRAWLS (crawl folder, shared with the web app by default).

claude.ai custom connector and hosted agents (HTTP)

claude.ai only reaches servers on the public internet over HTTPS. Expose the app:

npm start                                        # Arachnide on http://localhost:4000
cloudflared tunnel --url http://localhost:4000   # or: ngrok http 4000 · tailscale funnel 4000

In Arachnide → Settings → AI agents, paste the public URL (for example https://xyz.trycloudflare.com). In claude.ai → Settings → Connectors → Add custom connector: URL https://xyz.trycloudflare.com/mcp, and the token as the bearer token. The token is your key: never share it. For a hosted agent built with the Claude Agent SDK or the Messages API, add the server as an MCP server with the same URL and Authorization header, or call the REST door directly.

ChatGPT (GPT Actions) and OpenAI agents

In the GPT builder → Actions → Import from URL: https://<your-host>/openapi.json. Authentication: API Key, auth type Bearer, paste the token. Each tool becomes an action (arachnide_get_summary, arachnide_action_plan…); read-only tools are marked non-consequential, write tools (apply, rollback, configure) ask for confirmation. With the OpenAI Agents / Responses API, point a hosted MCP tool at https://<your-host>/mcp with the bearer header, or wrap the REST endpoints as functions from the OpenAPI spec.

The 25 tools

Every reading tool (arachnide_get_summary, arachnide_list_issues, arachnide_get_issue, arachnide_action_plan, arachnide_search_pages, arachnide_geo_readiness) accepts segment. Google Analytics 4: pass ga4_key_file (and ga4_property) or ga4_csv to the crawl tools. List mode (3.21): mode: "list" with urls or sitemap_url, max_pages up to 200 000. GET /api/agent/tools returns the same list with the JSON Schema of each input.

GEO readiness

arachnide_geo_readiness scores a crawl out of 100 for visibility in AI answers: AI crawlers allowed or blocked in robots.txt (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended…), llms.txt, structured-data coverage and answer-oriented types, content depth and thin pages, duplicates, titles / descriptions / H1 / Open Graph hygiene, with a score by part and ordered recommendations. The same signals appear in the Robots, AI & sitemaps tab of the app.

Suggested system prompt

Paste this in your agent's system prompt (Claude project instructions, GPT instructions, Agent SDK system):

You are an SEO and GEO (generative engine optimisation) consultant. You work with Arachnide SEO Spider through its tools.
- When the user mentions "the audit", "the site" or "the crawl", use crawl_id "current": it is what they see in the Arachnide app. Call arachnide_list_crawls if unsure.
- Start every analysis with arachnide_get_summary, then arachnide_action_plan (what to do first) and arachnide_geo_readiness (visibility in AI answers). Explain in plain language, cite numbers, never invent data that is not in the tool results.
- To explore, use arachnide_search_pages (sorted by Link Score) and arachnide_get_page; to write content, read the real page with arachnide_fetch_page first.
- A new crawl: arachnide_start_crawl, then poll arachnide_crawl_status every 15-30 s; the user can watch it in the app.
- Changes to the live site: arachnide_suggest_fixes → write the values yourself (same language as the site, natural, unique, within pixel limits) → arachnide_preview_changes → show a before/after table → wait for an explicit "yes" → arachnide_apply_changes → arachnide_verify_changes. Small batches (10-30), most important pages first. Every batch can be undone with arachnide_rollback_changes.
- Never ask the user to paste passwords or API keys in the chat: configure sites with "env:VARIABLE" references.

Connecting a site (MCP server)

Sites live in ~/.arachnide/sites.json. The agent creates the entry with arachnide_configure_site; secrets never go through the chat: the file stores env:VARIABLE and the value comes from the MCP server environment. Fields the agent can change: SEO title, meta description, H1, canonical, noindex, image alt text, 301 redirects.

{
  "sites": {
    "my-blog": {
      "url": "https://www.my-blog.com",
      "connector": "wordpress",
      "user": "admin",
      "appPassword": "env:ARACHNIDE_WP_APP_PASSWORD",
      "policy": { "allowedFields": ["seo_title", "meta_description", "image_alt"], "maxChangesPerBatch": 30 }
    }
  }
}

Connector settings and prerequisites are listed in Fix in 1 click.

Security model

• The token is required on /mcp and /api/agent/*; /openapi.json is public (it contains no secret) so that GPT builders can import it. /api/agent/settings (token, log) answers only to the local interface.
• Requests are rate-limited (240 per minute per IP) and the last 50 calls are shown in the app.
• Disable agents at any time with the switch in Settings → AI agents; renew the token if it leaks.
• Expose Arachnide only over HTTPS (tunnel or reverse proxy). Behind a proxy, forward X-Forwarded-Proto / X-Forwarded-Host or set the public URL in the app (ARACHNIDE_PUBLIC_URL).
• Write tools go through the same policy as the MCP server: per-site allowed fields, batch size, read-only mode, journal and rollback. Apply and rollback tools are flagged destructive, so Claude Code and Claude Desktop ask for permission. Only the site's own host is accepted.

Examples

# summary of the audit open in the app
curl -X POST https://<host>/api/agent/arachnide_get_summary -H "Authorization: Bearer ara_…" -H "Content-Type: application/json" -d '{}'
# the 10 pages with the highest Link Score under /blog/
curl -X POST https://<host>/api/agent/arachnide_search_pages -H "Authorization: Bearer ara_…" -H "Content-Type: application/json" -d '{"folder":"/blog/","limit":10}'
# GEO readiness of a saved crawl
curl -X POST https://<host>/api/agent/arachnide_geo_readiness -H "Authorization: Bearer ara_…" -H "Content-Type: application/json" -d '{"crawl_id":"www.example.com_2026-10-19T09-00-00"}'

Errors are JSON: 400 invalid arguments (with the field path), 401 missing or invalid token, 404 unknown tool, crawl, issue or page, 409 a crawl is already running.