crawlio-browser

Crawlio Browser

Documentation | API Reference | Chrome Extension

MCP server that gives AI full control of a live Chrome browser via CDP. 150 tools with framework-aware intelligence, typed evidence infrastructure, tracking pixel analysis, technographic fingerprinting, SEO auditing, and confidence-tracked findings — captures what static crawlers can't see.

When to use Crawlio Browser

Use Crawlio Browser when your AI needs to interact with a real browser — SPAs, authenticated pages, dynamic content, JS-rendered frameworks. Unlike headless browser tools, Crawlio Browser connects to your actual Chrome via a lightweight extension, giving the AI access to your logged-in sessions, cookies, and full browser state.

Crawlio Browser vs headless browser tools: Headless tools launch a separate browser process. Crawlio Browser connects to your existing Chrome — no separate browser, no login flows, full access to your tabs and sessions.

[!WARNING] This is the trade-off, stated plainly: connecting your own Chrome is the feature, and it means an AI agent can act as you on every site you are logged into. It attaches Chrome's debugger to the tab you connect, so it can read cookies, storage, and page content for that session.

Review what you connect it to. Prefer a dedicated Chrome profile for agent work. Nothing is captured until you connect a tab, and you can see the exact tool surface before configuring anything by running npx crawlio-browser tools. Sites can opt out with <meta name="crawlio-agent" content="disable">, which the extension honors.

Quick Start

1. Install the Chrome Extension
2. Run the init wizard:

   npx crawlio-browser init
   

That's it. Auto-detects and configures 14 MCP clients: Claude Code, Cursor, VS Code, Codex, Gemini CLI, Claude Desktop, ChatGPT Desktop, Windsurf, Cline, Zed, Goose, OpenCode, MCPorter, and Cline CLI.

Init wizard options

npx crawlio-browser init              # Default: code mode, stdio transport
npx crawlio-browser init --full       # Full mode (every tool exposed individually)
npx crawlio-browser init --portal     # Portal mode (persistent HTTP server)
npx crawlio-browser init --cloudflare # Add Cloudflare MCP (89 tools, no wrangler)
npx crawlio-browser init --dry-run    # Show what would happen
npx crawlio-browser init --yes        # Skip prompts (CI / scripted installs)
npx crawlio-browser init -a claude    # Target specific MCP client

As an Agent Plugin

The package is also an Agent Plugins v1 plugin, so a conformant client can load it directly instead of running the wizard — plugin.json at the root, the eleven skills under skills/, and the MCP server declared in mcp.json.

The product-facing crawlio-* workflows are folded into the eleven shipped skills. The similarly named definitions in agents/ remain repo-local development fixtures: they import src/evidence/* and loops/*, neither of which is part of the npm runtime. They are excluded from package.json instead of advertising a second, non-executable product surface.

Point the client at the installed package:

node_modules/crawlio-browser        # after `npm install crawlio-browser`

mcp.json resolves the server through ${PLUGIN_ROOT}/dist/mcp-server/index.js, which is why the plugin has to be an installed package rather than an unpacked tarball — the server imports its dependencies at runtime, so a bare extract starts and then dies without answering.

Inspecting what it exposes

npx crawlio-browser tools          # What code mode exposes (the default)
npx crawlio-browser tools --full   # Every tool, individually
npx crawlio-browser tools --json   # Machine-readable, for diffing across versions
npx crawlio-browser doctor         # Bridge, portal, native host, client configs
npx crawlio-browser --help         # All commands and options
npx crawlio-browser --version      # Version only

Both are read-only and run without a browser, an extension, or a network connection — you can see the whole surface before you configure any client. The numbers come from the same builders the server registers, so they cannot disagree with what your client receives.

Transport Modes

Portal mode is recommended for Claude Code — the server persists across context compaction and session restarts. On macOS, --portal installs a launchd agent for auto-start on login.

Manual setup (any client)

Per-client manual config

Claude Desktop — add to claude_desktop_config.json:

{ "mcpServers": { "crawlio-browser": { "command": "npx", "args": ["-y", "crawlio-browser"] } } }

Claude Code (Portal Mode) — start npx crawlio-browser --portal, then add to .mcp.json:

{ "mcpServers": { "crawlio-browser": { "type": "http", "url": "http://127.0.0.1:3001/mcp" } } }

Claude Code (stdio):

claude mcp add crawlio-browser -- npx -y crawlio-browser

Cursor — add to .cursor/mcp.json:

{ "mcpServers": { "crawlio-browser": { "command": "npx", "args": ["-y", "crawlio-browser"] } } }

Windsurf — add to Windsurf Settings > MCP:

{ "mcpServers": { "crawlio-browser": { "command": "npx", "args": ["-y", "crawlio-browser"] } } }

Cline (VS Code) — add to settings.json:

{ "cline.mcpServers": { "crawlio-browser": { "command": "npx", "args": ["-y", "crawlio-browser"] } } }

ChatGPT Desktop — Settings > Integrations > MCP: URL: http://127.0.0.1:3001/mcp | Type: Streamable HTTP

How It Works

AI Client (stdio/http)  -->  MCP Server (Node.js)  -->  Chrome Extension (MV3)
                             crawlio-browser               WebSocket -> CDP

The MCP server communicates with the Chrome extension via WebSocket. The extension controls the browser through Chrome DevTools Protocol (CDP).

Capabilities

Framework-Aware Intelligence

Every execute call probes the browser for framework signatures and injects a shape-shifting smart object with framework-native accessors. React state, Vue reactivity, Next.js routing, Shopify cart data — 17 framework namespaces across 4 tiers, detected at runtime and rebuilt on every navigation. The AI doesn't query a generic DOM; it queries the framework's own data structures.

Evidence-Based Analysis

Method Mode adds higher-order methods and a typed evidence system on top of Code Mode. smart.extractPage() runs 7 parallel operations in a single call — page capture, performance metrics, security state, font detection, meta extraction, accessibility audit, and mobile-readiness check. Failed operations produce typed CoverageGap records instead of silent nulls. Findings created with smart.finding() get their confidence automatically adjusted when supporting data is missing. The result: structured, auditable research output with gap tracking and confidence propagation.

Session Recording & Replay

Record browser interactions as structured data, then compile them into reusable SKILL.md automations. 12 interaction tools are automatically intercepted during recording — clicks, typing, navigation, scrolling — each capturing args, result, timing, and page URL. One compileRecording() call converts the session into a deterministic automation script.

Robot Training

Capture human-guided browser demonstrations as replayable robot-training bundles. The default-mode observe lifecycle starts an event-driven recorder inside the extension; collection keeps running if the MCP process disconnects or restarts. On reconnection, training_stop exports the retained run and materializes the complete 13-file RecordingBundle for replay and API synthesis. Full mode keeps the existing robot_training_* names as compatibility aliases.

Page monitoring is resident for the same reason: an extension-owned background tab and Chrome alarm collect bounded ARIA snapshots while no MCP server is present. Training and monitor history share a 25 MiB local budget, with 20 completed training runs, 50 monitor jobs, 200 snapshots total, and 50 snapshots per monitor as count caps. Old completed data is evicted first; active work is never silently evicted. Work starts only through explicit MCP lifecycle actions; the same actions report status, stop collection, clear monitor snapshots, and—with an exact id plus explicit confirmation—delete a stopped training/recording record from Chrome while preserving its materialized files. The extension popup remains a connection and browser-access status surface. Storage values are keys-only unless the caller explicitly opts in. Monitor snapshots intentionally retain compact ARIA page text locally; starting a monitor should therefore be treated as consent to retain the visible content of that URL until it is cleared.

Auto-Settling & Actionability

Every mutative action (click, type, navigate, select_option) runs actionability checks before acting — polling visibility, dimensions, enabled state, and overlay detection. After the action, a progressive backoff settle delay ([0, 20, 100, 100, 500]ms) waits for DOM mutations to quiesce. The AI doesn't need manual sleep() calls between actions.

Several Tabs at Once

Any command that acts on a page takes an optional tabId from list_tabs. Omit it and the command runs on the connected tab, exactly as before; supply one and it runs on that tab instead, with the whole command surface available on each. Commands overlap, so two tabs can be driven at the same time:

const [checkout, search] = await Promise.all([
  bridge.send({ type: "browser_snapshot", tabId: 42 }),
  bridge.send({ type: "browser_snapshot", tabId: 57 }),
]);

Targeting a tab never changes which tab connect_tab points at, so an agent working several tabs cannot reassign the one a human is watching. Frame selection, coverage sessions, and framework detection are per tab. Network capture is the exception — it records one tab at a time and says which tab holds it rather than interleaving two.

Chrome Profiles

An extension instance is confined to its own Chrome profile and cannot see any other, so with Crawlio enabled in more than one, commands land in whichever profile connected first. list_profiles shows the profiles that have connected and which is being driven; switch_profile moves the connection to another. One profile is driven at a time — the released extension reconnects in the background, so switching back is immediate.

Profiles identify themselves with a UUID minted into their own extension storage. It distinguishes a profile without describing it: no account, no email, no path, and no additional permission.

Selecting a profile keeps cooperating extensions out of each other's way — it is not a security boundary, since the id is asserted by the extension rather than proved. The bridge's existing protections are unchanged: one extension at a time, and each must prove the server holds the real bridge token before anything executes.

Architecture

A layered execution architecture where each layer absorbs a category of complexity that would otherwise fall on the model. The model sees four primary tools and a clean SDK. Everything beneath that surface is the runtime absorbing reality.

The layer worth understanding is the one that assembles itself. Detection runs against the live page on first use, and the smart object is built to match what that page turned out to be — smart.react.* exists only where React does. Nothing about the target is known at startup, so the surface is composed per tab rather than declared up front.

                     ┌───────────────────────────────────┐
                     │        AI Model (LLM)             │
                     │  Writes code, reads errors, loops  │
                     └───────────────┬───────────────────┘
                                     │  search, execute, observe, connect_tab (+ 3 job tools)
                                     ▼
┌─────────────────────────────────────────────────────────────────┐
│                     Crawlio Browser runtime                      │
│                                                                  │
│  ┌────────────────────────────────────────────────────────────┐  │
│  │  METHOD MODE                                               │  │
│  │  Behavioral protocol + higher-order methods                │  │
│  │  scrollCapture · waitForIdle · extractPage · comparePages  │  │
│  │  detectTables · extractTable · waitForNetworkIdle ·        │  │
│  │  extractData                                               │  │
│  │                                                            │  │
│  │  ↳ Absorbs: behavioral variance, ad-hoc composition,      │  │
│  │    inconsistent output shapes, data extraction patterns    │  │
│  ├────────────────────────────────────────────────────────────┤  │
│  │  POLYMORPHIC CONTEXT                                       │  │
│  │  17 framework namespaces, injected Just-In-Time            │  │
│  │  react · vue · angular · nextjs · shopify · ...            │  │
│  │                                                            │  │
│  │  ↳ Absorbs: framework opacity, minified code,             │  │
│  │    devtools hook complexity                                │  │
│  ├────────────────────────────────────────────────────────────┤  │
│  │  ACTIONABILITY ENGINE                                      │  │
│  │  7 core smart methods with built-in resilience             │  │
│  │  click · type · navigate · waitFor · evaluate ·            │  │
│  │  snapshot · rebuild                                        │  │
│  │                                                            │  │
│  │  ↳ Absorbs: DOM timing, hydration delays, CSS animations, │  │
│  │    disabled states, overlapping elements                   │  │
│  ├────────────────────────────────────────────────────────────┤  │
│  │  TETHERED IPC BRIDGE                                       │  │
│  │  WebSocket ↔ Chrome extension, message queue,              │  │
│  │  heartbeat, auto-reconnect, stale detection                │  │
│  │                                                            │  │
│  │  ↳ Absorbs: connection drops, tab refreshes,              │  │
│  │    port conflicts, extension lifecycle                     │  │
│  ├────────────────────────────────────────────────────────────┤  │
│  │  COMMAND CHANNEL                                           │  │
│  │  bridge.send → CDP browser control via the extension       │  │
│  │  crawlio.*   → Crawlio HTTP endpoints                      │  │
│  │  Live searchable catalog: browser + Crawlio HTTP           │  │
│  └────────────────────────────────────────────────────────────┘  │
└─────────────────────────────────────────────────────────────────┘
                                     │
                                     ▼
                     ┌───────────────────────────────────┐
                     │         Live Chrome Browser        │
                     │   Persistent session, real DOM,    │
                     │   framework runtime, user state    │
                     └───────────────────────────────────┘

What Each Layer Absorbs

Execution Lifecycle

1. Discovery — Model calls search("page capture performance") and gets documentation for relevant commands
2. Framework Detection — Runtime probes the live DOM, detects active frameworks, constructs polymorphic smart object with appropriate namespaces
3. Scope Assembly — Model's code is compiled into an async function with injected parameters: bridge (the browser command channel), crawlio (HTTP client), sleep, TIMEOUTS, smart (7 core + 18 higher-order methods + up to 17 framework namespaces), compileRecording
4. Execution — Method Mode methods compose the lower layers: extractPage() fires 7 parallel bridge.send() calls; click() runs the actionability engine; react.getVersion() evaluates framework-specific expressions
5. Error Recovery (Agentic REPL) — On failure, the browser stays in the exact state that produced the error. The model reads the structured error, adjusts, and calls execute again. Framework cache persists — no re-detection unless URL changed

Design Principles

1. Absorb complexity downward — Every category of difficulty (connection management, DOM timing, framework detection, multi-step composition) is handled by the layer best equipped for it. The model only encounters the clean interface at the top.
2. Shape the SDK to the target — The polymorphic context system detects what the page is and reshapes available methods to match. The model writes against a stable interface; the runtime adapts underneath.
3. Preserve state across cycles — The tethered architecture means the model can fail, learn, and retry against the same live environment — transforming error handling from "restart from scratch" into "adjust and continue."

How It Compares

Code Mode is Cloudflare's idea and a good one: present tools as a typed API and let the model write code against it, because models have seen far more code than tool calls. Crawlio Browser applies that pattern to a target it was not built for — a live browser holding your session. This is an MCP server, not an alternative to MCP.