# WebCrawlerAPI — how to use (mcp.ai)

Connect your WebCrawlerAPI account and use 10 tools for web scraping straight from your AI agent. Connect with your own API key. WebCrawlerAPI provides crawling, page scraping, AI web agents, and monitored website feeds through a unified API.

## Option A — via MCP (recommended)
Remote MCP endpoint (HTTP, streamable): `https://api.mcp.ai/p_webcrawlerapi?ms=1787293680000`
Add it as a custom/remote MCP connector in your client (Claude, Cursor, VS Code…), then authenticate when prompted. Once connected, ask the agent to use the server's tools (e.g. `webcrawlerapi_create_crawl`).

## Option B — via direct REST API
Base URL: `https://api.mcp.ai/api/webcrawlerapi`
Auth: `Authorization: Bearer sk_live_…` — create a workspace API key at https://mcp.ai/settings/api-keys
Discover endpoints: `GET https://api.mcp.ai/api/webcrawlerapi/_endpoints`

### Endpoints
- `POST https://api.mcp.ai/api/webcrawlerapi/create/crawl` — Start a metered asynchronous crawl over a site and return its job ID. The crawl may visit and charge for up to items_limit pages; use GET_CRAWL_JOB to monitor it.
  - body: { url: string, max_age?: integer, max_depth?: integer, items_limit?: integer, webhook_url?: string, output_formats?: string[], blacklist_regexp?: string, whitelist_regexp?: string, keep_query_params?: boolean, main_content_only?: boolean, respect_robots_txt?: boolean }
- `POST https://api.mcp.ai/api/webcrawlerapi/create/feed` — Create a recurring website-change feed. Creation immediately schedules an initial crawl that may consume metered credits, and future scheduled runs may also consume credits; delete the feed when it is
  - body: { url: string, name?: string, max_depth?: integer, items_limit?: integer, webhook_url?: string, output_format?: string, include_errors?: boolean, blacklist_regexp?: string, whitelist_regexp?: string, main_content_only?: boolean, respect_robots_txt?: boolean }
- `POST https://api.mcp.ai/api/webcrawlerapi/delete/feed` — Permanently cancel a feed so it cannot be resumed. The feed is removed from feed listings, but retained history may remain directly readable for 30 days.
  - body: { feed_id: string }
- `POST https://api.mcp.ai/api/webcrawlerapi/get/crawl/job` — Get a crawl job's status, configuration, per-page results, content URLs, errors, and observed costs. Respect recommended_pull_delay_ms when polling.
  - body: { crawl_id: string }
- `POST https://api.mcp.ai/api/webcrawlerapi/get/feed` — Get one feed's configuration, lifecycle status, and recent run history, including per-run crawl counts and cost. Canceled feeds remain directly readable while retained by the provider.
  - body: { feed_id: string }
- `POST https://api.mcp.ai/api/webcrawlerapi/get/organization/costs` — Return current spendable balance plus request count and USD usage for a date range. The connected API key is used for both organization routes.
  - body: { to_date?: string, from_date?: string, include_daily?: boolean }
- `POST https://api.mcp.ai/api/webcrawlerapi/list/feed/changes` — Return one page of detected feed changes as structured JSON, with an opaque continuation cursor for older pages.
  - body: { feed_id: string, page_size?: integer, next_cursor?: string }
- `POST https://api.mcp.ai/api/webcrawlerapi/list/feeds` — List active and paused feeds for the connected organization, newest first. Logically canceled feeds are omitted.
- `POST https://api.mcp.ai/api/webcrawlerapi/scrape/page` — Scrape one web page synchronously and return requested content or structured extraction. This consumes metered credits; supplying a prompt adds $0.002 per request.
  - body: { url: string, prompt?: string, max_age?: integer, output_formats?: string[], clean_selectors?: string, response_schema?: object, keep_query_params?: boolean, main_content_only?: boolean, respect_robots_txt?: boolean }
- `POST https://api.mcp.ai/api/webcrawlerapi/set/feed/state` — Pause an active feed's future scheduled runs or resume a paused feed. Pausing does not stop a run that has already started, and resuming schedules future crawls that may consume metered credits.
  - body: { feed_id: string, desired_state: string }

## Example prompts
- "What can I do in WebCrawlerAPI?"
- "Show me a summary of my WebCrawlerAPI account"

## More
- Page: https://mcp.ai/webcrawlerapi
- Agent spec (llms.txt): https://mcp.ai/webcrawlerapi/llms.txt
- Postman collection: https://mcp.ai/webcrawlerapi/postman.json