# Crawlbase — how to use (mcp.ai)

Connect your Crawlbase account and use 8 tools for web scraping straight from your AI agent. Connect with your own API key. Crawlbase provides APIs for crawling web pages, managing crawler queues, storing crawl results, and monitoring account usage.

## Option A — via MCP (recommended)
Remote MCP endpoint (HTTP, streamable): `https://api.mcp.ai/p_crawlbase?ms=1787291520000`
Add it as a custom/remote MCP connector in your client (Claude, Cursor, VS Code…), then authenticate when prompted. Once connected, ask the agent to use the server's tools (e.g. `crawlbase_crawl_url`).

## Option B — via direct REST API
Base URL: `https://api.mcp.ai/api/crawlbase`
Auth: `Authorization: Bearer sk_live_…` — create a workspace API key at https://mcp.ai/settings/api-keys
Discover endpoints: `GET https://api.mcp.ai/api/crawlbase/_endpoints`

### Endpoints
- `POST https://api.mcp.ai/api/crawlbase/crawl/url` — Fetch a public HTTP or HTTPS URL through Crawlbase using the connected Normal token and return its content with separate target and Crawlbase statuses.
  - body: { url: string, store?: boolean, device?: string, cookies?: string, country?: string, scraper?: string, autoparse?: boolean, user_agent?: string, tor_network?: boolean, output_format?: string, cookie_session?: string, request_headers?: object, custom_success_codes?: integer[] }
- `POST https://api.mcp.ai/api/crawlbase/delete/stored/pages` — Irreversibly delete explicitly selected pages from Crawlbase Cloud Storage by request ID and return each page's deletion outcome.
  - body: { rids: string[] }
- `POST https://api.mcp.ai/api/crawlbase/generate/user/agents` — Generate one to ten realistic randomized User-Agent strings for a selected device class.
  - body: { size?: integer, device?: string }
- `POST https://api.mcp.ai/api/crawlbase/get/crawling/usage` — Return current Crawling API usage and optionally previous-month statistics for the connected Crawlbase account.
  - body: { include_previous_month?: boolean }
- `POST https://api.mcp.ai/api/crawlbase/get/storage/count` — Return the number of pages currently held in Crawlbase Cloud Storage.
- `POST https://api.mcp.ai/api/crawlbase/get/stored/page` — Retrieve the latest Crawlbase Cloud Storage page by exactly one request ID or original URL.
  - body: { rid?: string, url?: string }
- `POST https://api.mcp.ai/api/crawlbase/get/stored/pages` — Retrieve up to 100 Crawlbase Cloud Storage pages by request ID without deleting them. Returns decoded page content and identifies requested IDs that were not found.
  - body: { rids: string[] }
- `POST https://api.mcp.ai/api/crawlbase/list/stored/page/ids` — Return one page of Crawlbase Cloud Storage request IDs and a short-lived continuation cursor.
  - body: { limit?: integer, order?: string, next_cursor?: string }

## Example prompts
- "What can I do in Crawlbase?"
- "Show me a summary of my Crawlbase account"

## More
- Page: https://mcp.ai/crawlbase
- Agent spec (llms.txt): https://mcp.ai/crawlbase/llms.txt
- Postman collection: https://mcp.ai/crawlbase/postman.json