Scraping APIs for AI agents

Bright Data Web Unlocker

Diffbot — Extract STRUCTURED, typed data from a URL: it classifies the page (article / product / discussion / image) and returns parsed fields — title…

Diffbot — Same structured extraction as diffbot/analyze, but over HTML YOU already fetched rather than a URL Diffbot fetches itself. This is the pairing for…

CRW — Start an ASYNC multi-page crawl from a seed URL, following links. Pass `url`; optional `maxPages` (1-100, default 100 — vendor bills 1 credit/page so…

CRW — Poll an async crawl started by crw/crawl. Pass `id` (from the crawl response). Returns `{ status: scraping|completed|failed|cancelled, completed, total…

CRW — Structured extraction over up to 10 URLs using an LLM. Pass `urls` plus `prompt` (natural language) and/or `schema` (JSON schema); optional `basis…

CRW — Poll an async extraction started by crw/extract, on the rare occasions it returns an `id` instead of inline results. Pass `id`. Returns `{ status…

CRW — Discover the URLs of a website without scraping content (sitemap + crawl fallback). Pass `url`; optional `maxDepth`, `useSitemap` (default true)…

CRW — Scrape a single URL to clean markdown/HTML/JSON (Firecrawl-compatible). Pass `url`; optional `formats`…

CRW — Search the web and optionally scrape the hits in one call. Pass `query`; optional `limit` (1-20, default 5), `tbs` (freshness: qdr:h|d|w|m|y), `sources`…

Firecrawl — Crawl a website starting from a URL, following links.

Firecrawl — Extract structured data from URLs using a schema.

Firecrawl — Map all URLs on a website without scraping content.

Firecrawl — Scrape a single URL and return clean markdown/HTML.

Firecrawl — Search the web and return scraped results.

Jina Reader — fetch a URL and return LLM-ready markdown (r.jina.ai). Pass `url`. Handles JS rendering and boilerplate stripping automatically; returns `{…

Jina Search — web search that returns the top hits WITH their full reader-processed page content in one call (s.jina.ai). Pass `q`; optional `num` (result…

Olostep — ask AI search engines the prompts your customers ask, and get back what each one actually answered: the answer text, the sources it cited, and any…

Olostep Answers — ask a question or hand it a data point to enrich; it searches and reads live pages, validates, and answers with sources in 3-30s. Pass…

Olostep — scrape up to 100 URLs as one ASYNC batch (1-3 min for small batches, 5-8 min for large). Pass `items` ([{ url, custom_id? }]; custom_id defaults to…

Olostep — the items of a completed batch, WITH content inline. Pass `id` ("batch_…"); optional `formats` (default ["markdown"]; use ["json"] for parser output…

Olostep — poll a batch started by olostep/batch or olostep/ai-visibility. Pass `id` ("batch_…"). Returns `{ status: in_progress|completed, total_urls…

Olostep — start an ASYNC crawl from `start_url`, following links. Required `max_pages` (1-100; it sets the price). Optional `include_urls` / `exclude_urls`…

Olostep — the pages of a crawl, WITH their content inline. Pass `id` ("crawl_…"); optional `formats` (default ["markdown"]; add "html" for source), `limit`…

Olostep — poll a crawl started by olostep/crawl. Pass `id` ("crawl_…"). Returns `{ status: in_progress|completed, pages_count, credits_consumed }`. Free. Poll…

Olostep — list the URLs of a website without scraping them. Pass `url`; optional `search_query` + `top_n` (rank and keep the most relevant), `include_urls` /…

Olostep — fetch one stored page by `retrieve_id` (from olostep/scrape, crawl-pages or batch-items). Optional `formats` (markdown | html | json). Use it for a…

Olostep — scrape one URL (JS rendered, residential IPs) in about a second. Pass `url_to_scrape`; optional `formats` (markdown default | html | text | json |…

Olostep — natural-language web search returning deduplicated links with title and description. Pass `query`; optional `limit` (1-25, default 12)…

Scrape.do — Fetch a page through a rotating datacenter-proxy pool with anti-bot handling. Surprisingly strong for the price: it returned the REAL page on a…

Scrape.do — The heavy rung: RESIDENTIAL/mobile proxy pool plus full JS rendering (`super` + `render`). For pages that bounce the plain scrapedo/scrape call…

ScrapingAnt — AI data extraction WITHOUT a schema: describe the fields in plain English and get structured JSON back. Pass `url` and `extract_properties` — a…

ScrapingAnt — Scrape a URL and return LLM-ready markdown (rendered in headless Chrome, then converted). Pass `url`; optional…

ScrapingAnt — Scrape a URL through a managed headless-Chrome cluster (datacenter proxies). Pass `url`; optional `browser` (default true — set false for plain…

ScrapingAnt — Scrape a HARD page through the 3M+ residential-proxy pool + headless Chrome: Cloudflare and anti-bot walls, geo-fenced content, sites that block…