AnyCrawl: A Web Scraping Toolkit That Turns Pages into LLM-Ready Data

Web scraping usually forces a trade-off: static pages want speed, while dynamically rendered pages need a real browser environment. AnyCrawl folds both needs into a single toolkit — a Node.js project written in TypeScript that picks between a fast static parser and a browser-rendering engine as needed, then formats whatever it scrapes into data an LLM can use directly. GitHubDaily recommended it back in June 2025.

Core features

• Three scraping engines: the Scrape endpoint bundles cheerio, playwright, and puppeteer. Static HTML goes through cheerio for the fastest parsing, while pages that need JavaScript rendering get handed off to the other two.
• Search-result collection: the /v1/search endpoint returns structured results directly, with configurable pagination and language parameters; per the README, Google is currently the only supported source.
• Full-site crawling and bulk scraping: Crawl can cap crawl depth, item count, and restrict to the same domain; Batch Scrape submits a list of known URLs as a single async job, which you then poll and page through to retrieve results.
• LLM structured extraction: pass a JSON Schema via the json_options parameter and an LLM will extract structured JSON from the page accordingly; model providers can be OpenAI-compatible services like Atlas Cloud.
• Multi-threaded concurrency: native multi-threading/multi-processing boosts throughput for batch jobs, with Redis also part of the stack.
• Proxies and caching: each request can carry an HTTP/SOCKS proxy, and cache lifetime plus whether to write to cache are both controlled per request.

Typical use cases

• Preparing corpora for LLM or RAG applications by converting web content into markdown and other model-friendly formats.
• Bulk-pulling Google search results for SEO analysis or market research.
• Crawling an entire site under depth and same-domain rules to build a dataset or archive.

Quick start

You can use the official hosted API or self-host with Docker. For self-hosting, first enable auth and generate an API key:

docker compose exec api pnpm --filter api key:generate

Then call the endpoint to scrape a page (swap in your own server address if self-hosting):

curl -X POST https://api.anycrawl.dev/v1/scrape \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_ANYCRAWL_API_KEY" \
  -d '{"url": "https://example.com", "engine": "cheerio"}'

Summary

AnyCrawl targets developers and AI teams who need web and search data in bulk: if your stack is mostly Node.js and TypeScript, or you’re preparing corpora for a RAG application, it’s ready to plug in as-is. The project is MIT-licensed and still actively maintained. A few practical caveats: search support is currently Google-only per the README; self-hosting depends on Redis, the browser engines require Node.js 20 or newer, and first use downloads a roughly 70MB GeoIP database; and LLM extraction still requires you to configure your own OpenAI-compatible provider and API key. Overall, its layered engine design lets static and dynamic pages each take the path best suited to them.