Crawl4AI

Crawl4AI is a web crawler and scraper for developers feeding websites into AI agents, RAG systems and data pipelines. It can extract useful page content without requiring an LLM.

• Markdown extraction: Preserve headings, tables and code blocks, filter page clutter or content unrelated to a query, and retain links as numbered references.
• Structured records: Extract JSON with CSS, XPath or regex rules, or use an open-source or hosted LLM provider to produce records matching a schema.
• Browser-based scraping: Process JavaScript pages and scrolling content with Chromium, Firefox or WebKit, and reuse saved browser profiles and login sessions.
• Site crawling: Follow links across a site, resume interrupted crawls and cache fetched pages; raw HTML and local files are also accepted.
• Page captures: Collect screenshots, PDFs, media metadata and links alongside extracted text.

The free, open-source crawler uses Apache 2.0 and runs as a Python library or Docker server on your hardware. Self-hosting keeps browser execution under your control; hosted LLM extraction sends content to the chosen provider.

The paid cloud service runs the browsers on Crawl4AI's infrastructure and handles proxies, queues and retries. Web search and remote MCP access are cloud capabilities, with clients including Claude Code, Codex and Cursor. Cloud extraction doesn't require your own LLM provider key.