Open-Source Web Scrapers for LLMs

Firecrawl turns web pages into content that developers can use in AI apps and agents. Its self-hosted API suits teams that want to collect web data on infrastructure they control.

Obscura is a headless browser engine for developers building AI agents and web scraping systems with many independent sessions. Each session starts with fresh state, keeping cookies, logins and cached data separate between tasks.

Scrapling is a Python web scraping framework for developers collecting data from individual pages or crawling whole sites. Its adaptive parser can relocate selected elements when a website's layout changes.

AI Job Search is a job application framework for people who want to assess roles and prepare tailored documents from their own career profile. It runs through Claude Code on your machine, but AI work uses Anthropic's service and requires paid Claude access or API credits, so it isn't fully offline.

Agent Reach gives AI agents access to online content for research, repository work and social media searches. It's for people using Claude Code, OpenClaw, Cursor, Windsurf or another agent that can run shell commands.

xberg, formerly Kreuzberg, is a local document extraction engine for developers building AI search and document processing applications. It turns files into text and structured content that those applications can use.

CyberScraper 2077 turns web pages into structured data through a conversational interface. It's for analysts and researchers who want to describe the information they need and export the results.

Stagehand is an open-source browser agent SDK for developers automating websites and collecting structured data in local Chrome. Model inference is separate from browser execution: direct provider calls send model context remotely, while a custom callback can connect a self-hosted model. Local runs require Chrome.

only-cli is a local web browsing CLI for AI agents that need to read websites without filling their context with raw HTML or screenshots. It suits coding assistants such as Claude Code and Codex that look up documentation, inspect repositories or read online discussions.

MCP Reference Servers gives developers examples for connecting AI applications to tools and data. Maintained by the MCP steering group, these are educational implementations, not production-ready servers.

Browser4 is a browser automation engine for developers building AI agents, web research tools, and data collection workflows. It controls real Chrome browsers and handles interactive pages that require clicks, forms, or JavaScript before their data is accessible.

ZeroToken is a local browser automation server for people using AI agents to repeat web tasks such as collecting reports, logging into sites or testing browser workflows. It turns recorded actions into scripts so later runs don't need an LLM to reason through every step again.

CamoFox Browser is a self-hosted browser server for developers whose AI agents need to interact with sites that detect browser automation. It builds on Camoufox, a Firefox fork with browser fingerprint spoofing, to reduce detection by systems such as Cloudflare and Google's bot checks.

Damru is a self-hosted Python framework for developers doing mobile web scraping, automation testing, or browser fingerprinting research. It controls Chrome inside a containerized Android environment; physical Android phones aren't supported.

ScrapeGraphAI extracts data from websites and documents based on a plain-language request. Developers can use its self-hosted Python library or call a separate managed cloud API.

BrowserWing is a local browser automation tool for developers and AI agent users who need repeatable website tasks and structured data. It lets you record a workflow once and make it available to agents as a reusable action.

search2ai is a self-hosted search gateway for developers adding web access to AI agents and chat clients. It connects your chosen search services through one interface, with backup providers when requests fail.

wigolo gives AI agents web access from a server on your own machine. It's for developers who want coding assistants or self-hosted agents to gather web evidence and reuse what they've already fetched.

Steel is a browser API for developers building AI agents that interact with websites. It supplies real Chrome sessions for tasks involving clicks, forms, logins and pages that rely on JavaScript.

OpenBrowser is a self-hosted browser automation agent for people who need visual judgment and interaction across web pages. Its server and Chrome extension run locally, but the documented Qwen setup sends model requests to DashScope's cloud API and requires an API key and internet access. Browser execution stays local; model processing uses the remote provider.

bb-browser gives AI agents access to your local Chrome session, including the website accounts you're already signed into. It's for people building agents that need to work with authenticated websites without a separate API integration for each service.

webctl gives developers and AI agents command-line control of a browser for website tasks. It uses Chromium through Playwright and connects with coding assistants through skills and prompts.

Fortress is a self-hosted Chromium fork for developers whose scrapers and AI agents encounter browser fingerprint checks. It changes fingerprint behavior inside the browser engine to help automated sessions access protected pages.

GPT Crawler collects website text into knowledge files for people building AI assistants around documentation or other web content. It prepares the material a custom GPT or assistant can use to answer questions about a site.

Crawl4AI is a web crawler and scraper for developers feeding websites into AI agents, RAG systems and data pipelines. It can extract useful page content without requiring an LLM.

OpenCLI lets AI agents and terminal users work with websites through their existing Chrome sessions. It's for people who want agents to interact with pages or repeat website tasks through reusable commands.

Moli is a headless browser for developers building AI agents, web scrapers and retrieval pipelines on their own machines or servers. It reads page content and executes JavaScript without continuously rendering a visual page.

Skill Seekers prepares documentation and code for AI assistants. It's for developers and teams who want reusable skills, coding context or retrieval-augmented generation (RAG) data without preparing the same sources separately for each tool.

Anakin is a web scraping API for developers feeding website content into AI agents, RAG pipelines, or data extraction workflows. You can run its scraping engine on your own server or use the hosted service.

Webcmd is a browser automation tool for developers whose AI agents repeatedly work with the same websites. It keeps what agents learn locally so later tasks can use familiar workflows instead of inspecting each site from scratch.

Jina Reader is a web extraction API for developers building AI agents, search tools and systems that answer questions from retrieved documents. It turns page content into Markdown or JSON instead of passing raw HTML to an LLM.