Open sourcePython · ≈88k stars · Apache-2.0
Crawl4AI
Open-source, LLM-friendly web crawler.
Crawl4AI crawls and renders pages into clean markdown and structured formats optimized for LLMs and embeddings — a free, self-hostable alternative to commercial scraping APIs.
Pros & cons
Each point cites its sources and says what kind of evidence backs it. A second reviewer tried to refute every claim before it was kept.
Pros
- DocsApache-2.0, LLM-friendly markdown output, with schema-based CSS/XPath extraction that needs no LLM, and LLM extraction via any LiteLLM provider.[1]
- User reportsPopular in home-lab agent stacks (with SearXNG and MetaMCP) for free, local web retrieval.[1][2][3]
- DocsFull browser control (persistent profiles, CDP, proxies, stealth, Chromium/Firefox/WebKit), plus deep and adaptive crawling.[1]
- DocsOptional cloud API with 1,000 free credits a month and paid plans from $10.[1]
Cons
- Independent testCVE-2026-26216 (CVSS 10) allowed unauthenticated code execution through the Docker API's hooks feature in versions before 0.8.0, which removed __import__ from hook builtins and disabled hooks by default.[1]
- DocsThe README asks for attribution (badges or a citation), which goes beyond plain Apache-2.0 use.[1]
- User reportsUsers say it takes a fair amount of glue code to integrate, and that it does not pierce shadow DOM.[1][2]
- User reportsBot defences like Cloudflare increasingly block crawlers of this kind.[1]
- Best for
- Self-hosters who want a free, Python-native crawler that turns pages into LLM-ready markdown.
- Watch out for
- If you run the Docker API, stay on 0.8.0 or later, set an API token and keep it off the public internet.
Assessed Oct 29, 2026 · adversarially reviewedSuggest a correction
People
Founder
Creator of Crawl4AI
Harness engineering
Activity
- Last push
- · yesterday
- Latest release
- v0.9.6
· yesterday - Last 90 days
- 4 releases
v0.9.6Release v0.9.6
· yesterday🎉 Crawl4AI v0.9.6 Released! 📦 Installation PyPI: Docker: Note: Docker images are being built and will be available shortly. Check the Docker Release workflow for build status. 📝 What's Changed See CHANGELOG.md for details.
v0.9.5Release v0.9.5
· 5w ago🎉 Crawl4AI v0.9.5 Released! 📦 Installation PyPI: Docker: Note: Docker images are being built and will be available shortly. Check the Docker Release workflow for build status. 📝 What's Changed See CHANGELOG.md for details.
v0.9.4Release v0.9.4
· 8w ago🎉 Crawl4AI v0.9.4 Released! 📦 Installation PyPI: Docker: Note: Docker images are being built and will be available shortly. Check the Docker Release workflow for build status. 📝 What's Changed See CHANGELOG.md for details.
v0.9.3Release v0.9.3
· 2mo ago🎉 Crawl4AI v0.9.3 Released! 📦 Installation PyPI: Docker: Note: Docker images are being built and will be available shortly. Check the Docker Release workflow for build status. 📝 What's Changed See CHANGELOG.md for details.
v0.9.2Release v0.9.2
· 4mo ago🎉 Crawl4AI v0.9.2 Released! 📦 Installation PyPI: Docker: Note: Docker images are being built and will be available shortly. Check the Docker Release workflow for build status. 📝 What's Changed See CHANGELOG.md for details.
Checked against GitHub on Nov 21, 2026. Release notes are shortened to plain text.
Neighbours on the workbench
TypeScript · ≈205k stars · AGPL-3.0
firecrawl/firecrawl
Turn websites into clean markdown for LLMs and agents.
Maintained by Firecrawl (formerly Mendable)
Python · ≈150k stars · MIT
langchain-ai/langchain
The most widely used framework for LLM-powered applications.
Maintained by LangChain
langchain-openai==1.8.5releasedGo · ≈94k stars · Apache-2.0
infiniflow/ragflow
Agent-native RAG engine and context infrastructure.
Maintained by infiniflow
Connections
Relations from the Harness Hub knowledge graph, as charted in the State of Harness reports.
