Harness Hub
All open source

Crawl4AI

Open-source, LLM-friendly web crawler.

unclecode/crawl4ai

Maintained by Unclecode

Crawl4AI crawls and renders pages into clean markdown and structured formats optimized for LLMs and embeddings — a free, self-hostable alternative to commercial scraping APIs.

View on GitHubDocs

Pros & cons

Each point cites its sources and says what kind of evidence backs it. A second reviewer tried to refute every claim before it was kept.

Pros

  • DocsApache-2.0, LLM-friendly markdown output, with schema-based CSS/XPath extraction that needs no LLM, and LLM extraction via any LiteLLM provider.[1]
  • User reportsPopular in home-lab agent stacks (with SearXNG and MetaMCP) for free, local web retrieval.[1][2][3]
  • DocsFull browser control (persistent profiles, CDP, proxies, stealth, Chromium/Firefox/WebKit), plus deep and adaptive crawling.[1]
  • DocsOptional cloud API with 1,000 free credits a month and paid plans from $10.[1]

Cons

  • Independent testCVE-2026-26216 (CVSS 10) allowed unauthenticated code execution through the Docker API's hooks feature in versions before 0.8.0, which removed __import__ from hook builtins and disabled hooks by default.[1]
  • DocsThe README asks for attribution (badges or a citation), which goes beyond plain Apache-2.0 use.[1]
  • User reportsUsers say it takes a fair amount of glue code to integrate, and that it does not pierce shadow DOM.[1][2]
  • User reportsBot defences like Cloudflare increasingly block crawlers of this kind.[1]
Best for
Self-hosters who want a free, Python-native crawler that turns pages into LLM-ready markdown.
Watch out for
If you run the Docker API, stay on 0.8.0 or later, set an API token and keep it off the public internet.

Assessed Oct 29, 2026 · adversarially reviewedSuggest a correction

People

Creator of Crawl4AI

Harness engineering

Activity

Last push
· yesterday
Latest release
v0.9.6
· yesterday
Last 90 days
4 releases
  1. v0.9.6Release v0.9.6

    · yesterday

    🎉 Crawl4AI v0.9.6 Released! 📦 Installation PyPI: Docker: Note: Docker images are being built and will be available shortly. Check the Docker Release workflow for build status. 📝 What's Changed See CHANGELOG.md for details.

  2. v0.9.5Release v0.9.5

    · 5w ago

    🎉 Crawl4AI v0.9.5 Released! 📦 Installation PyPI: Docker: Note: Docker images are being built and will be available shortly. Check the Docker Release workflow for build status. 📝 What's Changed See CHANGELOG.md for details.

  3. v0.9.4Release v0.9.4

    · 8w ago

    🎉 Crawl4AI v0.9.4 Released! 📦 Installation PyPI: Docker: Note: Docker images are being built and will be available shortly. Check the Docker Release workflow for build status. 📝 What's Changed See CHANGELOG.md for details.

  4. v0.9.3Release v0.9.3

    · 2mo ago

    🎉 Crawl4AI v0.9.3 Released! 📦 Installation PyPI: Docker: Note: Docker images are being built and will be available shortly. Check the Docker Release workflow for build status. 📝 What's Changed See CHANGELOG.md for details.

  5. v0.9.2Release v0.9.2

    · 4mo ago

    🎉 Crawl4AI v0.9.2 Released! 📦 Installation PyPI: Docker: Note: Docker images are being built and will be available shortly. Check the Docker Release workflow for build status. 📝 What's Changed See CHANGELOG.md for details.

Checked against GitHub on Nov 21, 2026. Release notes are shortened to plain text.

firecrawl/firecrawl

Turn websites into clean markdown for LLMs and agents.

Maintained by Firecrawl (formerly Mendable)

4 pros · 4 cons

langchain-ai/langchain

The most widely used framework for LLM-powered applications.

Maintained by LangChain

langchain-openai==1.8.5released
3 pros · 4 cons

infiniflow/ragflow

Agent-native RAG engine and context infrastructure.

Maintained by infiniflow

Connections

Relations from the Harness Hub knowledge graph, as charted in the State of Harness reports.

Maintained by · 1

  • UnclecodeOrganisation

Similar entries · 2

The Harness Hub newsletter

New harness companies, projects, releases and the monthly State of Harness report, in your inbox.