Crawl4AI
unclecode/crawl4ai
Open-source, LLM-friendly web crawler.
Crawl4AI crawls and renders pages into clean markdown and structured formats optimized for LLMs and embeddings — a free, self-hostable alternative to commercial scraping APIs.
Related projects
Firecrawl
firecrawl/firecrawl
Turn websites into clean markdown for LLMs and agents.
Firecrawl crawls, scrapes and converts entire sites into LLM-ready markdown with structure — the go-to ingestion primitive for agent context pipelines.
LangChain
langchain-ai/langchain
The most widely used framework for LLM-powered applications.
LangChain provides chains, tool abstractions, integrations and runnables for composing LLM applications. It became the default on-ramp for agent development and remains among the most-starred projects in the space.
Headroom
headroomlabs-ai/headroom
Context compression for coding agents.
Headroom compresses tool outputs, logs, files and RAG chunks before they reach the LLM — 20% fewer tokens for coding agents and 60–95% for other workloads. Context engineering as infrastructure: keeping the window usable as harnesses grow.