pagesift

pagesift fetches a URL (rendering JavaScript only when the page needs it) or reads a local file, keeps the main content and returns Markdown plus metadata. It runs as a CLI, an HTTP service, an MCP server, or a Python or Node library, and scrapes single pages, batches or whole sites.

Why it matters: it speaks the same v1 and v2 API as Firecrawl and offers a Jina Reader endpoint, so it can sit alongside or replace those without rewriting your code. It respects robots.txt and per-host rate limits by default. It does not do OCR: scanned PDFs and images are out of scope.

Rust, about 185 stars at the time of listing. AGPL-3.0 or a commercial license, so check the terms before building it into a product.