Context.dev Launches: Agents Can Scrape Websites Without Writing Parsers Anymore

Context.dev Turns “Web Scraping” Into a POST Request

On July 9, YC S26 startup Context.dev posted its Launch thread on Hacker News, officially opening its structured web data API to the public. The product positioning can be summed up in one sentence: give it a URL, and it returns clean Markdown, raw HTML, image lists, or even a full-site crawl result — built for Agent and RAG developers, with an OpenAI-style philosophy of handling every scenario through a single endpoint.

This is not just another generic crawler. Over the past two years, products like Firecrawl, Jina Reader, and Apify’s RAG Web Browser have already explored the “convert web pages to Markdown” path extensively. Context.dev enters the space with a more precise positioning: it compresses the entire crawling workflow into a few atomic endpoints highly aligned with LLM consumption patterns, so developers no longer need to build middleware between “crawl one page” and “crawl an entire site.”

Four Endpoints Cover 90% of Agent Web-Scraping Scenarios

A quick look at its currently available capabilities reveals a very clear logic:

• Extract Raw HTML: Fetch fully rendered HTML from a URL, handling long-standing problems like SPAs, infinite scroll, and cookie popups;
• Extract Images: Extract all image resources from a page and return a list of URLs;
• Discover URLs: Feed it a domain and get all reachable page URLs under the site — essentially sitemap reconstruction;
• Crawl Website: Start from any URL, crawl up to 500 pages, and convert everything into Markdown, with controls for crawl depth, page limits, URL filtering rules, and whether to follow subdomains.

The last one is the main attraction. Previously, achieving the same thing required stitching together three layers yourself: Playwright/Puppeteer for rendering, readability.js or a pile of handwritten rules for content extraction, and another layer like turndown to convert HTML into Markdown. You also had to handle concurrency, retries, robots.txt, and proxy pools. Building the full pipeline could easily consume two weeks of engineering time.

Context.dev compresses all of this into a single POST request with just a few control parameters:

{
  "url": "https://example.com/docs",
  "maxPages": 500,
  "maxDepth": 3,
  "includePatterns": ["/docs/*"],
  "followSubdomains": false
}

The response body returns an array of Markdown pages. This granularity lands exactly in the sweet spot for Agent consumption — not fragmented chunks that need reassembly, and not dirty raw HTML that still needs cleaning.

Why This Is Only Becoming a Business Now

The idea of abstracting “web scraping” into an API is not new. The real turning point came after the explosion of the Agent ecosystem over the past year, when the shape of scraping demand fundamentally changed.

In the chatbot era, web scraping was an offline task: run through a sitemap overnight, clean the content into a vector database, and perform retrieval during the day. You had time to perfect your crawler. The Agent era is different — it’s online, on-demand, and random. A user drops in a link, and the Agent has to determine within hundreds of milliseconds whether it’s a documentation site or an e-commerce page, then decide whether to crawl a single page or an entire /docs/ directory. Crawling itself becomes a Tool Call within the reasoning chain.

The Manus team recently made a striking observation in their article “Context Engineering for AI Agents”: a typical Agent task requires around 50 tool calls on average, with an input/output token ratio of 100:1. This means that every delay and every failure in a scraping tool gets amplified dozens of times through the Agent loop. Your crawler can no longer be something that “occasionally fails and users tolerate it” — it has to be as reliable as a cloud function call.

Building this infrastructure in-house is becoming increasingly uneconomical:

1. Anti-bot countermeasures are getting more expensive. Cloudflare, Datadome, and Akamai keep upgrading bot detection, and areas like fingerprint spoofing, residential proxies, and TLS fingerprints have become specialized industries;
2. JavaScript rendering is costly. Running a headless Chrome instance starts at around 200MB of memory, and at scale even GPUs become insufficient;
3. Markdown conversion quality is unstable. readability.js hasn’t been updated in a decade, and modern frontend frameworks often cause it to extract navigation bars and footers instead of content.

As a result, even technically capable teams are increasingly willing to outsource this layer. That’s the underlying logic behind Firecrawl reaching millions in annual recurring revenue within just over a year, and also why Context.dev believes there’s room to enter the market.

Compared With Firecrawl and Jina Reader, What Differentiates Context.dev?

Here’s a direct comparison:

| Product | Core Positioning | Billing Unit | Highlights | |------|----------|----------|------| | Firecrawl | Full-stack crawling + extraction | Per page/API call | JSON schema extraction, batch task queues | | Jina Reader | URL → Markdown | Free + rerank upsells | Strong anti-bot capabilities, direct r.jina.ai/URL usage | | Apify RAG Web Browser | Integrated Google search + crawling | Actor runtime | Built-in search, MCP-friendly | | Context.dev | Atomic endpoints for Agents | Pricing undisclosed | More granular endpoints, standalone URL discovery API |

What I find most interesting is that Context.dev separates Discover URLs into its own endpoint. Other products provide this capability to some extent, but usually bury it inside full-site crawling. Making it independent means an Agent can first perform reconnaissance before deciding whether a full crawl is worthwhile — which matters in token-cost-sensitive scenarios. You can spend a few cents to understand the structure of a site before deciding whether to spend tens of dollars crawling everything.

Its case study list already includes some recognizable names: Mintlify (documentation hosting), Chatwoot (customer support), Daily.dev (developer community), and Sunrun (solar energy company). The first three are obvious Agent/RAG users, but Sunrun’s presence is particularly interesting — traditional industries are starting to redesign their knowledge distribution methods for AI applications.

But There Are Still Risks

A reality check: APIs like this face several unavoidable structural issues.

First, the legal and ethical boundaries are tightening. Reddit suing Anthropic, The New York Times suing OpenAI, and Forbes calling out Perplexity all reflect an industry-wide decline in tolerance toward “scraping public web pages for AI training/services.” As an API provider, Context.dev effectively centralizes this risk — if a major legal case emerges one day, every SaaS in the chain could be affected.

Second, dynamic content extraction still faces quality bottlenecks. Documentation sites and blogs convert cleanly into Markdown, but scenarios involving logins, dynamic loading, and complex interactions (such as SPA tab switching) still require optimization for specific websites. Context.dev’s current API doesn’t reveal any special handling for these cases.

Third, pricing models are a difficult battle. Firecrawl has already revised its pricing strategy multiple times because the marginal cost of crawling is difficult to compress. Context.dev’s website still lacks a public pricing page, so it’s hard to judge whether it will pursue a developer-friendly route or focus on enterprise customers.

What This Means for Developers

If you’re currently building an Agent or RAG system, the recommendation is straightforward: stop writing parsers by hand. This space has already passed the cost-effectiveness tipping point for self-hosted solutions. Unless you have highly specialized crawling requirements, calling an API is the more rational choice.

For product selection:

• If you only need single-page Markdown extraction, Jina Reader’s free quota is sufficient, and you can even use direct URL concatenation;
• If you need schema-based structured extraction, Firecrawl’s Extract is more mature;
• If you want integrated “search + crawling,” Apify’s RAG Web Browser is a better fit;
• If you need fine-grained crawl scope control and standalone URL discovery capabilities, Context.dev is worth trying.

The next-level question is: who processes the Markdown after extraction? Crawling is only the first step in the RAG pipeline. After that come chunking, embeddings, retrieval, reranking, and synthesis — each step involves model choices. Developers are increasingly leaning toward a “one key for multiple model providers” approach to avoid lock-in. That’s where aggregation platforms like OpenAI Hub become valuable: use Context.dev for crawling, Voyage for embeddings, Claude or GPT for synthesis, all without maintaining three separate authentication systems.

Conclusion: The “Water Sellers” of the Agent Era

In a gold rush, the people who make money are the ones selling shovels. In the Agent wave, model companies attract most of the attention, but beneath them a stable layer of “water sellers” has already formed: vector databases, retrieval services, evaluation platforms, prompt management, and crawling APIs.

Context.dev belongs to one of the most fundamental yet indispensable layers. It’s not glamorous, but every serious Agent project needs it. YC had good reason to include it in S26 — this position will likely produce at least one independent mid-sized company within the next few years. The only question is who executes best.

Based on its current product maturity and endpoint design, Context.dev at least deserves a spot on your shortlist.

References

• Firecrawl Open Source Repository · GitHub: Currently the most active open-source web-to-Markdown crawling tool, useful as a reference point for Context.dev.
• Crawl4AI · GitHub: An open-source crawler framework designed for LLMs with a highly active community, suitable for self-hosted solutions.
• Discussion on RAG Web Crawling Solutions · Zhihu: Discussion in the Chinese community about selecting web scraping tools for RAG scenarios.
• YC S26 Batch Discussions · Reddit: Community discussions around the latest Y Combinator startup batch.
• Playwright · GitHub: If you plan to build your own scraping solution, Playwright remains the de facto standard for browser automation.