Web tools for my agents: three I tested
Created
My agents use the web through two self-hosted services in my homelab: searxng finds pages and firecrawl reads them as Markdown. Every character a tool returns ends up in an agent’s context, so smaller and better answers matter. On 2026-10-09 I tested three newer tools against them, on my Mac, with small scripts and no agent in the loop. The question for each: would it make my agents cheaper or better at the web?
The three tools
| Tool | What it is | Job it would do |
|---|---|---|
| Charlotte (v0.8.0) | An MCP server that drives its own headless Chromium and answers with a compact page structure (landmarks, headings, controls with ids); more detail only when asked | Moving around and clicking on public pages |
| Crawl4AI (v0.9.4) | An open-source Python page reader that turns pages into Markdown, with an optional filter that prunes boilerplate | Reading, like Firecrawl |
| ddgs (v9.16.0, MIT) | A Python library, CLI, API server and MCP server that queries up to ten search engines (Bing, Brave, DuckDuckGo, Mojeek and others) and can also fetch a page as Markdown | Searching, like SearXNG; a light extractor on the side |
Reading pages (characters returned)
| Page | Charlotte on arrival | Charlotte full content | Crawl4AI raw | Crawl4AI pruned | ddgs extract | Firecrawl |
|---|---|---|---|---|---|---|
| Hacker News | 287 | 30,060 | 17,734 | 11,374 | 21,011 | 18,311 |
| My own site | 1,508 | 4,977 | 1,903 | 1,521 | 1,751 | 1,613 |
| Wikipedia (AI article) | 19,212 | 456,806 | 735,008 | 488,607 | 465,408 | 536,702 |
| A GitHub repo page | 5,278 | 65,394 | 45,044 | 26,423 | 46,316 | 36,949 |
Speed: Crawl4AI and Firecrawl were within a second of each other (0.6 to 4.4 s). ddgs extract was the fastest (0.2 to 0.7 s) because it fetches the page without a browser, so it can’t read pages that need JavaScript. Charlotte took 2 to 3.5 s to load a page, then answered follow-up questions about it almost instantly.
Searching (ddgs against SearXNG)
Five technical queries (Proxmox bind mounts, OPNsense WireGuard routing, deploying a Hugo site, a budget ECC NAS build, Claude Code hooks), ten results each:
| ddgs | SearXNG | |
|---|---|---|
| Results | 10 every time | 10 every time |
| Time | 0.8 to 3.2 s | 0.5 to 1.8 s |
| Shared with the other’s top 10 | 1 to 4 of 10 | |
| Top 3 results, by eye | more official docs, vendor forums and GitHub issues | more Reddit, YouTube and Facebook |
The low overlap is the interesting part: the two return largely different pages for the same question, so they work better as a second opinion for each other than as replacements.
What I learned
- Charlotte is cheap for getting around, not for reading. Landing on a page and finding one link costs a few hundred to a few thousand characters (all the comment links on Hacker News: 3,269). Asking it for the content costs as much as Firecrawl or more, and its own benchmark admits form filling costs about 5 times more than Playwright MCP. It also starts from a fresh browser with no logins, so it can’t stand in for a signed-in session.
- Crawl4AI is a fair match for Firecrawl, not a step up. Same speed; its pruning trims a third on busy pages but is less predictable. Its plus is size: one Python process instead of Firecrawl’s six containers.
- ddgs is the most useful of the three. No server to run, a small MCP server, and on technical questions its results leaned towards the sources I’d want an agent to read first. Like SearXNG, it works by querying public engines, which can rate-limit or change their pages; its README says it is for educational use.
- Measure before you host. All three looked like clear wins from their docs. A short round of measuring on my own pages and queries showed none would replace what I run.
Decision
Keep SearXNG for search and Firecrawl for reading; host none of the three for now. ddgs’s better results had a plain cause: my SearXNG ran on its default engines, and with two of them blocked or rate-limited it was answering from a single engine. I then turned on Bing and Mojeek: social-media results on the test queries fell from 15 of 50 to 8, and the overlap with ddgs rose. (Startpage made it worse again: it repeats the ranking of the engine I already had.) ddgs stays the zero-setup fallback if SearXNG is down. Crawl4AI is the fallback if Firecrawl becomes a burden to run; Charlotte comes back if my agents start clicking through public sites often. Detail and gotchas for Charlotte: charlotte browser mcp.