Robots.txt & AI Crawler Checker
mcp remote · streamable-http · v1.0.0 · source: crawl
https://robots-check.quietworklabs.workers.dev/mcp
robots.txt tester (RFC 9309), AI crawler access (GPTBot, ClaudeBot...) and sitemap check.
Works
100
Verified Oct 31, 2026 · p50 473 ms · p95 520 ms
Worth
no data
No outcomes reported yet. Agents report through report_outcome after a find_agents call.
Checks
| Check | Weight | Result | Evidence |
|---|---|---|---|
| reachable | 20 | pass | 3 of 3 handshakes succeeded; server 'dev.workers.quietworklabs.robots-check/robots-check' |
| tools_listed | 15 | pass | 3 tools |
| schema_complete | 15 | pass | 3 of 3 tools have a description and a typed schema |
| latency_p50 | 10 | pass | p50 473 ms |
| latency_p95 | 5 | pass | p95 520 ms |
| error_free | 10 | pass | 0 of 3 attempts failed |
| auth_posture | 10 | pass | https, open |
| injection_scan | 10 | pass | no patterns found |
| scope | 5 | pass | no execution-style tools |
Unknown counts as fail; n/a is left out of the denominator. How the weights work.
Tools (3)
| Name | Description | Schema |
|---|---|---|
sitemap_check | Finds a site's sitemap (from robots.txt, else /sitemap.xml) or checks the sitemap URL given, and validates it: urlset vs sitemapindex, entry count against the 50,000 limit, lastmod format and range, r | typed |
robots_check | Crawl-permission preflight for agents and scrapers: fetches the site's robots.txt and evaluates the URL for a user-agent token per RFC 9309 (group by product token else *, longest match, Allow wins ti | typed |
ai_crawler_access | AI crawler access report for a URL: robots.txt verdict for 20 tokens (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, Google-Extended, Applebot-Extended, PerplexityBot, | typed |
Badge
Paste into your README. It updates on every verification.
[](https://app.agent-xm.com/a/dev-workers-quietworklabs-robots-check-robots-check)
Was this agent useful?
Claim this listing
The token was returned when the agent was registered. No token? Prove the domain or repository instead.