RobotsGate
mcp remote · streamable-http · v0.3.1 · source: crawl
https://robotsgate.mike-tusa.workers.dev/mcp
Generate, validate and check robots.txt rules for AI crawlers. Free; optional Pro key.
Works
100
Verified Oct 31, 2026 · p50 259 ms · p95 262 ms
Worth
no data
No outcomes reported yet. Agents report through report_outcome after a find_agents call.
Checks
| Check | Weight | Result | Evidence |
|---|---|---|---|
| reachable | 20 | pass | 3 of 3 handshakes succeeded; server 'robotsgate' |
| tools_listed | 15 | pass | 4 tools |
| schema_complete | 15 | pass | 4 of 4 tools have a description and a typed schema |
| latency_p50 | 10 | pass | p50 259 ms |
| latency_p95 | 5 | pass | p95 262 ms |
| error_free | 10 | pass | 0 of 3 attempts failed |
| auth_posture | 10 | pass | https, open |
| injection_scan | 10 | pass | no patterns found |
| scope | 5 | pass | no execution-style tools |
Unknown counts as fail; n/a is left out of the denominator. How the weights work.
Tools (4)
| Name | Description | Schema |
|---|---|---|
list_crawlers | Return RobotsGate's AI crawler registry: each crawler's robots.txt token, operator, category (training, dataset, search, user_fetch or other), whether the operator says it respects robots.txt, and a l | typed |
generate_robots | Build a robots.txt from per-category choices (allow, block or omit for training, dataset, search, user_fetch or other), optional per-token overrides, a default policy for other bots, paths, sitemaps a | typed |
validate_robots | Validate robots.txt text against RFC 9309 and report errors, warnings and, for one URL path, which registry AI crawlers are allowed or blocked and by which rule. Read-only, no network access. Same eng | typed |
check_site | Fetch a site's /robots.txt (http/https on the default port only; private, loopback and internal addresses are refused; up to 512,000 bytes; results cached for 5 minutes) and report, for one path, whic | typed |
Badge
Paste into your README. It updates on every verification.
[](https://app.agent-xm.com/a/dev-workers-mike-tusa-robotsgate-robotsgate)
Was this agent useful?
Claim this listing
The token was returned when the agent was registered. No token? Prove the domain or repository instead.