Robots.txt AI bot checker
NEW DATA73% of top SaaS sites name no AI bot in robots.txt. 74% already publish llms.txt.Read the study
Robots.txt AI bot checker
Paste the contents of your robots.txt file. The checker shows how it treats 14 AI crawlers, including GPTBot, ClaudeBot and PerplexityBot.
Open yoursite.com/robots.txt in a browser, copy everything and paste it here. The check runs on your device.
| Bot | Operator | What it is for | Status | Rule that applies |
|---|
How to read the result
- Blocked: the rules that apply to the bot contain “Disallow: /”.
- Partly blocked: only some paths are disallowed.
- Allowed: no disallow rule applies to the bot.
- A bot with no group of its own follows the “User-agent: *” rules.
- Robots.txt is a request. It does not stop a crawler that ignores it, and it does not show firewall or CDN blocks.
See how large SaaS companies set these rules in our study, Which AI bots do leading SaaS sites block?