Which AI bots do leading SaaS sites block?
Data & research / AI search
Which AI bots do leading SaaS sites block?
35 of 48 leading B2B SaaS websites (73%) do not mention a single AI crawler in robots.txt. Only 7 sites (15%) fully block at least one AI bot, and just 3 block OpenAI’s GPTBot.
By Georgi Todorov · Data collected 5 October 2026
What we found
Most large SaaS companies treat AI crawlers like any other visitor. 35 of the 48 sites we checked have no rule for any of the 14 AI user agents, so those bots fall under the general rules for all crawlers. None of the 48 sites blocks all crawlers site-wide.
The 7 sites that fully block at least one AI bot are: calendly.com, canva.com, figma.com, github.com, loom.com, notion.com, servicenow.com. The bot blocked most often is CCBot, the Common Crawl crawler, at 4 sites. GPTBot is fully blocked by 3 sites and ClaudeBot by 2.
A few companies go the other way and write explicit allow rules. Zapier and Cloudflare both name several AI bots only to allow them.
How many sites name each bot
Sites out of 48 with any rule (block, partial or allow) for the bot.
Rules per bot
| Bot | Operator | Fully blocked | Partly blocked | Explicitly allowed | Not mentioned |
|---|---|---|---|---|---|
| GPTBot | OpenAI | 3 | 4 | 2 | 39 |
| OAI-SearchBot | OpenAI | 1 | 4 | 1 | 42 |
| ChatGPT-User | OpenAI | 1 | 3 | 2 | 42 |
| ClaudeBot | Anthropic | 2 | 5 | 1 | 40 |
| anthropic-ai | Anthropic | 1 | 2 | 1 | 44 |
| Claude-Web | Anthropic | 1 | 0 | 1 | 46 |
| PerplexityBot | Perplexity | 1 | 4 | 3 | 40 |
| Perplexity-User | Perplexity | 0 | 3 | 1 | 44 |
| Google-Extended | 1 | 3 | 2 | 42 | |
| CCBot | Common Crawl | 4 | 1 | 1 | 42 |
| Bytespider | ByteDance | 3 | 1 | 0 | 44 |
| Applebot-Extended | Apple | 1 | 2 | 0 | 45 |
| meta-externalagent | Meta | 2 | 3 | 0 | 43 |
| Amazonbot | Amazon | 1 | 3 | 0 | 44 |
Full dataset: 48 sites × 14 bots
Blocked = the bot has a group with “Disallow: /”. Blocked* = the same, with some paths allowed as exceptions. Partial = only specific paths are disallowed. Allowed = named with no disallow rule. A dash means the bot is not mentioned.
| Site | GPTBot | OAI-SearchBot | ChatGPT-User | ClaudeBot | anthropic-ai | Claude-Web | PerplexityBot | Perplexity-User | Google-Extended | CCBot | Bytespider | Applebot-Extended | meta-externalagent | Amazonbot |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| canva.com | Blocked* | – | – | Blocked* | Blocked* | Blocked* | Allowed | Allowed | – | Blocked* | Blocked | Blocked* | Blocked | – |
| hootsuite.com | Partial | Partial | Partial | Partial | – | – | Partial | Partial | Partial | – | – | Partial | Partial | Partial |
| squarespace.com | Partial | – | – | Partial | Partial | – | – | – | Partial | Partial | Partial | Partial | Partial | Partial |
| figma.com | Blocked | Blocked* | Blocked* | Blocked | – | – | Blocked* | – | Blocked | Blocked | Blocked | – | – | – |
| cloudflare.com | Allowed | – | Allowed | – | Allowed | Allowed | Allowed | – | Allowed | Allowed | – | – | – | – |
| klaviyo.com | Partial | Partial | Partial | Partial | – | – | Partial | Partial | – | – | – | – | Partial | – |
| github.com | Partial | Partial | – | Partial | Partial | – | Partial | – | – | – | Blocked | – | – | – |
| monday.com | – | Partial | Partial | Partial | – | – | Partial | Partial | Partial | – | – | – | – | – |
| zapier.com | Allowed | Allowed | Allowed | Allowed | – | – | Allowed | – | Allowed | – | – | – | – | – |
| loom.com | Blocked | – | – | – | – | – | – | – | – | – | – | – | – | Partial |
| notion.com | – | – | – | – | – | – | – | – | – | – | – | – | Blocked | Blocked |
| calendly.com | – | – | – | – | – | – | – | – | – | Blocked* | – | – | – | – |
| servicenow.com | – | – | – | – | – | – | – | – | – | Blocked | – | – | – | – |
| adobe.com | – | – | – | – | – | – | – | – | – | – | – | – | – | – |
| ahrefs.com | – | – | – | – | – | – | – | – | – | – | – | – | – | – |
| airtable.com | – | – | – | – | – | – | – | – | – | – | – | – | – | – |
| asana.com | – | – | – | – | – | – | – | – | – | – | – | – | – | – |
| atlassian.com | – | – | – | – | – | – | – | – | – | – | – | – | – | – |
| box.com | – | – | – | – | – | – | – | – | – | – | – | – | – | – |
| clickup.com | – | – | – | – | – | – | – | – | – | – | – | – | – | – |
| datadoghq.com | – | – | – | – | – | – | – | – | – | – | – | – | – | – |
| docusign.com | – | – | – | – | – | – | – | – | – | – | – | – | – | – |
| dropbox.com | – | – | – | – | – | – | – | – | – | – | – | – | – | – |
| freshworks.com | – | – | – | – | – | – | – | – | – | – | – | – | – | – |
| gitlab.com | – | – | – | – | – | – | – | – | – | – | – | – | – | – |
| gong.io | – | – | – | – | – | – | – | – | – | – | – | – | – | – |
| grammarly.com | – | – | – | – | – | – | – | – | – | – | – | – | – | – |
| hubspot.com | – | – | – | – | – | – | – | – | – | – | – | – | – | – |
| intercom.com | – | – | – | – | – | – | – | – | – | – | – | – | – | – |
| mailchimp.com | – | – | – | – | – | – | – | – | – | – | – | – | – | – |
| miro.com | – | – | – | – | – | – | – | – | – | – | – | – | – | – |
| okta.com | – | – | – | – | – | – | – | – | – | – | – | – | – | – |
| pipedrive.com | – | – | – | – | – | – | – | – | – | – | – | – | – | – |
| salesforce.com | – | – | – | – | – | – | – | – | – | – | – | – | – | – |
| semrush.com | – | – | – | – | – | – | – | – | – | – | – | – | – | – |
| shopify.com | – | – | – | – | – | – | – | – | – | – | – | – | – | – |
| slack.com | – | – | – | – | – | – | – | – | – | – | – | – | – | – |
| snowflake.com | – | – | – | – | – | – | – | – | – | – | – | – | – | – |
| surveymonkey.com | – | – | – | – | – | – | – | – | – | – | – | – | – | – |
| trello.com | – | – | – | – | – | – | – | – | – | – | – | – | – | – |
| twilio.com | – | – | – | – | – | – | – | – | – | – | – | – | – | – |
| typeform.com | – | – | – | – | – | – | – | – | – | – | – | – | – | – |
| webflow.com | – | – | – | – | – | – | – | – | – | – | – | – | – | – |
| wix.com | – | – | – | – | – | – | – | – | – | – | – | – | – | – |
| workday.com | – | – | – | – | – | – | – | – | – | – | – | – | – | – |
| zendesk.com | – | – | – | – | – | – | – | – | – | – | – | – | – | – |
| zoho.com | – | – | – | – | – | – | – | – | – | – | – | – | – | – |
| zoom.com | – | – | – | – | – | – | – | – | – | – | – | – | – | – |
Method
- We chose 48 widely used B2B software companies. This is our own selection, not a ranking.
- On 5 October 2026 we opened each site’s live robots.txt in a browser and parsed the raw text with a script.
- For each of 14 AI user agents we read the rules in the group that names it. Matching is case-insensitive.
- We checked the main marketing domain only, not app, docs or regional subdomains.
Limits of this data
- Robots.txt is a request, not an enforcement tool. Sites can also block bots at the firewall or CDN, which this study cannot see.
- A bot that is “not mentioned” is not necessarily welcome. It simply follows the general rules.
- Two sites we planned to include, stripe.com and moz.com, could not be opened in our setup and are left out.
- Robots.txt files change. This is a snapshot of one day.
Want to check your own file? Use the free robots.txt AI bot checker.
Use this data
Free to use with attribution and a link to this page.
Create & Grow (2026). Which AI bots do leading SaaS sites block?. https://createandgrow.com/research/ai-bots-saas-robots-txt/
About the author
Georgi Todorov is the founder of Create & Grow, a digital PR and SEO agency for B2B brands. More about Georgi