Testing AI bot access...
Test Results
| Bot | Owner | robots.txt | Indexing Directives | HTTP Status | Redirect | Content | Content Length | Verdict |
|---|
Understanding the Results
🤖 Bot
The AI bot that was tested (e.g., GPTBot, ClaudeBot, PerplexityBot)
🏢 Owner
The company that owns the bot (e.g., OpenAI, Anthropic, Google)
📋 robots.txt
Whether the bot is allowed to crawl your site according to robots.txt rules
- ✅ Allowed: Bot can crawl your site
- ❌ Disallowed: Bot is blocked by robots.txt
- 📄 Empty robots.txt: File exists but is empty (allows all bots)
- ⚠️ robots.txt missing: No robots.txt file found
🔍 Indexing Directives
Whether the bot is allowed to index your site based on HTML meta tags and headers
- ✅ Indexable: No blocking directives found
- 🚫 Noindex: Page has noindex directive (prevents indexing)
- 🔗 Nofollow: Page has nofollow directive (prevents following links)
- ❓ Error: Could not check directives
🌐 HTTP Status
The actual HTTP response when the bot tries to access your site
- 200 OK: Site is accessible
- 403/404/500: Site blocks or has errors
- Error: Connection failed entirely
🔄 Redirect
Whether the bot gets redirected to a different URL
- 🟢 No Redirect: Bot reaches the intended page
- 🔀 Redirected: Bot is sent to a different URL
- ❌ Failed: Request failed before redirect
📄 Content
What content the bot actually receives
- 📄 Content Present: Bot gets meaningful content (>200 chars)
- 🕳 Blank or Blocked: Empty page or anti-bot message
- 🚫 Status XXX: HTTP error page
- ❌ Failed: Request failed completely
📏 Content Length
Number of characters in the visible text content the bot receives
- 0-200: Likely blocked or empty
- 200+: Has substantial content
⚖️ Verdict
Final assessment of whether the bot can access meaningful content
- ✅ Visible: Bot can access substantial content
- ❌ Blocked: Bot is blocked or gets no meaningful content