Back to Dashboard
Bot Configuration

Robots.txt AI Crawler Auditor

Ensure search bots can index your content while protecting your intellectual property from LLM crawlers. Paste your robots.txt or select a template to check which agents can access your pages.

Robots.txt Editor

Simulating crawling target directory: /

Crawler Audit Report(Simulated path: /)

Googlebot(googlebot)

Google's primary search crawler. Must be allowed for organic index ranking.

INDEXING ALLOWEDRule: Default (Implicit allow)
Bingbot(bingbot)

Microsoft Bing's search indexer. Essential for standard web search visibility.

INDEXING ALLOWEDRule: Default (Implicit allow)
GPTBot(gptbot)

OpenAI's web scraper. Gathers data to train GPT-4o, GPT-5, and ChatGPT models.

IP PROTECTED (Safe)Rule: Disallow: /
ClaudeBot(claudebot)

Anthropic's crawler. Scrapes content to train Claude conversational AI models.

IP PROTECTED (Safe)Rule: Disallow: /
Google-Extended(google-extended)

Google's AI training opt-out crawler. Disallowing blocks Gemini training ingestion.

IP PROTECTED (Safe)Rule: Disallow: /
PerplexityBot(perplexitybot)

Perplexity AI's real-time crawler. Must be allowed to get cited in search results.

SEARCH CITATION ALLOWEDRule: Allow: /
ChatGPT-User(chatgpt-user)

Triggered when ChatGPT users ask to browse live sites. Recommended to allow.

SEARCH CITATION ALLOWEDRule: Allow: /
SEO Recommendation: Make sure standard web crawlers (Googlebot, Bingbot) are allowed on your main product directories to retain Google search traffic. If you publish proprietary insights, keep GPTBot and ClaudeBot blocked, but allow PerplexityBot and ChatGPT-User to drive conversion traffic from AI recommendations.
Edge Bot Firewall & Crawlers Optimization

Dynamic AI Scraper Shielding & Crawler Optimization

Protect your bandwidth, server resources, and intellectual property. We deploy cloud-edge bot firewalls (Cloudflare/Vercel Edge) that block predatory AI scrapers dynamically while optimizing search citations.

Request Edge Firewall Audit