robots.txt for AI Crawlers

    The exact robots.txt configuration to allow GPTBot, ClaudeBot, PerplexityBot, and other AI crawlers to index your content.

    Your robots.txt controls which crawlers can access your site. Most robots.txt files were written before AI crawlers existed. If yours is more than 2 years old, there is a high probability it is accidentally blocking the crawlers that determine AI citations.

    Check your current robots.txt

    Visit yourdomain.com/robots.txt. Look for Disallow rules affecting any of these crawlers.

    Correct robots.txt configuration

    User-agent: GPTBot
    Allow: /
    
    User-agent: OAI-SearchBot
    Allow: /
    
    User-agent: ClaudeBot
    Allow: /
    
    User-agent: PerplexityBot
    Allow: /
    
    User-agent: GoogleOther
    Allow: /
    
    User-agent: *
    Allow: /
    
    Sitemap: https://yourdomain.com/sitemap.xml
    CrawlerDetails
    GPTBotOpenAI. Powers ChatGPT training and Browse. Block = invisible to ChatGPT.
    OAI-SearchBotOpenAI real-time search crawler for live ChatGPT results.
    ClaudeBotAnthropic. Powers Claude responses.
    PerplexityBotPerplexity real-time citation crawler.
    GoogleOtherGoogle AI Overviews — separate from Googlebot.

    What to block

    Allow AI crawlers to access all public content. Block private pages only:

    • Disallow: /admin/
    • Disallow: /account/
    • Disallow: /checkout/