Web Crawlers — what they see and how to control them

Googlebot, GPTBot, ClaudeBot, PerplexityBot, Bingbot, and friends.

Plain HTTPS, often no JavaScript.

What this audience needs

Crawlers walk the public web following links and reading HTML. Most respect robots.txt. Many — Googlebot included — render some JavaScript, but Dash's full JS shell is fragile to index reliably. AI training crawlers (GPTBot, CCBot) usually skip JS entirely.

For this audience, 2.0 ships:

1. /robots.txt with bot-class access policies via RobotsConfig.
2. /sitemap.xml generated from dash.page_registry minus hidden pages.
3. A bot-detection middleware that intercepts every request, classifies

the User-Agent, and returns a prerendered static HTML page for crawlers — so they get the page's LLMS_DOC as visible content instead of an empty Dash shell.

1. AI-training-bot blocking via block_ai_training=True, which

returns 403 to known training UAs.

How configuration flows

from dash_improve_my_llms import RobotsConfig, mark_hidden

app._base_url = "https://myapp.com"
app._robots_config = RobotsConfig(
    block_ai_training=True,    # GPTBot, CCBot, anthropic-ai → 403
    allow_ai_search=True,      # ClaudeBot, ChatGPT-User → allowed
    allow_traditional=True,    # Googlebot, Bingbot → allowed
    crawl_delay=10,            # seconds between requests
    disallowed_paths=["/admin"],
)

mark_hidden("/admin")          # 404 for crawlers, skipped in sitemap

RobotsConfig drives both the /robots.txt body and the middleware's runtime decisions — they always agree.

Bot classes recognized

• AI Training — GPTBot, anthropic-ai, Claude-Web, CCBot,

Google-Extended, FacebookBot, Omgili, ByteSpider. Default policy: blocked.

• AI Search — ChatGPT-User, ClaudeBot, PerplexityBot, OAI-SearchBot.

Default policy: allowed.

• Traditional — Googlebot, Bingbot, DuckDuckBot, Yandex, plus

generic patterns (bot, crawler, spider). Default policy: allowed.

Verifying it works

Use curl to impersonate a crawler:

# Training bot — should return 403 with block_ai_training=True
curl -A "Mozilla/5.0 (compatible; GPTBot/1.0)" https://myapp.com/

# Search bot — should return prerendered static HTML
curl -A "Mozilla/5.0 (compatible; Googlebot/2.1)" https://myapp.com/

# Inspect the prose surface
curl https://myapp.com/llms.txt