# Web Crawlers — what they see and how to control them

> Googlebot, GPTBot, ClaudeBot, PerplexityBot, Bingbot, and friends.
> Plain HTTPS, often no JavaScript.

**Site index:** [https://llms.2plot.dev/llms.txt](https://llms.2plot.dev/llms.txt) — every page on this site, as Markdown.  
**Network index:** [https://2plot.dev/llms.txt](https://2plot.dev/llms.txt) — The 2plot network; start here to discover sibling sites.  
**Sibling sites:** 3 more in The 2plot network — listed in the site index above.  
**Sitemap:** https://llms.2plot.dev/sitemap.xml  


## What this audience needs

Crawlers walk the public web following links and reading HTML. Most
respect `robots.txt`. Many — Googlebot included — render some
JavaScript, but Dash's full JS shell is fragile to index reliably. AI
training crawlers (GPTBot, CCBot) usually skip JS entirely.

For this audience, 2.0 ships:

1. `/robots.txt` with bot-class access policies via `RobotsConfig`.
2. `/sitemap.xml` generated from `dash.page_registry` minus hidden pages.
3. A bot-detection middleware that intercepts every request, classifies
   the User-Agent, and returns a prerendered static HTML page for
   crawlers — so they get the page's `LLMS_DOC` as visible content
   instead of an empty Dash shell.
4. AI-training-bot blocking via `block_ai_training=True`, which
   returns 403 to known training UAs.

## How configuration flows

```python
from dash_improve_my_llms import RobotsConfig, mark_hidden

app._base_url = "https://myapp.com"
app._robots_config = RobotsConfig(
    block_ai_training=True,    # GPTBot, CCBot, anthropic-ai → 403
    allow_ai_search=True,      # ClaudeBot, ChatGPT-User → allowed
    allow_traditional=True,    # Googlebot, Bingbot → allowed
    crawl_delay=10,            # seconds between requests
    disallowed_paths=["/admin"],
)

mark_hidden("/admin")          # 404 for crawlers, skipped in sitemap
```

`RobotsConfig` drives both the `/robots.txt` body and the middleware's
runtime decisions — they always agree.

## Bot classes recognized

- **AI Training** — GPTBot, anthropic-ai, Claude-Web, CCBot,
  Google-Extended, FacebookBot, Omgili, ByteSpider.
  Default policy: blocked.
- **AI Search** — ChatGPT-User, ClaudeBot, PerplexityBot, OAI-SearchBot.
  Default policy: allowed.
- **Traditional** — Googlebot, Bingbot, DuckDuckBot, Yandex, plus
  generic patterns (`bot`, `crawler`, `spider`).
  Default policy: allowed.

## Verifying it works

Use `curl` to impersonate a crawler:

```bash
# Training bot — should return 403 with block_ai_training=True
curl -A "Mozilla/5.0 (compatible; GPTBot/1.0)" https://myapp.com/

# Search bot — should return prerendered static HTML
curl -A "Mozilla/5.0 (compatible; Googlebot/2.1)" https://myapp.com/

# Inspect the prose surface
curl https://myapp.com/llms.txt
```