# Bot Policy Sandbox > Build a RobotsConfig with switches and watch the robots.txt it would generate, plus the per-vendor verdict table — against a throwaway config that never touches what this site serves. **Site index:** [https://llms.2plot.dev/llms.txt](https://llms.2plot.dev/llms.txt) — every page on this site, as Markdown. **Network index:** [https://2plot.dev/llms.txt](https://2plot.dev/llms.txt) — The 2plot network; start here to discover sibling sites. **Sibling sites:** 13 more in The 2plot network — listed in the site index above. **Sitemap:** https://llms.2plot.dev/sitemap.xml --- ### Showcase B — the bot-policy sandbox `robots.txt` is a file most projects write once, by hand, and never check again. This package generates it from a **vendor registry**, which means it can also show you what any configuration *would* produce before you ship it. Move the switches. The document on the right is generated by the same function that serves [`/robots.txt`](/robots.txt) on this host. ```python # Live component, rendered above on the browser lane. # Source: docs/robots_sandbox/robots_sandbox.py """Showcase B — build a RobotsConfig, see the robots.txt it would generate. THE INVARIANT, and it has a test (tests/test_showcase.py): this module never assigns `app._robots_config`. Dash callbacks are global on a shared server, so a sandbox that mutated the live config would let any visitor rewrite what every other visitor — and every crawler — is served. Every render here builds a THROWAWAY config and passes it to `generate_robots_txt(config=...)`. Exec-module rules: ids all start `rbsx-`; no import-time registry walk. """ from dash import Input, Output, callback, html import dash_mantine_components as dmc from dash_iconify import DashIconify ID = "rbsx" SITEMAP = "https://llms.2plot.dev/sitemap.xml" BASE_URL = "https://llms.2plot.dev" _POLICY_COLOR = {"allow": "teal", "block": "red", "meter": "yellow"} def _vendor_rows(): """(key, display, operator, class) for every vendor in the registry.""" from dash_improve_my_llms.vendors import VENDORS return [(v.key, v.display, v.operator, v.cls) for v in VENDORS] component = html.Div( [ dmc.Alert( "This sandbox builds a throwaway config on every render. It never " "touches what this site actually serves — open /robots.txt in " "another tab and it will not change.", title="Nothing here is live", color="blue", variant="light", icon=DashIconify(icon="tabler:shield-check"), ), dmc.Space(h="md"), dmc.Grid( [ dmc.GridCol( dmc.Stack( [ dmc.Text("Coarse flags", fw=600, size="sm"), dmc.Switch(id=f"{ID}-training", checked=True, label="block_ai_training"), dmc.Switch(id=f"{ID}-search", checked=True, label="allow_ai_search"), dmc.Switch(id=f"{ID}-traditional", checked=True, label="allow_traditional"), dmc.Switch(id=f"{ID}-docs", checked=False, label="block_ai_training_docs"), dmc.NumberInput(id=f"{ID}-delay", label="crawl_delay", value=10, min=0, max=120, w=140), dmc.Space(h="xs"), dmc.Text("Per-vendor override (W2)", fw=600, size="sm"), dmc.Select( id=f"{ID}-vendor", label="Vendor", data=[{"value": key, "label": f"{display} ({cls})"} for key, display, _op, cls in _vendor_rows()], searchable=True, clearable=True, placeholder="none", ), dmc.SegmentedControl( id=f"{ID}-action", data=[{"value": "allow", "label": "allow"}, {"value": "meter", "label": "meter"}, {"value": "block", "label": "block"}], value="block", fullWidth=True, ), ], gap="sm", ), span={"base": 12, "md": 4}, ), dmc.GridCol( dmc.Tabs( [ dmc.TabsList([ dmc.TabsTab("Generated robots.txt", value="robots"), dmc.TabsTab("Per-vendor verdicts", value="verdicts"), ]), dmc.TabsPanel(html.Div(id=f"{ID}-robots"), value="robots", pt="sm"), dmc.TabsPanel(html.Div(id=f"{ID}-verdicts"), value="verdicts", pt="sm"), ], value="robots", ), span={"base": 12, "md": 8}, ), ], gutter="lg", ), ] ) @callback( Output(f"{ID}-robots", "children"), Output(f"{ID}-verdicts", "children"), Input(f"{ID}-training", "checked"), Input(f"{ID}-search", "checked"), Input(f"{ID}-traditional", "checked"), Input(f"{ID}-docs", "checked"), Input(f"{ID}-delay", "value"), Input(f"{ID}-vendor", "value"), Input(f"{ID}-action", "value"), ) def _render(training, search, traditional, docs, delay, vendor, action): from dash_improve_my_llms import RobotsConfig from dash_improve_my_llms.robots_generator import generate_robots_txt # THROWAWAY. Never assigned to the app — see this module's docstring. from dash_improve_my_llms.vendors import effective_policies config = RobotsConfig( block_ai_training=bool(training), allow_ai_search=bool(search), allow_traditional=bool(traditional), block_ai_training_docs=bool(docs), crawl_delay=int(delay) if delay else None, vendor_policy={vendor: action} if vendor else None, ) robots = generate_robots_txt(sitemap_url=SITEMAP, base_url=BASE_URL, config=config) policies = effective_policies(config) robots_panel = dmc.Paper( dmc.Code(robots, block=True, style={"maxHeight": "560px", "overflow": "auto", "display": "block"}), withBorder=True, radius="md", p="sm", ) verdicts = dmc.Table( [ dmc.TableThead(dmc.TableTr([ dmc.TableTh("Vendor"), dmc.TableTh("Operator"), dmc.TableTh("Class"), dmc.TableTh("Effective policy"), ])), dmc.TableTbody([ dmc.TableTr([ dmc.TableTd(display), dmc.TableTd(dmc.Text(operator, size="xs", c="dimmed")), dmc.TableTd(dmc.Text(cls, size="xs")), dmc.TableTd(dmc.Badge(policies.get(key, "?"), color=_POLICY_COLOR.get(policies.get(key), "gray"), variant="light")), ]) for key, display, operator, cls in _vendor_rows() ]), ], striped=True, highlightOnHover=True, withTableBorder=True, ) return robots_panel, verdicts ``` ### One invariant, and it has a test The sandbox builds a **throwaway** `RobotsConfig` on every render and passes it as `generate_robots_txt(config=...)`. It never assigns `app._robots_config`. That is not fastidiousness. Dash callbacks are **global on a shared server**: a sandbox that mutated the live config would let any visitor rewrite the policy every other visitor — and every crawler — is served. `tests/ test_showcase.py` asserts the module contains no such assignment, because this is exactly the kind of thing a later refactor makes "simpler". ### What the switches do | Knob | Effect | |---|---| | `block_ai_training` | the training bucket (GPTBot, ClaudeBot, CCBot, …). With it `False` the bucket is **not emitted at all**, which silently *allows* training — not "balanced" | | `allow_ai_search` | Claude-User, ChatGPT-User, PerplexityBot, OAI-SearchBot — the fetchers that send readers | | `allow_traditional` | Googlebot, Bingbot, Slurp, DuckDuckBot. `False` emits real `Disallow` groups | | `block_ai_training_docs` | closes the documentation surfaces to blocked training crawlers. **Off by default** — the corpus exists to get the package used | | `crawl_delay` | rendered inside the `User-agent: *` group, where RFC 9309's strict parsers expect it | | per-vendor override | 2.7.0's `vendor_policy`: one vendor, `allow` / `block` / `meter` | ### `meter`, and why it renders as Allow `meter` means *"fetchable under the rate contract"*. A `Disallow` would kill the funnel the meter exists for, so robots.txt renders it as `Allow` and the middleware treats it as allow until the limiter consumes it. ### One fold, two consumers The verdict table and the generated document are both rendered from `vendors.effective_policies(config)` — one function. That is the design that makes "what the site says" and "what the site does" the same statement rather than two things to keep in sync. ```python from dash_improve_my_llms.vendors import effective_policies effective_policies(app._robots_config) # {'gptbot': 'block', 'googlebot': 'allow', ...} ``` The package's read-only operator panel (`/llms-policy`) renders its vendor table from this same call, which is why it cannot drift from `robots.txt`. --- *Source: /showcase/robots-sandbox*