# Dash Improve My LLMs — the AI and crawler surface for Dash apps — full corpus ## Access policy - Terms: these documents are free to fetch. A free account unlocks any gated document. - Identity: agents may present a key by appending `?key=` to any document URL. Get one: https://2plot.ai - Rate: prefer ONE `/llms-full.txt` fetch over N per-page fetches. On 429, honour `Retry-After` and back off exponentially. - Coordination: start at https://2plot.dev/llms.txt — one index enumerates every site; do not rediscover the network by crawling it. - Crawler policy (mirrors /robots.txt): allowed: GPTBot, ClaudeBot, CCBot, Google-Extended, FacebookBot, Omgili, ByteSpider, Amazonbot, Applebot-Extended, meta-externalagent, AI2Bot, Diffbot, Timpibot, ImagesiftBot, ChatGPT-User, Claude-User, Claude-SearchBot, PerplexityBot, OAI-SearchBot, Perplexity-User, Googlebot, Bingbot, Slurp, DuckDuckBot, GoogleOther, Google-InspectionTool, Storebot-Google, AdsBot-Google. - Accounting: every document read is logged with the requesting vendor (verified against published IP ranges where the operator publishes them). See https://2plot.dev/llms.txt > dash-improve-my-llms — the crawler, agent and SEO companion every Dash app mounts in one line: /llms.txt, /robots.txt, /sitemap.xml, bot detection, static-HTML prerender, an MCP bridge, per-vendor policy, the country guardrail and the operator policy panel. Flask, FastAPI and Quart. By Pip Install Python. Generated from 12 pages. The page index lives at https://llms.2plot.dev/llms.txt and a compact briefing at https://llms.2plot.dev/llms-small.txt; each page also serves its own document at `/llms.txt`. --- # Dash Improve My LLMs — the AI and crawler surface for Dash apps > **`dash-improve-my-llms` — the crawler, agent and SEO companion every Dash app mounts in one line.** By [Pip Install Python](https://2plot.dev). A Dash app is a JavaScript shell. A crawler that fetches it gets an empty `
`; an agent that reads it learns nothing; a search engine indexes a loading spinner. This package closes that gap without asking you to change how you write Dash. --- ## What it does One call — `add_llms_routes(app)` — mounts the whole machine-facing surface: | Surface | Who reads it | |---|---| | `/llms.txt`, `/llms-small.txt`, `/llms-full.txt` | agents pasted a URL, and the crawlers that follow it | | `//llms.txt` | one page's prose, content-negotiated: Markdown for agents, rendered for browsers | | `/robots.txt` | every crawler, generated from one vendor registry | | `/sitemap.xml` | search engines, with priority inferred from your page tree | | a static-HTML prerender | crawlers that do not run JavaScript — the actual fix for the empty `
` | | an MCP resource per page | Claude, ChatGPT and any other MCP client, natively | Installed here: **2.10.0** — this site reads the number from the package that is actually serving it, never from prose. --- ## The three audiences This site is organised the way the package is: by who is asking. - **[MCP clients](/audiences/mcp-clients)** — an assistant that speaks MCP mounts your docs as a resource and reads them natively. No copying, no scraping, no context window spent on HTML. - **[Web crawlers](/audiences/web-crawlers)** — Googlebot, ClaudeBot, GPTBot, PerplexityBot. Each gets a document rendered for it, under a policy you declare once and serve everywhere. - **[Paste-to-chat](/audiences/llm-context)** — a person pastes your URL into a chat window. The assistant fetches prose, not a bundle. --- ## Control, not just discovery Discovery is the floor. Since 2.7 the package is also the layer that decides **who gets served at all**: - **Per-vendor policy** — allow, block or meter each crawler by name. One fold drives `robots.txt` and the middleware, so what you publish and what you enforce cannot drift apart. - **The country guardrail** — `configure_geo(deny_countries=[...])` answers 451 on *every* surface for a listed country: pages, assets, the corpus, `robots.txt`, even the favicon. Compliance, uniformly applied. - **The rate contract** — a stated ceiling on the corpus routes, enforced, failing open on any limiter error. - **The operator panel** — a read-only, token-gated page showing the live effective policy of every surface, including which header resolved this request's country. Each of those takes a **callable** as well as a static value, and the callable is read per request. That is the seam this site's control board writes through: flip a country on the map, and the next request from it gets 451 — in every worker, with no restart and no redeploy. --- ## Install ```bash pip install dash-improve-my-llms[flask] # or [fastapi], [quart], [all] ``` ```python import dash from dash_improve_my_llms import add_llms_routes, LLMSConfig app = dash.Dash(__name__, use_pages=True) add_llms_routes(app, LLMSConfig(warn_missing_llms_doc=True)) ``` That is the whole integration. Every page that defines a module-level `LLMS_DOC` string is served verbatim at its own `llms.txt`; every page without one is still indexed, still prerendered, still in the sitemap. ### Writing a page's prose ```python # pages/pricing.py LLMS_DOC = """ # Pricing Three tiers. The free tier has no time limit. """ ``` No layout walking, no extraction heuristics, no second copy of your docs to keep in sync. --- ## Backends Dash 4.1 made the server pluggable, and the package follows it: the same surface under **Flask**, **FastAPI** and **Quart**, auto-detected. Nothing in the list above is backend-specific, and the package's suite runs against all three. --- ## Resources - **PyPI**: [dash-improve-my-llms](https://pypi.org/project/dash-improve-my-llms/) - **GitHub**: [pip-install-python/dash-improve-my-llms](https://github.com/pip-install-python/dash-improve-my-llms) - **The network**: [2plot.dev](https://2plot.dev) indexes every component site that runs this package — including this one. - **The live directory**: every peer site, and this host's own policy, on the [policy panel](/showcase/policy-panel) ### Community - **GitHub**: [@pip-install-python](https://github.com/pip-install-python) ![GitHub](https://img.shields.io/github/followers/pip-install-python?style=social) - **YouTube**: [2plot.ai](https://www.youtube.com/@2plotai?sub_confirmation=1) — build-alongs and component walkthroughs --- ## License MIT License — see [LICENSE](https://github.com/pip-install-python/dash-improve-my-llms/blob/main/LICENSE) for details. --- # Paste-to-Chat > What happens when a person pastes this site's URL into ChatGPT or Claude — and how to make sure the assistant fetches your prose instead of a JavaScript bundle. --- ### The audience Not a crawler. Not an MCP client. **A person**, mid-conversation, who pastes your documentation URL into a chat window and expects the assistant to read it. This is the most common way an AI system encounters a package's docs, and the least designed-for. The assistant sends a fetcher — `ChatGPT-User`, `Claude-User`, `Perplexity-User` — which is a *user-triggered* fetch, not a crawl. It usually gets one request, it does not run JavaScript, and whatever comes back is what your package's reputation is built from in that conversation. For a stock Dash app, what comes back is an empty `
`. ### The fix, in the order it applies 1. **The prerender.** The initial HTML carries the page's real prose and real links, so a single non-JS fetch is already useful. 2. **`//llms.txt`.** The same content as clean Markdown, at a URL a person can paste deliberately. 3. **The tiered corpus.** `/llms-full.txt` for "read all of it", one fetch. 4. **Vendor policy.** User-triggered fetchers are `search`-class and **allowed** by default, even on a site that blocks AI training. Blocking them would mean nobody could ask an assistant about your package. ### Try it Copy any of these into a chat window: ```text https://llms.2plot.dev/llms.txt https://llms.2plot.dev/audiences/mcp-clients/llms.txt https://llms.2plot.dev/llms-full.txt ``` Every documentation page on this site also has a **copy button** — the `.. llms_copy::` directive at the top of each page — which puts that page's Markdown on the clipboard directly. Sometimes the fastest path is not a URL at all. ### Why the `?key=` suffix exists A page can be gated. When it is, the interactive site asks for a sign-in — but an assistant cannot sign in, and the person who *can* is right there in the chat. `/api/agent-key` turns a browser session into a portable `?key=` that a copied `llms.txt` URL carries with it, so the handoff from person to agent does not lose the person's entitlements. Until Clerk and the hub are configured the route answers `204` and the whole mechanism is inert — public docs need none of it. ### What an assistant should do, and what this site asks for Since 2.7.0 the corpus documents carry an **"Access policy"** section stating the terms in the document body, where an agent reading the corpus will actually see them — the identity convention, the coordination point, and the rate rule: > Prefer **one** `/llms-full.txt` fetch over N per-page fetches. That is enforced, not merely requested: over the ceiling the origin answers `429` with `Retry-After`. Humans are never rate-limited, and neither are `/robots.txt` and `/sitemap.xml` — a crawler that cannot read the rules is a crawler with no rules. ### Checking your own site ```bash # what an assistant's fetcher gets curl -s -A 'ChatGPT-User/1.0' https://your-site.example/ | grep -c '
' # 0 means your prose is not in the initial HTML ``` --- *Source: /audiences/llm-context* --- # MCP Clients > How an MCP-speaking assistant mounts this site's documentation as a resource and reads it natively — no scraping, no copying, no context window spent on HTML. --- ### The audience An **MCP client** is an assistant that speaks the Model Context Protocol — Claude Desktop, an IDE extension, an agent framework. It does not browse. It asks a server for *resources* and reads what comes back. A Dash app has nothing to offer such a client by default. The HTML is a loading shell, and the content only exists after React has run. So the client either gets nothing, or burns a large fraction of its context window on markup that carries none of your meaning. `dash-improve-my-llms` closes that in one line. On Dash 4.3+, every page that registers prose becomes an MCP resource automatically: ```python add_llms_routes(app, LLMSConfig(register_mcp_resources=True)) # the default ``` ### What a client sees Pick a page and read exactly what this site hands an MCP client for it. The list is this app's own live page registry — not a fixture. ```python # Live component, rendered above on the browser lane. # Source: docs/mcp_clients/mcp_registry.py """Showcase exec module: what an MCP client actually mounts from this site. Two exec-module rules apply here and are load-bearing enough to restate: 1. **Globally-unique id prefix.** Every id in this module starts `mcpx-`. Dash ids share one namespace across ~45 exec modules on this site; a collision does not error, it silently wires the wrong callback. 2. **No import-time registry walk.** `dash.page_registry` is INCOMPLETE while `pages/markdown.py` is still globbing — this module is imported from inside that loop. So `component` below is a placeholder and the table is populated by a callback, which runs long after registration finishes. """ from dash import Input, Output, callback, dcc, html, no_update import dash_mantine_components as dmc from dash_iconify import DashIconify ID = "mcpx" component = html.Div( [ dmc.Group( [ dmc.Select( id=f"{ID}-page", label="Page", description="Every page this site registers is an MCP resource.", placeholder="Loading the registry…", searchable=True, w=340, ), dmc.Switch( id=f"{ID}-raw", label="Show the raw resource body", checked=False, mt=28, ), ], gap="lg", align="flex-start", ), dmc.Space(h="md"), dmc.Paper( dmc.Stack( [ dmc.Group( [ DashIconify(icon="tabler:plug-connected", width=18), dmc.Text("Resource URI", size="sm", fw=600), ], gap="xs", ), dmc.Code(id=f"{ID}-uri", block=True), dmc.Text(id=f"{ID}-hint", size="xs", c="dimmed"), ], gap="xs", ), withBorder=True, radius="md", p="md", ), dmc.Space(h="md"), html.Div(id=f"{ID}-body"), dcc.Interval(id=f"{ID}-boot", interval=200, max_intervals=1), ] ) @callback( Output(f"{ID}-page", "data"), Output(f"{ID}-page", "value"), Input(f"{ID}-boot", "n_intervals"), ) def _populate(_tick): """Read the registry HERE, not at import time. A one-shot Interval rather than a layout-time walk: by the time this fires, every page in docs/ has registered and the list is complete. """ import dash options = sorted( ( {"value": entry["path"], "label": entry.get("name") or entry["path"]} for entry in dash.page_registry.values() if not entry["path"].startswith("/admin/") ), key=lambda option: option["label"], ) if not options: return no_update, no_update return options, options[0]["value"] @callback( Output(f"{ID}-uri", "children"), Output(f"{ID}-hint", "children"), Output(f"{ID}-body", "children"), Input(f"{ID}-page", "value"), Input(f"{ID}-raw", "checked"), ) def _describe(path, raw): if not path: return "—", "Pick a page.", None from lib.constants import BASE_URL doc = _llms_doc_for(path) suffix = "llms.txt" if path == "/" else f"{path.strip('/')}/llms.txt" uri = f"{BASE_URL}/{suffix}" hint = ( "An MCP client mounts this as a resource and reads it directly. " "A browser opening the same URL gets it rendered instead — the route " "content-negotiates on Accept, and sends Vary: Accept so a CDN cannot " "hand cached HTML to the next agent." ) if not doc: body = dmc.Alert( "This page registers no LLMS_DOC, so the resource carries its " "generated summary rather than hand-written prose.", color="yellow", variant="light", icon=DashIconify(icon="tabler:info-circle"), ) elif raw: body = dmc.Paper( dmc.Code(doc[:4000] + ("\n…" if len(doc) > 4000 else ""), block=True), withBorder=True, radius="md", p="sm", ) else: body = dmc.Paper(dcc.Markdown(doc[:4000]), withBorder=True, radius="md", p="md") return uri, hint, body def _llms_doc_for(path): """The prose dash-improve-my-llms would serve for `path`. Read through the package's own registry so this demo cannot drift from what the route actually returns. """ try: import dash_improve_my_llms as dimll store = getattr(getattr(dimll, "_state", None), "page_metadata", None) or {} return (store.get(path) or {}).get("llms_doc") or "" except Exception: # Private state, read defensively: if the package moves it, this demo # degrades to "no prose registered" rather than breaking the page. return "" ``` ### The three ways to give a page prose **1. A module-level `LLMS_DOC` string.** The package picks it up automatically. No layout walking, no extraction heuristics. ```python # pages/pricing.py LLMS_DOC = """ # Pricing Three tiers. The free tier has no time limit. """ ``` **2. Markdown-driven pages.** This site's docs are markdown files; the loader expands the directives and registers the result, so `//llms.txt` serves the *expanded* prose rather than the source. **3. `register_page_metadata(...)`** for anything that is not a page — the home page, a pseudo-path, a document assembled at boot. ### Content negotiation, and why `Vary` matters `//llms.txt` serves the same URL two ways: | Client | Sends | Gets | |---|---|---| | agent / MCP client | no `Accept: text/html` | the Markdown, byte for byte | | browser | `Accept: text/html,…` | the rendered viewer | `?raw=1` and `?format=html` force either side. Both variants send **`Vary: Accept`**, which is what stops a CDN handing an agent the HTML it cached for the last human. ### The tiered corpus One page at a time is often not what an agent wants. Three documents cover the other shapes: | Document | For | |---|---| | `/llms.txt` | the index — every page, plus the cross-host network directory | | `/llms-small.txt` | a compact briefing, when the full corpus is too much | | `/llms-full.txt` | everything, in one fetch | **Prefer one `/llms-full.txt` fetch over N per-page fetches.** That is not politeness — since 2.7.0 it is the published rate contract, and the origin answers `429` with a `Retry-After` when an agent ignores it. ### Verifying it works ```bash # what an agent gets curl -s https://llms.2plot.dev/audiences/mcp-clients/llms.txt | head # what a browser gets from the same URL curl -s -H 'Accept: text/html' https://llms.2plot.dev/audiences/mcp-clients/llms.txt | head ``` If the first command returns HTML, the negotiation is broken — that is the one check worth putting in your deploy pipeline. --- *Source: /audiences/mcp-clients* --- # Web Crawlers > What Googlebot, ClaudeBot, GPTBot and PerplexityBot actually receive from this site — the policy verdict, the crawler document, and the headers, run in-process against the real page registry. --- ### The problem, stated once A Dash app ships an empty `
`. Everything a reader sees is assembled by JavaScript afterwards. Google runs JavaScript — eventually, on a second pass, budget permitting. **Most crawlers do not run it at all**, and neither do the AI fetchers that decide whether your package gets cited. So the document a crawler stores for your page is a loading spinner, and nothing anywhere reports this. The page looks perfect to you. ### Showcase A — what the crawler sees Pick a page and a User-Agent. The verdict, the document and the headers below are produced by running the package's **own** pure handlers in this process against this site's real page registry — not by a mock, and not by a description of what they would do. ```python # Live component, rendered above on the browser lane. # Source: docs/crawler_view/crawler_view.py """Showcase A — what a crawler actually receives from this site. Runs the package's own PURE handlers in-process for a chosen page and User- Agent, and shows the three things that are normally invisible: the policy verdict, the crawler document's markup, and the headers that came with it — next to what a browser gets for the same URL. Exec-module rules (both tested elsewhere in this repo): 1. **Globally-unique id prefix** — everything here starts `crwv-`. Ids share one namespace across every exec module on the site. 2. **No import-time registry walk** — this module is imported from inside `pages/markdown.py`'s glob loop, so `dash.page_registry` is incomplete. `component` is a placeholder; the page list arrives by callback. """ import html as _html from dash import Input, Output, State, callback, dcc, html, no_update import dash_mantine_components as dmc from dash_iconify import DashIconify ID = "crwv" BROWSER_UA = ( "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 " "(KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36" ) _VERDICT_COLOR = {"allow": "teal", "block": "red", "meter": "yellow"} _CLASS_COLOR = {"training": "red", "search": "teal", "traditional": "blue"} def _ua_options(): """Real UA tokens grouped by class, straight from the vendor registry. Runs at LAYOUT time, so it depends on `vendors` existing — which the >=2.7.0 floor in requirements.txt guarantees. It carried a pre-2.7.0 fallback while that floor was still 2.6.1; the floor moved on 2026-08-23 and the fallback went with it. """ from dash_improve_my_llms.vendors import VENDORS groups = {"training": [], "search": [], "traditional": []} for vendor in VENDORS: # Not every vendor publishes a robots token: `anthropic-legacy` is a # UA-only identity (anthropic-ai, claude-web) with an empty # robots_tokens tuple, so fall back to what the middleware matches on. token = (vendor.robots_tokens or vendor.ua_tokens or (vendor.key,))[0] groups.setdefault(vendor.cls, []).append( {"value": f"{token}/1.0", "label": f"{vendor.display} — {vendor.operator}"} ) data = [{"group": "A human's browser", "items": [{"value": BROWSER_UA, "label": "Chrome 120 (a person)"}]}] for cls, label in (("training", "AI training crawlers"), ("search", "AI search & citation"), ("traditional", "Traditional search engines")): if groups.get(cls): data.append({"group": label, "items": sorted(groups[cls], key=lambda o: o["label"])}) return data component = html.Div( [ dmc.Group( [ dmc.Select( id=f"{ID}-page", label="Page", placeholder="Loading the registry…", searchable=True, w=320, ), dmc.Select( id=f"{ID}-ua", label="User-Agent", data=_ua_options(), value=BROWSER_UA, searchable=True, w=380, ), ], gap="lg", align="flex-start", ), dmc.Space(h="md"), html.Div(id=f"{ID}-verdict"), dmc.Space(h="md"), dmc.Tabs( [ dmc.TabsList( [ dmc.TabsTab("Rendered", value="rendered", leftSection=DashIconify(icon="tabler:eye", width=15)), dmc.TabsTab("View source", value="source", leftSection=DashIconify(icon="tabler:code", width=15)), dmc.TabsTab("Headers", value="headers", leftSection=DashIconify(icon="tabler:list-details", width=15)), ] ), dmc.TabsPanel(html.Div(id=f"{ID}-rendered"), value="rendered", pt="md"), dmc.TabsPanel(html.Div(id=f"{ID}-source"), value="source", pt="md"), dmc.TabsPanel(html.Div(id=f"{ID}-headers"), value="headers", pt="md"), ], value="rendered", id=f"{ID}-tabs", ), dcc.Interval(id=f"{ID}-boot", interval=200, max_intervals=1), ] ) @callback( Output(f"{ID}-page", "data"), Output(f"{ID}-page", "value"), Input(f"{ID}-boot", "n_intervals"), ) def _pages(_tick): import dash options = sorted( ({"value": e["path"], "label": e.get("name") or e["path"]} for e in dash.page_registry.values() if not e["path"].startswith("/admin/")), key=lambda o: o["label"], ) if not options: return no_update, no_update return options, "/" @callback( Output(f"{ID}-verdict", "children"), Output(f"{ID}-rendered", "children"), Output(f"{ID}-source", "children"), Output(f"{ID}-headers", "children"), Input(f"{ID}-page", "value"), Input(f"{ID}-ua", "value"), ) def _show(path, user_agent): if not path or not user_agent: return None, None, None, None import dash from dash_improve_my_llms.bot_detection import get_bot_type, is_any_bot is_bot = is_any_bot(user_agent) bot_class = get_bot_type(user_agent) if is_bot else "human" from dash_improve_my_llms.bot_detection import get_bot_vendor from dash_improve_my_llms.vendors import VENDORS, effective_policies vendor = next((v for v in VENDORS if v.key == get_bot_vendor(user_agent)), None) policy = "allow" if vendor is not None: try: policy = effective_policies( getattr(dash.get_app(), "_robots_config", None) )[vendor.key] except Exception: # The fold reads a live config object; a showcase must degrade # rather than break the page it is embedded in. policy = "allow" verdict = _verdict_card(is_bot, vendor, bot_class, policy) if is_bot and policy == "block": blocked = dmc.Alert( [ dmc.Text("403 Forbidden", fw=700, size="lg"), dmc.Text( "This crawler is blocked on ordinary pages by the site's " "vendor policy, so it never reaches the document below. " "The documentation surfaces (/llms.txt and the tier " "documents) stay open to it — that is deliberate, and it " "is what `block_ai_training_docs` would close.", size="sm", ), ], color="red", variant="light", icon=DashIconify(icon="tabler:ban"), ) return verdict, blocked, blocked, _headers_table( {"Status": "403 Forbidden", "Cache-Control": "no-store"} ) document = _crawler_document(path) if not document: empty = dmc.Alert( "No prerendered document is available for this page in-process.", color="yellow", variant="light", ) return verdict, empty, empty, None rendered = html.Iframe( srcDoc=document, style={"width": "100%", "height": "480px", "border": "1px solid var(--mantine-color-default-border)", "borderRadius": "8px", "background": "white"}, ) source = dmc.Paper( dmc.Code(_truncate(document), block=True, style={"maxHeight": "480px", "overflow": "auto", "display": "block"}), withBorder=True, radius="md", p="sm", ) headers = _headers_table({ "Status": "200 OK", "Content-Type": "text/html; charset=utf-8", "X-Robots-Tag": "(none — this page is indexable)", "Vary": "Accept (on //llms.txt)", }) return verdict, rendered, source, headers def _verdict_card(is_bot, vendor, bot_class, policy): rows = [ ("Classified as", "bot" if is_bot else "human (untouched by policy)"), ("Vendor", f"{vendor.display} — {vendor.operator}" if vendor else "—"), ("Class", bot_class), ("Effective policy", policy if is_bot else "n/a"), ] return dmc.Paper( dmc.Group( [ dmc.Stack( [ dmc.Text(label, size="xs", c="dimmed", tt="uppercase"), dmc.Badge( str(value), color=_VERDICT_COLOR.get(value, _CLASS_COLOR.get(value, "gray")), variant="light", size="lg", ) if label in ("Class", "Effective policy") else dmc.Text(str(value), size="sm", fw=600), ], gap=4, ) for label, value in rows ], gap="xl", ), withBorder=True, radius="md", p="md", ) def _headers_table(headers): return dmc.Table( [ dmc.TableThead(dmc.TableTr([dmc.TableTh("Header"), dmc.TableTh("Value")])), dmc.TableTbody([ dmc.TableTr([dmc.TableTd(dmc.Code(k)), dmc.TableTd(v)]) for k, v in headers.items() ]), ], striped=True, withTableBorder=True, ) def _crawler_document(path): """The crawler HTML for `path`, built by the package's own generator. Calls the pure function rather than re-implementing it, so this showcase cannot claim something the real route would not serve. """ try: import dash import dash_improve_my_llms as dimll from dash_improve_my_llms.html_generator import generate_static_page_html store = getattr(getattr(dimll, "_state", None), "page_metadata", None) or {} meta = store.get(path) if not meta: return "" all_pages = [ {"path": p, "name": (m or {}).get("name") or p} for p, m in store.items() ] app = dash.get_app() app_config = { "name": getattr(app, "title", "") or "", "base_url": getattr(app, "_base_url", "") or "", } return generate_static_page_html(path, meta, all_pages, app_config) except Exception as exc: # the showcase degrades, it never breaks the page return ( "" f"

Crawler document unavailable in-process: {_html.escape(str(exc))}

" ) def _truncate(text, limit=12000): return text if len(text) <= limit else text[:limit] + "\n… truncated …" ``` Three things worth trying: - switch between **Chrome** and **ClaudeBot** on any page — the verdict flips to `block` and the crawler gets a 403 before the document is ever built; - switch to **Googlebot** and open **View source** — that is the prerendered document, with real prose and real links, in the *initial* HTML; - switch to **PerplexityBot** — allowed, because AI *search* citations are how people find a package, while AI *training* is a different bargain. ### The policy behind the verdict One vendor registry drives both halves, so the site cannot say one thing and do another: | Class | Default | Why | |---|---|---| | AI training (GPTBot, ClaudeBot, CCBot, …) | **block** | robots.txt was the published promise; 2.7.0 made the middleware keep it | | AI search (Claude-User, ChatGPT-User, PerplexityBot, …) | **allow** | citations send readers; that is the point of docs | | Traditional (Googlebot, Bingbot, …) | **allow** | ordinary search | ```python app._robots_config = RobotsConfig( block_ai_training=True, allow_ai_search=True, allow_traditional=True, crawl_delay=10, ) ``` Since 2.7.0 the coarse flags have a finer companion — `vendor_policy`, per vendor, `allow` / `block` / `meter` — and it takes a **callable**, read on every request. That is what the control board writes through. ### The documentation surfaces stay open Note what did *not* 403 above: `/llms.txt`, `/llms-small.txt`, `/llms-full.txt` and every `//llms.txt` answer a blocked training crawler with `200`. That is deliberate. The corpus exists to get the package used, and an upgrade must not silently start refusing it. `RobotsConfig(block_ai_training_docs=True)` closes that half if you want it closed. It is opt-in on purpose. ### Live policy This site's `robots.txt` and `sitemap.xml` are generated, never hand-written, and are always open to everyone — including crawlers the site blocks elsewhere. A crawler that cannot read the rules is a crawler that has no rules: - [`/robots.txt`](/robots.txt) — generated from the vendor registry - [`/sitemap.xml`](/sitemap.xml) — with priority inferred from the page tree - [`/llms.txt`](/llms.txt) — the index, plus the cross-host network directory ### Hidden pages `mark_hidden("/admin/control-board")` removes a path from the sitemap, 404s its `llms.txt`, skips the MCP resource, and returns 404 to crawler requests on the page URL. It is **not** in `robots.txt` — a `Disallow` line publishes the path it is trying to protect. --- *Source: /audiences/web-crawlers* --- # Changelog > Version history of Dash Improve My LLMs. The timeline on this page is rendered from `CHANGELOG.md`, reproduced below. --- All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). ## [llms-2plot-dev 1.6.0] - 2026-09-03 **Sync 1.6.43, all three items.** ### Fixed - **The read table drops internal traffic too.** `track_visit` has dropped requests carrying `INTERNAL_UA_TOKEN` since the network's internal-traffic contract existed; `record_read` — the `on_document_read` hook added with the 2.8.0 floor — never learned it. So the hub's health sweep, this site's own link audit and every post-deploy battery were writing `reads` rows, and were the busiest unidentified "vendor" on this host's board. Measured before the fix: one token-carrying probe wrote one row, `vendor_key` null, crawler lane. "Counted nowhere" now includes the read table. **Reporting consequence:** read counts DROP from this release, and the drop is the network's own probes leaving the numbers. The clean window for this host's ledger starts 2026-09-02 (ops' fleet ruling); earlier rows keep the inflation and are not being re-sent, because re-importing them would re-import exactly what this fix removes. ### Changed - **Three verification traps added to `.claude/CLAUDE.md`** — which branch Render builds can be timed on a green push but is only PROVEN by a red one (this host has that proof, run 33337712632); a disagreeing lane IS the finding, and the browser lane spans three artifacts of which curl sees two; and assert a corpus is non-empty before trusting any negative, with four families of silent-green measured in this tree. ### Notes - The drop keys on `ua`, which is what the package's `EVENT_FIELDS` calls it — `user_agent` is the visits row's name and keying on it here would have been a silent no-op. Verified by reading `EVENT_FIELDS` out of the wheels rather than inferring: `ua` present at 2.8.0 (local), 2.9.0 (**what production actually resolves**) and 2.9.4 (what CI pulls from the `>=2.8.0` floor); `user_agent` present at none. 2.9.4 adds `vendor_class`, which this fork picks up automatically because the row is built by iterating `EVENT_FIELDS`. ## [llms-2plot-dev 1.5.0] - 2026-08-31 **Sync item 18 — the 1.6.41 remainder, nineteen files as one contract.** ### Fixed - **`.. exec::` reached only the browser.** A markdown2dash directive that renders Dash components puts its output in the React tree alone, while the machine lane, the prerender and the crawler HTML are built from the markdown SOURCE with the directive line stripped. Four pages published prose that referred to a component the reader could not see — `/showcase/robots-sandbox/llms.txt` read "Move the switches. The document on the right is generated by…" and then stopped. The directive now expands into the module's source through the same fence-aware pass `.. source::` uses: one parse, two consumers. - **The battery probed a path that does not exist and never probed one that does.** `network_smoke.HIDDEN_DOC_PATHS` listed `/admin/llms.txt` (a redirect, not a page) and omitted `/admin/traffic/llms.txt`, added with the ledger round. It is now pinned against the page registry. - **A bare test client sends `Werkzeug/x.y`**, which dimll ≥2.8 puts on the crawler lane, so `mark_hidden` pages 404 and every-page-200 loops go red at a floor bump. `tests/test_agent_key_route.py` now names the browser lane and keeps the internal token. ### Added - **A skip link** as the first tab stop on every page, and prop tables that scroll inside their own box. - **`nav:` frontmatter** for short sidebar labels (no page needs one here — the longest name is 18 characters). - **`/changelog` carries a real ``**, derived from the newest dated release heading. The sitemap honesty pin learns that second source rather than being loosened. - **`tests/test_exec_lane_parity.py`** — content pins, derived from the docs tree, that MUTATION-CHECK themselves. The expansion was disabled and every content pin confirmed red before being restored. - **CD fails fast when superseded** (a later commit already serving, via the compare API) instead of timing out, and refuses to push when a `release/*` branch would make `release` an unusable ref. - **`GITHUB_URL` must resolve** on the wire — a wrong repo slug is a live 404 that no amount of "profile vs repo" framing catches. ### Changed - **`header.py` and `test_nav_contract.py` are byte-copies again.** Both seams this fork reported at 1.6.38 as template-side were closed at 1.6.41 (`LOGO_ASSET` et al., and a registry-derived aside pin), so `DIVERGENCES.md` 8 is retired. The byte-identity evidence recorded against 519d496 was stale and is superseded by a re-measurement against 4ac02e0. - **`/api` remains unregistered** — `dash_improve_my_llms` ships no component metadata, so all four empty-`/api` mechanisms are moot here. ## [llms-2plot-dev 1.4.0] - 2026-08-30 **Sync items 16 and 17 — the navigation contract and the battery's lane.** ### Added - **`/changelog`** — this file as a Timeline, linked under Home in the sidebar and from the footer, and served as its own `llms.txt`. - **A footer** — © Pip Install Python LLC, the owner's GitHub profile, Discord and YouTube, every icon labelled. - **An Admin section in the sidebar, visible to the OWNER.** Both admin pages (`/admin/control-board`, `/admin/traffic`) were previously hidden from everyone including the owner by a blanket `/admin/*` path filter; they are now listed for `is_admin_user()` and for nobody else — the anonymous tree carries no `/admin/` href at all. - **An `Other Apps` menu in the top bar**, built from `network_directory.PRIMARY` rather than a hand-typed list. - **`lib/aside.py`** — the aside column collapses on pages with no `.. toc::`, so `/changelog` and `/` render full width. ### Changed - **The sidebar is generated from frontmatter**, not from three hand-typed link clusters in `components/navbar.py`. Every page already declared `category:`; the navbar simply ignored it. Each page now also declares `order:`. **`/audiences/web-crawlers` is listed once** — it appeared twice, as "Web Crawlers" under This package and as "A · What the crawler sees" under Showcase. - **The sidebar's `Pip Components` and `Other Apps I've built` sections are gone** — the network is listed once, in the top bar's menu. `Resources` is third-party only: `dmc`. No `community.plotly.com`, no `2plot.dev`. - **`pages/home.py` renders through markdown2dash**, not `dcc.Markdown`, so home's headings, tables and code fences match every docs page. - **`/admin/traffic` uses `dmc.DatePickerInput`** instead of `dcc.Dropdown`, and gains a **People** section — the day's human hits, visitors, sessions and median session — above the crawler tables, with the line that humans never enter the read ledger. - **Mobile fit**: code blocks inside a List item, Blockquote or Timeline scroll in their own box instead of widening the document; the rules are in `main.css` against public Mantine class names, never per page. - **`/api` is NOT registered here, and that is a measurement.** `dash_improve_my_llms` ships no component metadata — no `metadata.json`, no bundled JS, no generated component classes (checked on 2.8.0). It is a library of routes, middleware and config objects, so there are no props to tabulate. `API_PACKAGES` is `[]`. - **The live battery's default User-Agent names the browser lane** (item 17). At dimll ≥2.8 a UA with no browser engine token is crawler-lane, so `scripts/network_smoke.py`'s bare internal token made every default-UA check read the prerendered crawler document. A Chrome token now leads, the internal token follows it (substring match, so internal-traffic exclusion still holds), and `CRAWLER_UA` is untouched. ### Fixed - **Admin URLs no longer leak into the machine corpus.** Five pages hyperlinked `/admin/control-board` in their prose, which put an admin path in `/llms.txt`. Found by item 16's rewritten `tests/test_excluded_links_hidden.py`. The prose keeps the words and drops the link — the URL is useless to anyone who cannot authenticate, and publishing it invited crawlers to an admin path. ## [llms-2plot-dev 1.3.0] - 2026-08-30 **Round 3.4, the posture flip — consumed as sync item 15, with boilerplate.2plot.dev as the other canary.** ### Changed - **The AI-training wall is retired** (owner decision, 2026-08-30). `run.py` sets `RobotsConfig(block_ai_training=False)`. It stood while a refused read was the only thing this site could say about a training crawler; since the 2.8.0 ledger round every read is RECORDED and reconcilable — vendor, tier, verdict, bytes, verified — and the hub's Ledger tab agrees with this host's own count (llms 52 = 52). A wall that refuses the read also refuses the evidence. Per-vendor block/meter through the callable `vendor_policy` seam is the instrument now: aimed, live, revocable per request, which a blanket bucket never was. Claude-User, Claude-SearchBot, ChatGPT-User, OAI-SearchBot and PerplexityBot were never in the training bucket and are unaffected. - **In-process, both real UAs (ClaudeBot, GPTBot), before → after:** `/` 403 → 200 (13,778 B crawler document) · `/llms.txt` 200 → 200 · `/healthz` 403 → 200. On the wire at 14:09Z the pre-flip triple was identical to the in-process one, which means every 403 this host serves is the APP's — no separate edge wall has been observed here. - **Three robots.txt fingerprint checks became posture checks** (`tests/test_llms_routes.py`, `scripts/network_smoke.py`, `scripts/smoke_live.py`). `ClaudeBot -> Disallow: /` was asserted as proof the running artifact was the intended package; it was really a statement of policy, and policy changed. The vendor SPLIT still fingerprints the package (OAI-SearchBot, Claude-User, Claude-SearchBot all `Allow: /`), and ClaudeBot/GPTBot/CCBot are now asserted NOT to carry `Disallow: /`. - **`tests/test_vendor_policy.py`'s default-posture pin now asserts 200** where it asserted 403, and `tests/test_showcase.py` asserts the panel shows the served document rather than a 403 — the showcase exists to show the truth about this host, so it moves with the posture. - **On the wire after deploy (build 625c91c, 22:05Z), both real UAs:** `/` 200 (14,133 B crawler document) · `/llms.txt` 200 (13,801 B) · `/healthz` 200, and `/robots.txt` carrying `Allow: /` for ClaudeBot, GPTBot and CCBot. **No edge wall appeared when the app's came down** — every 403 this host ever served was its own, and the owner has since confirmed no Cloudflare rule exists on the zone. The `DIVERGENCES.md` posture fence is re-dated to `200/200/200`. ## [llms-2plot-dev 1.2.0] - 2026-08-29 **Consumed `SYNC-1.6.22-1.6.35` items 12 and 13 at dash-documentation-boilerplate 1.6.35 (`4c63992`).** The ledger round and the release-branch round. Both items are contract-class: ported into this fork's shape, not byte-copied. Per-item dispositions are in the sync report. ### Added - **`/admin/traffic`** — this host's own crawler ledger, behind the control board's exact gate (fails CLOSED without Clerk, for the same reason). Vendor × day, vendor → tier for a picked day, top paths per vendor, and the v3 headline numbers for the same day so the two accountings can be read side by side. Plain tables, no charts, no interval callback: a 14 × 40 table of strings is about a millisecond, five charts were ten seconds (fleet fact 18). - **The read table.** `dash-improve-my-llms` 2.8.0 emits one event per corpus document it serves (`on_document_read`) and does no I/O with it; `run.py` registers `AnalyticsTracker.record_read` once, which keeps the row as a `reads` table in the SAME analytics file — same buffer, lock, flush cadence and retention as `visits`, with `client_ip` dropped unless `ANALYTICS_KEEP_CLIENT_IP=1`. `reads` is JOINED by the rollup, never summed into `human_hits` / `bot_hits` / `pages`. - **Rollup v4**, additive and present only on a day with reads: `vendors[]` (one row per vendor key × verified × policy, with per-tier counts and bytes, null key kept as the unverifiable bulk) and `reads`. Every v3 key is byte-identical; the reporter POSTs what `daily_rollup` returns and is unchanged. - **CD promotes `main` → `release`.** `.github/workflows/cd.yml`'s `deploy` job now fast-forwards `release` to the run's own sha after the matrix is green, and Render watches `release` (`render.yaml`). `needs: [test]` is the whole gate, so `release` cannot receive an uncertified commit by construction. `tests/test_cd_promotes_release.py` pins the structure. - **A `posture` fence in `DIVERGENCES.md`** — measured, not intended: `ai_bots`, `healthz: full`, `runtime: python`, `deploy: release-branch`. ### Notes on the first promoted run - **The first promoted run went red on its build-match wait, and that was the owner step outstanding, not a defect.** `5c73a53` reached `main` at 17:43:00Z and `release` at 17:45:24Z; at 18:22:44Z `/healthz` still served `ae1dce6`. This service is not Blueprint-managed, so `render.yaml`'s `branch:` is documentation and the dashboard Branch field is the switch — item 13 says exactly this in its notes. The owner switched it at ~18:00Z and the next run (33281935425, `0081f65`) went fully green in five minutes: promote, wait, and verify including its `/healthz build == github.sha` step. `main == release == wire`. - On the same run, item 13's `verify` fix proved itself: the job was **skipped** rather than run. Under the previous `always() && != 'cancelled' && != 'skipped'` gate it would have run after the failed deploy and reported GREEN against `ae1dce6` — the previous build. That is the defect the gate change exists to prevent, observed here on the first run that could exhibit it. ### Changed - **`human_hits` DROPS and `bot_hits` RISES from this release.** UA-less and library clients (`httpx`, `Go-http-client`, `node-fetch`, an empty User-Agent) move from the human lane to the crawler lane, because `classify()` puts them there and this app no longer disagrees with it. The hub's day-over-day view will show a step on the adoption date. That is the number becoming true, not a regression. - **There is ONE classifier.** `lib/analytics_tracker.py` delegated `is_bot` and `detect_bot_type` to `dash_improve_my_llms.classify()` and now ends with ZERO User-Agent strings. Its own lists had filed ClaudeBot — Anthropic's TRAINING crawler — under "search", still named the retired `anthropic-ai` / `claude-web` tokens, and knew nothing of `bytespider` or `Claude-User`. The names and signatures are kept for callers. Crawler rows gain `vendor_key`, `vendor_class`, `verified`, `lane`; human rows are byte-identical to before, and the `INTERNAL_UA_TOKEN` drop still happens FIRST. - **The floor moves to `dash-improve-my-llms>=2.8.0`** in every encoding: `requirements.txt` (four lines), `run.py`'s `LLMS_PKG_FLOOR`, and CI's install line plus both version asserts. Not a degradable feature — `lib/analytics_tracker` imports `classify` and `_ledger.EVENT_FIELDS` at module scope. - **`verify` runs only on `needs.deploy.result == 'success'`.** The old `always() && != 'cancelled' && != 'skipped'` admitted `failure`, so a failed promote could still be followed by a green verify of the PREVIOUS build. Verify's first step now also asserts `/healthz` build == this run's sha itself. This fork's `SITE_URL` guard (DIVERGENCES.md 4) is kept as a second conjunct. ### Removed - **The Render deploy-hook secret**, and every trace of its name from `cd.yml`. A push to `main` is no longer a deploy; it is a candidate. ### Fixed - **`tests/test_proxy_scheme.py` sends `BROWSER_UA`.** At the 2.8.0 floor an absent User-Agent is the crawler lane, so the UA-less probe received the crawler document — which carries no `twitter:url` — and failed on "no tag" without saying anything about the forwarded scheme. Either lane can be the one you did not mean to test. ## [llms-2plot-dev 1.1.0] - 2026-08-27 **Kit adoption — consumed `SYNC-1.6.10-1.6.16`, `SYNC-1.6.17-1.6.21` and `SYNC-1.6.22-1.6.29` at dash-documentation-boilerplate 1.6.29 (`5589318`).** This host was a live kit-lineage site that had never been on the fleet roster; it joins with this release. Per-item dispositions are in the sync report; the deliberate differences are now recorded in `DIVERGENCES.md`. ### Added - **The `.claude/` development kit** — `CLAUDE.md` (this site's own guide above the network's behavioral contract and verification traps, both ported verbatim), `settings.json` pointed at THIS host, and the three shipped skills (`wire-verify`, `sync-template`, `report`) byte-verbatim. `.gitignore` moves from a blanket `.claude/` ignore to the template's ALLOW-LIST form, which is what keeps credentials under `.claude/` structurally uncommittable, and gains the session-document block. - **`DIVERGENCES.md`** — six recorded divergences with reasons, plus the machine-readable `byte-owned` fence (empty: this fork makes no byte-level claim on any `sync-verbatim` path, and drift is never fenced). - **`tests/test_claude_kit.py`, `tests/test_auth_demos.py`** — byte-verbatim kit cargo. **`tests/test_python_version.py`** — the one-fleet-Python agreement pins. - **`python` on `/healthz`**, all three backends, plus the battery's `python_matches_declared` check: the serving interpreter is now on the wire, so a stale image can be contradicted from outside. - **The configured-auth branch is certified.** Two pins render `lib.auth.register()` with a FAKE non-empty Clerk config — the branch a zero-secret suite had never executed, including the `pk_live` satellite-mode auto-enable that can only exist in production. - **CI asserts Docker's own health verdict** (`docker inspect .State.Health.Status`), failing on `none`: the external curl proves the app answers, never the HEALTHCHECK instruction itself. ### Changed - **One fleet Python: 3.14**, in every encoding at once — `Dockerfile` (`python:3.14-slim`, a MINOR tag: the old `3.11.8` patch pin could never receive a 3.11.x security release), the CI matrix main, the lint and pip-audit jobs, `cd.yml`'s verify job, and `render.yaml`'s `PYTHON_VERSION` (full `X.Y.Z`, as Render's native runtime requires). The window legs are 3.13 and 3.12. - **`Dockerfile` honors `$PORT`** — shell-form `CMD` with the default at the point of use (`${PORT:-8550}`), and the HEALTHCHECK probes the same variable. Exec-form `CMD` never expands env, so the old form hardcoded the port whatever the platform asked for. - **`render.yaml` names THIS host.** Every identity field still said `boilerplate` — service name, domain, `APP_BASE_URL`, `SATELLITE_APP_KEY`, `AD_APP_ID` — inherited at fork time and never corrected, while production served `llms.2plot.dev` with `app: "llms"`. `POLICY_STORE_FILE` is now declared on the mounted disk beside the other two stores. - **CD is sized for the worst build**: the build-match wait runs 100 × 15s under a 30-minute job timeout (a floor bump busts the pip cache by design, so this pipeline's most important deploy is also its slowest), a hookless deploy emits `::warning` rather than a quiet notice, and the verify job now stands down on a `skipped` deploy as well as a `cancelled` one. - **`.github/dependabot.yml`** — the 1.6.24 rewrite: the pip ecosystem is removed entirely. On range requirements dependabot can only propose FLOOR RAISES, so the old allow-list group structurally produced the very PR class it existed to suppress. Floors move through sync specs, every encoding at once. Security updates ride GitHub's separate channel. - **`scripts/smoke_live.py`** is the template's current file: the auth POST now carries the same SSL context as `fetch` (without it every POST died in the macOS handshake and read as missing auth wiring), and `wake()` tolerates a legacy `fetch` stub. A source pin in `tests/test_auth_wiring.py` holds the SSL half — no wired test can, they all monkeypatch `post`. - **The gate card promises only what ships** — "and the AI assistant" is gone from the demo-card copy; nothing here wires one. - **`lib/auth_demos.py` points at a demo this site can render.** The inherited `/examples/visualization` entry named a page and a module that exist on no fork; every gate card rendered demo-less and silent since fork time. This site's hero is `/showcase/robots-sandbox`. ### Removed - **The vestigial Node layer** (template issue #12, CVE-2026-1615, removed upstream in 1.6.9 and carried here with this Dockerfile sync). `package.json` / `package-lock.json` were dash-mantine-components' component-build toolchain, inherited through the fork lineage and used by nothing in this repo — no webpack config, no `src/ts`, no CI job, no served asset — while the image apt-installed nodejs+npm and `npm install`ed a known-vulnerable `jsonpath@1.1.1` into every production build. ## [llms-2plot-dev 1.0.0] - 2026-08-22 **This repository forked here.** Everything below this entry is the history of `dash-documentation-boilerplate`, the template this site was forked from at 1.6.7 — kept because the machinery is inherited and its reasoning still applies. Everything from here up is `llms-2plot-dev`, the documentation site for `dash-improve-my-llms` and the 2plot network's owner-control bench. ### Added - **The site's own content.** Three audience pages (`/audiences/mcp-clients`, `/audiences/web-crawlers`, `/audiences/llm-context` — URLs preserved byte for byte from the retiring service), a five-page Reference section, and three showcases that run the package's own pure handlers in-process. - **`lib/policy_store.py`** — the writable layer. Flock-guarded JSON, validated on write, atomic on replace, fail-open on read, re-stat'd on every call. Reaches `dash-improve-my-llms` 2.7.0 through its callable seams, so a control-board toggle lands on the next request in every worker with no restart. - **The control board's country guardrail** — a click-to-select world map over the inherited page-visibility board, with the admin gate re-checked server-side in the write callback. - **`BUGS-2.7.0.md`** — the pre-release soak that gates the package's tag. ### Changed - **Identity, on every surface**: brand, description, origin, favicons from the hook mark, header logo and wordmark, the GitHub link, the social-card object, and the JSON-LD blocks. - **The template's documentation is DELETED, not hidden** (owner decision). `excluded_links` hid the eleven tutorial pages from the sidebar but left them in `sitemap.xml`, `/llms.txt`, `/llms-full.txt` and the MCP resource set — so this host would have published the boilerplate's documentation as its own. This overrides the migration kickoff's "NEVER deleted (wave-sync purity)" rule: template syncs touching `docs/` now need resolving by hand, and that cost was accepted for a site that stands on its own. - Sixteen `301` redirects for retired and deleted URLs. ### Notes - `requirements.txt` still floors at `dash-improve-my-llms>=2.6.1`. 2.7.0 is unpublished, so every 2.7.0 call site sits behind the `LLMS_HAS_27` capability probe in `run.py` and the app boots on either release. --- ## [1.6.7] - 2026-08-22 ### Added - **Auth-wiring guards, both halves** (the flexlayout finding): dash-clerk-auth wires either side of `Dash(...)` — `register()` is the UI half, `configure_app(app)` the server half (`/api/auth/*` routes + per-request identity). Flexlayout's batch-2 pass shipped the first call without the second: components rendered and ClerkJS reported signed-in while every server render read signed-out — the control board served the owner the sign-in card forever, `POST /api/auth/session` answered 405 through Dash's GET-only page catch-all, and sign-out never revoked. Invisible to every suite, because Clerk is off in test environments and `configure_app` no-ops without keys. Two guards now, one per environment: `tests/test_auth_wiring.py` pins structurally (AST) that run.py calls BOTH halves; `scripts/smoke_live.py` gains an "Auth wiring" block that POSTs both endpoints on the live host (registered = 2xx/4xx; unregistered = 404/405), gated on the package's inline bootstrap being present in the served shell so clerk-off hosts skip rather than fail. Measured baselines: boilerplate answers 401/200, flexlayout answered 405/405. Note: the battery's POST probes need real egress — sandboxed environments that allow only GET report transport-0. ## [1.6.6] - 2026-08-22 ### Changed - **dimll floor 2.6.0 → 2.6.1** (requirements incl. the commented backend extras, run.py's boot floor + its message, and the test — the floor lives in more than one place; all moved together). 2.6.1 makes the universal prerender VISIBLE to non-JS consumers: below it the injected block carries a literal `hidden` attribute, so every visibility-respecting reader (html-to-text extractors, arguably crawler content-weighting) saw only "Loading..." — the outside-audit finding of 2026-08-22, diagnosed live across six hosts and fixed at the package. The generic-UA prerender test now asserts the fixed shape: div without `hidden`, plus the marked synchronous hide script that keeps JS browsers flash-free (React's mount wipes the pair, so nothing changes for humans). The fleet inherits 2.6.1 on each host's next deploy with no requirements edit; this release is the reference host's own pickup plus the floor that makes the guarantee permanent. ## [1.6.5] - 2026-08-22 Batch-1 closeout: the wave's other three hosts (emojimart, modelviewer, excalidraw) shipped dark, and four of their findings trace to this template. All four are fixed at the source so batch 2 and every future fork inherit the fix instead of rediscovering it. ### Added - **Runtime-imports guard** (`tests/test_runtime_imports.py`, the modelviewer finding): a fork died in production on a function-local `import PIL` that every dev machine happened to satisfy — suite green, boots locally, dies in a clean image, and one docs example took all ten pages down because Dash imports every page at construction. The test AST-walks every runtime module and asserts each absolute import resolves in the environment CI installs (requirements.txt and nothing else); nesting is deliberately ignored because it does not predict boot-fatality. The optional-backend exemption (fastapi/quart select by env) is earned by two companion tests: the extras must stay documented as commented requirements lines, and the carrier modules must never be hoisted to run.py's unconditional top level. A third companion pins that runtime code never imports build-time `scripts/`. - **CSS hygiene guard** (`tests/test_css_hygiene.py`, the excalidraw finding, landed at the source): fails on any hashed `.m_*` Mantine selector in `assets/*.css`. Three forks have paid for this class — leaflet's floating drawer, emojimart's 63vh drawer, and excalidraw inheriting two dead-or-harmful hashed rules **from this template**. - **modelviewer + excalidraw joined the canonical network directory** (`lib/network_directory.py`): both were deliberately absent until they deployed; both are live and build-identity-verified as of 2026-08-21/22. The fleet re-copy carries the entries everywhere. - **Markdown tables scroll in their own box** (`table.m2d-table`, GitHub's recipe: content-width, capped at the container, scrollable past it — the excalidraw finding): a `` is min-content sized, so one wide prop table dragged an entire page 105px sideways at 414px. A no-op for tables that already fit; covers kwargs prop tables too, since markdown2dash stamps the class on every table. ### Fixed - **The three hashed-selector fossils removed from `assets/main.css`** (dmc-docs fork era, present since the initial commit): `.m_46b77525` put an `!important` margin on every Input wrapper in every docs example; `.m_5caae85b` was dead in DMC 2.7 **and** 2.8; `.m_9cdde9a` restated Mantine's own aside declarations around one intentful pixel — the TOC's 15px breathing gap, which moved to the static `aside.mantine-AppShell-aside` rule. - **`scripts/make_favicons.py` now flattens the apple-touch icon onto opaque white** (the emojimart finding): iOS composites the icon's alpha onto its own background — black on some surfaces, white on others — so every fork that ran this script shipped an icon that renders differently everywhere it appears. Every other size keeps its transparency. The template's own `apple-touch-icon.png` is regenerated (the other seven files regenerated byte-identical, confirming provenance), and a header-level PNG colour-type test pins opacity without needing Pillow in CI. - **The header wordmark now hides below `xs` with the accessible name preserved** — the pattern both modelviewer and excalidraw needed and implemented divergently. `visibleFrom` keeps the node in the DOM (the typing animation still finds it) but `display:none` DOES remove it from the accessibility tree, so the home link now carries a permanent `aria-label` and the logo img is explicitly decorative (`alt=""`). Without the label, phones would get a home link with no name at all — the modelviewer defect, which excalidraw's pass reasoned incorrectly about and likely still ships. ## [1.6.4] - 2026-08-21 Two fleet-class fixes surfaced by the wave's first pair, landed at the source so the other eighteen forks inherit them. ### Fixed - **CD now verifies the artifact it shipped, not "whatever is live"** (the muicharts finding): with `RENDER_DEPLOY_HOOK_URL` unset, the old workflow skipped the wait and ran the live battery seconds after the push — against the previous release, every run, invisibly. `/healthz` now reports the running instance's commit (`RENDER_GIT_COMMIT`, optional field — the fleet probe contract is unchanged), and the CD wait holds until it matches the run's SHA, falling back once (with a warning) on builds predating the field. - **The byte-copy identity trap** (the pannellum finding): the reporter must stay byte-identical across forks, so its fallback app key says "boilerplate" everywhere — while a fork's other modules default to the fork's own key. `run.py` now claims the identity via `os.environ.setdefault("SATELLITE_APP_KEY", ...)` before any hub-facing import — the marked FORK POINT; forks change that one string and keep the reporter byte-identical. A real env value always wins. ## [1.6.3] - 2026-08-21 ### Changed - **Vendored dash-clerk-auth 1.0.4 → 1.0.5** (sha256 `a2f9062e…b74f3`, full provenance in requirements.txt). Fixes the return-trip stale gate the owner observed live: landing back on an auth-gated page after signing in on the primary showed the gate card until a manual refresh, because the first server render precedes `__dca_identity` minting. 1.0.5 syncs the session and reloads once, with a sessionStorage no-loop marker shared by both reconciliation paths. The provenance rule is now general: only the recorded sha admits a tarball — stale early builds have bitten on both of the last two releases and are indistinguishable by name, size, or date. ## [1.6.2] - 2026-08-21 The pre-wave hygiene pass, from the four-repo review. ### Fixed - **Date-skew corrections (leaflet handoff §8):** seven committed provenance stamps read `2026-08-22` for events whose verified date is `2026-08-21` (git author dates corroborate) — CHANGELOG headers 1.5.3–1.6.1, `components/header.py`, `lib/ad_client.py`, `requirements.txt`. All corrected; the three release commit SUBJECTS carrying the wrong date are immutable and stand corrected by this entry. A date nobody can trace is worse than no date. - `docs/authentication/authentication.md` now documents the control-board override layer (override → frontmatter → `PAGE_DEFAULT_TIER`, hub ceiling on top) instead of contradicting shipped behavior; `lastmod` bumped accordingly. - `run.py`'s floor failure message now names what a 2.5.x actually loses first — silently swallowed `lastmod`, the lying sitemap — matching the comment that raised the floor. - `lib/auth.py`'s signout-shim docstring caught up with reality (upstream fix shipped in 1.0.3/1.0.4; the shim is a deliberate duplicate until the fleet-wide retirement pass). ### Changed - README caught up three releases: dimll floor 2.5.1 → 2.6.0 in five places, a new Access Control & Live Page Management section (control board, admin allowlist, gate teasers), the mobile-drawer standard under UI/UX, and the admin env vars in Configuration. - `.env.example` gains the admin surface (`ADMIN_EMAILS`, `ADMIN_USER_IDS`, `ALLOW_UNGATED_ADMIN`) — the gate for the 1.6.0 headline feature was previously undiscoverable from the env template. - `.claude/CLAUDE.md` Customization Points now lists the control board, the override store, and the auth-demo teasers. ## [1.6.1] - 2026-08-21 ### Fixed - Accessibility + agentic-browsing names on the header's icon controls (hamburger, theme toggle, GitHub link — `create_link` now requires a label), and the network-ad image reserves a square box via `aspect-ratio` so the aside no longer layout-shifts when the creative loads. All three were Lighthouse findings on the pilot host measured against template code — every fork inherits the fix. ## [1.6.0] - 2026-08-21 Every fork gets its own live control board — the leaflet pilot's proven UX, ported with its scar tissue included. ### Added - **`/admin/control-board`** (`pages/control_board.py`): flip any docs page between public / auth / admin / hidden and toggle its llms.txt exposure, live — changes apply on the next render, no restart. Gated by the ADMIN_EMAILS/ADMIN_USER_IDS allowlist + owner; **fails CLOSED** without Clerk (`ALLOW_UNGATED_ADMIN=1` for local work), and the write callback re-checks the gate server-side (pattern-matching callbacks stay callable by anyone who can POST). The board stays OUT of both tier ledgers — its machine surfaces are silenced package-side via `mark_hidden()` (sitemap, llms.txt, MCP, prerender, crawler HTML all treat it as absent) so `access.gating_configured()` stays False on all-public forks and the hot path stays check-free. - **`lib/page_visibility.py`** — the override store, with both fleet lessons built in: mtime-throttled cross-worker reload (a toggle lands on every gunicorn worker within ~1s — the pilot's coin-flip defect) and loud persistence guards (boot warns when `PAGE_VISIBILITY_FILE` is unset OR points under /var/ without a real mount — the twice-observed silent-reset-per-deploy class). - Override-first resolution in `lib/access.py`: board override → frontmatter → env default, with the hub ceiling still applied on top (an override can loosen a local declaration, never a network restriction). `pages/markdown.py` registers every docs page on both ledgers from the one declared value. - The sign-in card's live-demo teaser now ships ARMED: DEMOS carries a working entry (`/examples/visualization` → the theme-aware chart), so gating that page shows "Live demo — try it" above "Authentication required — You're looking at a live preview of {page}. Create a free account to unlock the full documentation — every interactive example, the complete API reference, and the AI assistant." - `render.yaml` + `.env.example`: `PAGE_VISIBILITY_FILE` on the /var/data disk, with the blueprint-vs-dashboard drift warning inline. 14 new tests (`tests/test_control_board.py`). ## [1.5.4] - 2026-08-21 ### Changed - Navigation order: "Other Apps I've built" now sits above "Resources", and "Resources" is the LAST section — own-work ranks above third-party links, and the only section that navigates away from the network closes the list. ## [1.5.3] - 2026-08-21 ### Changed - **Vendored dash-clerk-auth 1.0.3 → 1.0.4** (sha256 `7a7c333a…cf701a`, recorded in full in requirements.txt with the stale-first-build warning). What 1.0.4 fixes, from the live network certification: the FastAPI auth endpoints were never callable (un-annotated request param → required query field → 422 on every POST — inert on this Flask host, fatal on fastapi ones), and the ghost-cookie fresh-load case — a page loading with ClerkJS signed-out while the server still held the identity now reconciles with a signout POST + single reload, which is the cross-host sign-out path no click shim can cover. `revokeServerSession` also verifies its response now. The 1.5.1 shim remains an idempotent duplicate; retirement is one clean release cycle after the fleet is on >=1.0.4. ## [1.5.2] - 2026-08-21 ### Changed - **Vendored dash-clerk-auth 1.0.2 → 1.0.3** (sha256 `2c6b40f4…da1944`, recorded in full in requirements.txt — the tarball IS the release; there is no PyPI for this package). 1.0.3 fixes sign-out revocation package-side (both entry points + the signed-in→signed-out listener transition, so sign-outs propagate across tabs and hosts), replaces the DiceBear default avatar with an inline SVG data URI (no third party in the UI path), and discards non-absolute `satellite_sign_in_redirect` values loudly. 1.5.1's app-side signout shim is idempotent alongside it and retires next release. - Provenance caveat recorded in requirements.txt: vendor from the hook repo's `dist/` artifact ONLY — its `main` currently holds a broken build (boot-time collection error on Python 3.10/3.11) until the import-fix PR lands; verify the sha before re-vendoring. ## [1.5.1] - 2026-08-21 Pilot-week hotfix: Sign Out that actually signs out, and an honest floor comment. ### Fixed - **Sign Out now revokes the server session.** dash-clerk-auth 1.0.2's logout runs `window.Clerk.signOut()` client-side and reloads — but the server keeps trusting the signed `__dca_identity` cookie (max-age `session_lifetime_days`, default **7 days**) and the Flask session it minted at sign-in, so a signed-out browser kept rendering every auth-gated page; on a shared computer the next person inherited the previous user's access. The package ships the endpoint that fixes this (`POST /api/auth/signout`) but nothing ever called it. New `lib/auth.py:_install_signout_delegation()` — a capture-phase delegate on the logout menu item (the sign-in delegation's proven pattern) — owns the click and sequences `Clerk.signOut()` FIRST (so the slow path can't re-verify `__session` and re-mint), then the server signout, then the reload, awaited so the reload never races the cookie clears. The package-side fix ships in dash-clerk-auth 1.0.3; this delegate is idempotent alongside it and retires a release after the fleet vendors `>=1.0.3`. ### Changed - **Floor-comment honesty** (`run.py`, `pages/markdown.py`): 1.5.0's claim that passing `lastmod=` "TypeErrors on anything older" was false — measured on 2.5.1 by the pip-docs+ stage-4 session, the signature is `(path, name=None, description=None, llms_doc=None, **kwargs)`, so older packages accept the date and silently ignore it. The 2.6.0 floor stays load-bearing, but for honesty (below it, every stamped date is swallowed and the sitemap goes back to swearing everything changed at build time), not crash avoidance. ## [1.5.0] - 2026-08-20 The reference host proves dimll 2.6.0 (stage 2 of the network rollout order). The floor is load-bearing: pages/markdown.py passes `lastmod=` unconditionally, which TypeErrors on anything older. ### Changed - `dash-improve-my-llms[flask]>=2.6.0` (was 2.5.1), and `LLMS_PKG_FLOOR = (2, 6, 0)`. What arrives: icon autodiscovery, truthful sitemap ``, JSON-LD `publisher.logo`, and the llms.txt viewer banner de-dup (package-side, free). - Every docs page's frontmatter now declares `lastmod:` with its REAL git last-commit date (2025-11-09 through 2026-08-19 — eleven pages, zero invented dates). The `Meta` model gains the field with a YAML-date-to-ISO validator; `register_page_metadata` passes it through; unset pages omit the tag — truth or silence. Deliberately not scripted from file mtimes, which reset on every Docker build and would re-invent the daily-lie sitemap 2.6.0 exists to end. - `configure_seo(icons=)`'s `.ico` entry moved to the `assets/favicon/favicon.ico` copy (byte-identical to the root one index.html links) so the declared list is SET-equal to what 2.6.0's discovery finds. ### Added - `tests/test_seo_icons.py`: discovery-vs-declaration set-agreement (the proof the fleet can rely on discovery alone once its pixels are right — order-inequality is not a failure, per the release notes) and sitemap-honesty pins (every emitted `` traceable to a frontmatter declaration; the undeclared home page carries none). ## [1.4.1] - 2026-08-19 ### Changed - `dash-clerk-auth` is now installed by requirements.txt (from the vendored tarball) rather than riding the image uninstalled. 1.4.0 shipped the whole sign-in surface — avatar, gate cards, delegation — but the deployed reference site could not render any of it because the package it wires was never on `sys.path`. Runtime posture is unchanged: with no `CLERK_*` keys the site is exactly as public as before, so forks inherit the capability, never a login wall. Alongside it, the fleet security floors are now asserted rather than merely permitted: `clerk-backend-api>=7.0.0,<8` and `cryptography>=50.0.0` (the four-advisory baseline dash-clerk-auth 1.0.1 widened its cap for). ## [1.4.0] - 2026-08-19 The interactive gate and the real-time half of the fleet's analytics land on the template. Humans meet a sign-in card on gated pages while agents keep reading the machine surfaces through the data window — the two lanes split onto separate axes, each flipped per host by one env var. The satellite reporter grows a presence beacon so the hub board can show "active right now" without waiting for a rollup. On THIS host the gate ships dark twice over: every tier is public, and dash-clerk-auth is deliberately not in requirements.txt (the vendored tarball exists for the docs' optional-auth install command) — the presence beacon is what this deploy turns on. ### Added - **The interactive gate** (`lib/gate_layouts.py`): every markdown docs page renders through a per-request verdict — sign-in card at HTTP 200 (with an optional live teaser demo via `lib/auth_demos.py`, table empty in the template), forbidden and 404 cards, the content on allow. The verdict is the new `access.resolve_page_access()`: docs fall open without Clerk, admin fails closed, and `?key=` never unlocks a browser layout. The gate switch is `PAGE_DEFAULT_TIER=auth` per deployment; `/`, `/getting-started` and the corpus pseudo-paths are pinned public so no env flip can gate the funnel. Card buttons ride `assets/auth_gate.js`/`.css` (satellite mode navigates to the primary with `?returnTo=`; local dev opens the Clerk modal). - **The second tier axis, `llms_public`** (frontmatter, or `LLMS_PUBLIC_DEFAULT`, default open): a gated page's machine twin — `//llms.txt`, crawler HTML, the prerender — stays public while the interactive page is gated. That split is the data-window posture, and the later agent flip is `LLMS_PUBLIC_DEFAULT=0`, env only. The exemption never applies to a hub-imposed tier: a satellite's env default cannot loosen what the network restricted. - **`GET /api/agent-key`** (`lib/agent_key.py`, all three backends): turns the browser's Clerk session into the hub-minted `?key=` that the "Copy for LLM" button (`assets/llms_copy.js`) now appends, so a copied URL keeps working inside an assistant that has no cookie. 204 for anonymous / Clerk-off / hub-down; `Cache-Control: private, no-store` always; the token is read from the `__session` cookie, never the query. - **The presence beacon** (`lib/satellite_reporter.py`): a second, fail-silent daemon thread POSTs `{app, active}` to the hub's `/api/satellite/active` every 60s (`SATELLITE_PRESENCE_INTERVAL_S`, floor 30, `0` disables) — distinct human visitors inside the session window, the same derivation as the hub's own count. Display-only and ephemeral hub-side; the daily rollup stays the sole source of the daily numbers. A hub that predates the endpoint 404s harmlessly. - Clerk avatar in the header (`components/header.py::create_clerk_avatar`), rendered only when Clerk is configured. ### Changed - `render.yaml`: rollup cadence `SATELLITE_REPORT_INTERVAL_S=900` (the fleet is on paid instances and the hub board now reads near-real-time), the full Clerk satellite env block, and the two gate knobs — remembering that env/plan changes apply on Blueprint sync, not git push. - `lib/auth.py`: the hand-rolled 0.9.0/0.9.1 satellite fixups are retired — both are upstream in the vendored dash-clerk-auth 1.0.2. What remains is capture-phase *delegation* (`_install_satellite_signin_delegation`, back-ported from the leaflet pilot 2026-08-19): late-rendered `#clerk-login-button`s get exactly one handler, preferring `buildSatelliteRedirect()` with `?returnTo=`, falling back to `redirectToSignIn` on origin+pathname so stale `__clerk_*` params never ride into the next sign-in. - The corpus pseudo-paths (`/llms-small.txt`, `/llms-full.txt`) register `public` explicitly instead of falling through the tier default, so `PAGE_DEFAULT_TIER` can never gate them; `/` likewise (it registers via pages/home.py, which no frontmatter ever tiers). - Vendored `dash_clerk_auth` 0.9.1 → 1.0.2 (the clerk-backend-api `<8` cap for the `cryptography>=50` floor, plus the avatar session fix). ### Fixed - The peer-host key-leak test judges parsed origins, not substrings — bare-host matching flags a site's own links whenever a peer host is a substring of its own (`2plot.dev` ⊂ `leaflet.2plot.dev`; found by the leaflet pilot, this repo was saved only by its hostname). The invariant stated properly: any URL carrying a key must be same-origin. - `lib/agent_key.py` records why it must not use `from __future__ import annotations`: PEP 563 turns the FastAPI `Request` annotation into a string resolved against module globals, where the locally imported class does not exist — the parameter silently becomes a required query field and the route 422s. ## [1.3.0] - 2026-08-15 Instrument first: the 402 groundwork lands on the template. The network's metered lane is gated on ~30 days of crawl data (owner decision 2026-08-10); this release is what makes that data exist and stay true on every satellite forked from here — machine-surface demand reported per document, counted once, tested, and tierable per deployment. No payment code ships here. Rollout plan: `kickoff/KICKOFF-x402-instrumentation-rollout.md` (local). ### Added - **The daily rollup now reports the machine surfaces** (the network's v3 analytics fields): unique bot visitors per day (`bot_visitors`, a daily distinct count), and llms.txt / robots / sitemap / page.json rows in `pages` with a per-row bot split — mirroring the hub's own self-report semantics exactly. These fetches were always recorded; they were only hidden from the report. A day with only machine-surface fetches is now reported instead of skipped — crawlers hammering llms.txt with zero human visits is exactly the signal the hub's day-pass board exists to see. - **The machine-surface rollup is tested** (`tests/test_traffic_rollup.py`) — 15 hand-checkable cases pinning the partition (every path is a page visit or a machine-surface hit, never both), the machine-only-day report, the per-row bot split, and the distinct `bot_visitors` count. This data is the evidence base for the network's 402 pricing decision; untested measurement code deciding a revenue model was the wrong risk to carry. - **Tier registrations for the corpus documents.** Every satellite built from this template now declares access tiers for `/llms-small.txt` and `/llms-full.txt` (served by dash-improve-my-llms ≥ 2.4.0; inert on older versions): `LLMS_SMALL_TIER` / `LLMS_FULL_TIER` env vars set them locally (unset = public; documented in `.env.example` and visible in `render.yaml` so every fork sees the knob), and the hub's page-tier ceilings can tighten either network-wide with no redeploy here. The dependency-floor message notes the 2.4.0 requirement for the tier documents. - **Generic version placeholder `{{VERSION:}}`** (new `lib/versions.py`, used by both markdown loaders). Prose may now state the installed version of *any* package — not just dash-improve-my-llms — so every satellite can write `{{VERSION:}}` for the library it documents and a package upgrade propagates to the browser page, the copy button, `/llms.txt` and every `//llms.txt` on the next deploy, with no prose edit. `{{DIMLL_VERSION}}` remains as a legacy alias. Fenced code blocks and inline code spans are left verbatim (the network-standard page shows the syntax in a fence), and a placeholder naming an uninstalled distribution fails the boot instead of leaking. The identity tests now also sweep for bold version claims next to any PyPI link, not only dash-improve-my-llms's. ### Fixed - **Machine-surface fetches were double-counted.** `_SKIP` excluded `/llms.txt`, `/robots` and `/sitemap` from page visits by substring — but `/llms.txt` does not substring-match `/llms-small.txt`, so the tier documents and `page.json` twins landed in BOTH `load_visits` and `load_agent_hits`, inflating `human_hits`/`bot_hits`/`pages` for exactly the surfaces the 402 board prices. `_SKIP` now names all three. The hub's `traffic_insights._SKIP` has the same gap (its comment claims the exclusion; its tuple doesn't deliver it) — port this fix there before the data window opens. - **Dash-built components rendered empty props tables.** The numpy-docstring branch in `lib/directives/kwargs.py` (for dash-mantine-components' hand-written docs) shadowed the base markdown2dash parser for the `Keyword arguments:` format that dash-generate-components emits — the format of every component a library satellite documents — so their `.. kwargs::` tables rendered silently empty. Found on muicharts' `/api`; pannellum's likely affected too. The directive now falls back to the base parser for that shape. ## [1.2.5] - 2026-08-01 ### Fixed — `scripts/smoke_live.py` failed CD on healthy sites The post-deploy battery is the fleet's deploy gate: `cd.yml` runs it against the live host after every merge and its exit code decides whether the run goes green. Its `fetch` was a single `urlopen` — no retry, no wake-up — while most of the fleet sits on Render tiers where a cold start or a dropped connection is routine. Measured on dash-flows-upgraded: two runs minutes apart against the same host, `FAIL canonical on /interactions` then `ok canonical on /interactions`. A misdiagnosed failure is worse than a slow one; it sends you to look at canonical tags that were correct all along. Both fixes already existed in-fleet and never met (blueprint LESSONS §21 states the rule outright): - **A wake-up loop before the first check.** `/healthz` is polled up to 24 times, 10s apart — deliberately wider than §21's "12×5s is plenty", because a free-tier cold start routinely takes 60–90s and the window only costs time when the host is actually down. Awake means `ok: true`, not any 200: Render's loading page and a CDN error page can both be 200s. A host that never wakes is ONE failure ("nothing else was tested"), not a cascade of forty per-check failures that all mean the same thing. - **A retry ladder inside `fetch`** — the shape `scripts/network_smoke.py` already had, and that leaflet's copy of this very script grew without the fix ever flowing back to the canonical here. Transport errors and 5xx retry with backoff; 2xx/3xx/4xx return immediately, because a 404 is a verdict and retrying it only slows the battery. Retries print to the CD log — a green run that shows retries is a host worth watching. Proven live before shipping, twice over: on the first run after the change, flows' llms.txt dropped the connection mid-body (`IncompleteRead`) and passed on retry — the exact flake that triggered this fix — and a run against email.2plot.dev saw its OWN pages do the same on two *fatal* checks that would have turned that deploy red. Tunables (env, so satellites stretch them without editing the file): `SMOKE_WAKE_ATTEMPTS`, `SMOKE_WAKE_INTERVAL_S`, `SMOKE_FETCH_RETRIES`. Exit semantics unchanged; no check weakened, removed, or reordered. The file remains the canonical copy — satellites take it verbatim on their next touch. ## [1.2.4] - 2026-08-01 ### Fixed — the network bulletin was never wired up `NETWORK_BULLETIN_URL` has been set in production, pointing at a hub endpoint that works, against code that never read it. The wiring sat **commented out** in `run.py` under a note saying "2plot.dev does not serve /api/network/bulletin yet". The hub started serving it; the comment did not change. Nothing failed. `configure_bulletin` is opt-in, so an unwired app makes no request at all and the viewer header renders perfectly well on the package's built-in tips and an "No announcements." empty state. The only symptom was an announcement that never appeared — which nobody goes looking for. Now `lib/bulletin.py`, shaped like `lib/proxy.py` and `lib/access.py`: a `configure()` that returns whether it wired, and a boot line that says which of the two states the process is in. No commented-out code to go stale, and `tests/test_bulletin.py::test_run_py_wires_it_rather_than_leaving_it_commented_out` fails the moment someone comments it out again — commented wiring cannot define the name it asserts on. Two details worth keeping: - **`app_id` comes from `SATELLITE_APP_KEY`**, reused from `lib.satellite_reporter.app_key()` rather than hard-coded. The hub scopes announcements by `?app=` and uses it to see which satellites actually render the bulletin, so a fork left announcing itself as `boilerplate` would receive this template's news *and* be miscounted. One notion of "which satellite am I", not two that can disagree. - **The TTL is floored at 60s.** It is configurable via `NETWORK_BULLETIN_TTL_S`, and a small value would refetch on nearly every llms.txt view; junk falls back to the default rather than raising at boot. Verified end to end against the live hub: the rendered header carries the hub's own tip wording ("Append /llms.txt to any URL") rather than the package's default ("Append /llms.txt to any page URL"), and the current announcement. One thing that cost time and is worth recording: on macOS the package's bulletin client fails with `CERTIFICATE_VERIFY_FAILED`, because it uses a bare `urlopen` with no CA bundle and the system Python has no OS trust-store integration. That is a local-development artifact only — Linux containers have a working store — but locally it looks exactly like a broken fetch. Run with `SSL_CERT_FILE=$(python -c "import certifi;print(certifi.where())")` to tell the two apart. `scripts/smoke_live.py` and `scripts/audit_links.py` already carry their own certifi context for the same reason. ### Changed — one identifier for this app on every hub surface `AD_APP_ID` now defaults to **`boilerplate`**, not `dash-documentation-boilerplate`. Four modules present an identity to the hub — `lib/ad_client.py`, `lib/satellite_reporter.py`, `lib/hub_client.py` and `lib/bulletin.py` — each with its own fallback, and the ad client was the odd one out. The visible cost was a column on `/admin/ad-board` that did not line up with `/traffic`; the invisible one is that `hub_client.app_id()` falls back to `AD_APP_ID` when `SATELLITE_APP_KEY` is unset, so a deployment that set the long name for ads alone was silently presenting it as its hub identity too. `lib/satellite_reporter.app_key()` still refuses to chain to `AD_APP_ID`. The two agreeing here is a convenience, not a contract — leaflet.2plot.dev runs `AD_APP_ID=dash-leaflet2` against directory key `leaflet`, and setting one for ads must never re-key a satellite's analytics series. `render.yaml` now sets `AD_APP_ID` explicitly rather than leaning on the code default, so the deployed value is visible in the blueprint. **This splits ad history.** The ad server keys impressions and clicks by `app`, so anything already logged under `dash-documentation-boilerplate` stays there — worth a look at `/admin/ad-board` on 2plot.dev before assuming the numbers reset. ### Added — `.env.example` The repo had none, so every configurable was discoverable only by reading `lib/`. Each block states what turns ON when set and what the app does when it is not — because almost every one of these fails silently rather than loudly: no `APP_BASE_URL` deindexes a fork, no `CROSS_APP_WEBHOOK_SECRET` means the hub simply never charts this app, no `NETWORK_BULLETIN_URL` renders a header that looks complete. Not gitignored (the pattern is `.env`, exactly), and `.dockerignore` already whitelists it against the `.env*` exclusion added in 1.2.2. `render.yaml` gains `NETWORK_BULLETIN_URL` so the deployment documents itself rather than depending on someone remembering to set it in the dashboard. ## [1.2.3] - 2026-08-01 **The social card, finished.** 1.2.2 closed three of four defects and left this one open because the artwork did not exist. It does now. ### Added — `scripts/make_social_card.py` Renders the 1200×630 card: the artwork composited onto a frame carrying the brand, tagline and domain, using the manifest's own `background_color` and `theme_color` so the card, the browser chrome and the install splash cannot disagree. Output lands in `build/social-cards/.png`, which is gitignored. A TEMPLATE FILE, and that is the point — pass `--brand/--tagline/--domain` and every satellite is framed identically, instead of each card being made by hand once and drifting. Three details that are not incidental: the artwork's alpha bounding box is cropped before fitting (`assets/ddb.png` carries ~66px of transparent margin that would otherwise be centred as if it were image); a brand too long for two lines shrinks once rather than colliding with the domain strip; and fonts resolve from a candidate list (macOS, then Debian/Ubuntu) rather than being bundled, because shipping a licensed TTF in a template every satellite forks is a question best not answered. Pillow stays out of `requirements.txt`. Nothing at runtime renders images, and a docs site should not carry an image library into production for a script run by hand every few months. **1200×630 = 1.91:1**, the Open Graph documented ideal, which also degrades cleanly into Twitter's 2:1 `summary_large_image` slot. Deliberately not leaflet's 1280×515 (2.49:1), which is wider than both and gets cropped on each — and what sits at that URL today is the 2plot wordmark rather than a per-site card at all. This is the shape for the network to converge on. ### Changed — `og:image` moved to the CDN ``` was: https://boilerplate.2plot.dev/assets/ddb.png 784×741 (1.06:1) now: https://cdn.2plot.ai/github_assets/boilerplate.2plot.dev.png 1200×630 ``` The old image's declared dimensions were honest, so nothing was broken — it was simply near-square, and `summary_large_image` letterboxed it into a wide slot with bars either side. Moving it off the app is the network rule and it is about cold starts, not tidiness: a card the app serves is fetched by the scraper at unfurl time, and on a cold free-tier container that request lands mid-wake and times out. The preview renders blank **once**, and the platform caches the miss — so the first person to share the link poisons it for everyone. The CDN has no cold start. `og:image:secure_url` and `og:image:type` join the auxiliaries in `templates/index.html`, matching what leaflet carries. Both are tags Dash does not emit, which is the only reason they belong in the template. ### Added — the live card check that no offline test can make The card's dimensions are now declared in **three** places: `lib/constants.py`, `templates/index.html`, and the CDN object itself. `test_social_card.py` pins the first two against each other, but nothing offline can look at the third — so replacing the uploaded file with a differently-shaped one would leave every test green while the platform reserves the wrong box and crops into it. `scripts/smoke_live.py` now fetches the real file after every deploy and reads its actual pixel dimensions out of the PNG's IHDR chunk, checking them against the declared tags, plus the ratio, plus that `og:image` is neither empty nor app-served. Two tests prove the check fires rather than merely existing: `test_a_reshaped_card_on_the_cdn_fails_the_deploy` and `test_an_empty_og_image_fails_the_deploy`. That second case is not hypothetical — it is 2plot.dev's live state today, and the reason `kickoff/` now holds a handoff for it. `fetch()` in that script changed from `errors="replace"` to `errors="surrogateescape"` to make this possible. `"replace"` substitutes U+FFFD for every invalid byte and is one-way, so the PNG header was gone before it could be read; surrogateescape round-trips exactly and behaves identically for text. ### Changed — the two peer tests narrowed to peers `test_smoke_script_rejects_a_peer_serving_its_spa_shell` and `test_a_dead_peer_is_reported_but_does_not_fail_the_deploy` stubbed *every* off-host URL, which now included the CDN-hosted card and failed the (correctly fatal) card checks. The card is off-host but it is this deployment's own responsibility, not a peer's — the distinction 1.2.2 drew between "this host is fatal, somebody else's host is a warning" holds, the stubs just needed to respect it. ### Changed — `build/` and `kickoff/` are gitignored `build/` because the card is published to the CDN and never committed or served. `kickoff/` because handoff notes start a session in *another* repo: a task list for 2plot.dev has no business in the template's checkout, and every satellite forking this repo would inherit a to-do that was never theirs. ## [1.2.2] - 2026-08-01 **Finishing 1.2.1, and the three things it exposed.** 1.2.1 shipped the right template and half the change. Everything below was measured against the live site rather than a local boot, because the local/deployed gap is precisely what hid the first defect for a day. ### Fixed — the 1.2.1 files were never committed `assets/favicon/` (the whole icon set plus `site.webmanifest`) and `tests/test_social_card.py` were sitting UNTRACKED. The committed template pointed at `/assets/favicon/…`, the deploy builds from git, so production 404'd the manifest, the apple-touch-icon and every PNG icon link — the entire installable-app surface — while every local boot looked perfect because the files were on disk. Nothing in the app reported it; `git status` was the only place it appeared. Measured on the live site: ``` /assets/favicon/site.webmanifest 404 /assets/favicon/apple-touch-icon.png 404 /assets/favicon/favicon-32x32.png 404 ``` The guard test was untracked too, so the one thing that would have caught this had never run in CI either. Both are now tracked, and `test_every_asset_the_template_references_resolves` widens the check from "the manifest icons resolve" to "**every** `/assets/…` the template references resolves" — because the failure was never about icons, it was about a template referencing a file the repository does not have. A checkout is what CI tests, so it fails there the moment something is not committed. The manifest's contents needed no change; they were already correct. ### Fixed — the fork source's brand on every share card `PAGE_TITLE_PREFIX` still read `"Dash Pip Components | "`, inherited from the upstream this template was forked from and never changed. That is not only a browser-tab string: Dash passes each page's title straight into `og:title` and `twitter:title` (`dash/_pages.py:_page_meta_tags`), so every unfurl of `boilerplate.2plot.dev` advertised **a different site**, while ``, `og:site_name` and the `/llms.txt` H1 all correctly said this one. Now `f"{SITE_SHORT_NAME} | "`, matching the network convention the other satellites already use (`dash-leaflet2 | `, `Dash Email | `) and derived from the brand rather than retyped, so the two cannot drift. `tests/test_site_identity.py` pins the prefix, the derivation, the rendered `og:title`/`twitter:title`, and sweeps the identity surfaces for any surviving mention of the old brand. Nobody sees their own share cards, which is the whole reason this needed a test rather than a look at the page. ### Fixed — `twitter:url` advertised `http://` (`lib/proxy.py`) Dash builds that tag from `request.url`, and on Flask `request.url` comes from `wsgi.url_scheme`. Requests arrive over Cloudflare → Render → gunicorn and the last hop is plaintext, so production told every social scraper `http://boilerplate.2plot.dev/`. `og:url` looked fine throughout because the template hard-codes it. gunicorn does try to fix this — it rewrites the scheme from `X-Forwarded-Proto`, but only when the immediate peer is in `forwarded_allow_ips`, which defaults to `127.0.0.1`. Reading the header ourselves one layer above gunicorn sidesteps the question entirely: `HTTP_X_FORWARDED_PROTO` is in the environ either way. Notes on the implementation, all of them load-bearing: - **Only the scheme is taken.** Host is not rewritten from `X-Forwarded-Host`; `BASE_URL` is already this project's single source of truth for the public origin, and a second header-derived notion of "what host am I" is how a fork ends up serving two. - **The FIRST entry of the header wins.** Proxies append, as with `X-Forwarded-For`, so the last entry is the hop nearest the app — the plaintext one being seen past. Reading from the wrong end reinstates the bug and still passes a single-proxy test, so there is a test for it. - **`TRUST_PROXY_HEADERS=0`** turns it off. This trusts a header from whoever connected, which is correct behind Render (it overwrites the header on every inbound request) and wrong for an app exposed directly, where a client could forge it. - **The server object is wrapped, never rebound** — `app.server` stays the Flask/FastAPI/Quart instance that gunicorn imports as `run:server` and that `run.py` hangs `before_request` off. All three backends are handled. The sibling `leaflet.2plot.dev` already serves `https` in the same tag from an identical Cloudflare/Render/gunicorn stack with no proxy configuration of its own; the difference we could observe is that it deploys as a Docker service rather than a native one, which would plausibly put the proxy on loopback and satisfy gunicorn's default. That is inference — Render's internal topology is not visible to us — and the fix deliberately does not depend on which explanation is true. ### Added — client-side URL sync on SPA navigation Ported from `leaflet.2plot.dev` and adapted: that site hard-codes a static canonical and this one does not (dash-improve-my-llms injects a per-page one), so this version only ever *corrects* tags that exist and never creates one. Three tags go stale after the first client-side route change, each for a different reason: `og:url` is static in the template, `twitter:url` is server-rendered from the entry request, and the injected canonical is right on arrival and wrong thereafter. Dash routes through `history.pushState`, which fires no event, so the tags advertise the landing URL for the rest of the session. The origin is read from the existing `og:url` tag rather than hard-coded a second time. This helps Google, which runs JS. It cannot help social scrapers, which do not — which is why the scheme half had to be fixed server-side. ### Changed — `test_exactly_one_canonical_tag_for_browsers` counts elements It counted the substring `rel="canonical"`, and the new sync script's selector (`link[rel="canonical"]`) is not a canonical tag. Same lesson as the `dv-banner` chrome check it sits beside: match the markup, not the words, so a file may legitimately discuss what it is being checked for. ### Still open — the card image is not on the CDN `og:image` remains `/assets/ddb.png`, 784×741, served by the app. The declared dimensions match the file honestly, so nothing is broken, but it misses two network rules: cards belong on `cdn.2plot.ai` so a cold free-tier container cannot blank a preview, and `summary_large_image` wants roughly 1.91:1 (leaflet's is 1280×515). `https://cdn.2plot.ai/github_assets/boilerplate.2plot.dev.png` does not exist yet, and pointing `og:image` at a 404 is strictly worse than the present state, so this waits on the asset. When it lands, the change is `OG_IMAGE_URL` plus the width/height constants, plus the `og:image:secure_url` and `og:image:type` tags leaflet carries. ## [1.2.1] - 2026-07-31 **The social card and the installable app** — the two surfaces that live entirely outside the app, and so fail where nobody is looking. Found while rolling the standard onto `leaflet.2plot.dev`, which inherited the same shapes from this template. Satellites copy `tests/test_social_card.py` verbatim. ### Fixed - **Two `og:image` tags per page, and the wrong one won.** `templates/index.html` declared `og:image` / `twitter:image` statically while Dash also emits both per page. With no `image_url=` passed, Dash *inferred* an image from the assets folder, found `assets/logo.svg`, and emitted it alongside the static tag. Every major scraper rejects SVG, and the inferred tag came last — so the card described so carefully in the template lost to an image nothing can render. `lib/constants.OG_IMAGE_URL` is now passed to `register_page`, and the template keeps only the auxiliaries Dash omits. - **The same duplication across nine other tags** — `description`, `og:type`, `og:title`, `og:description`, `twitter:card`, `twitter:url`, `twitter:title`, `twitter:description`, `twitter:image` were all declared statically *and* emitted by Dash. The static copies described the site where Dash's describe the page, so the duplicate was both redundant and the less accurate of the two. `test_no_meta_tag_dash_emits_is_also_declared_statically` pins the rule. - **The home page published an empty `description`.** `pages/home.py` never passed one, so Dash emitted `description`, `og:description` and `twitter:description` as `content=""` on the most-linked page on the site. - **The web app manifest was inert, and named the wrong product.** Its link and the `apple-touch-icon` were commented out behind a note saying the files were missing — a note that outlived their arrival in `assets/favicon/` — and the commented hrefs pointed at `/assets/`, one level above where they live. The manifest itself still read *"Dash Email — Email components for Plotly Dash"*, copied in from another repo; that string is what an installed app would have shown on the home screen. Fixed, linked, and its `theme_color` reconciled with the `theme-color` meta tag. ### Added - `tests/test_social_card.py` — a template file. Asserts the image is declared exactly once, is absolute, is not an SVG and resolves; that the manifest is linked, served, correctly named and has resolving icons; and that `templates/index.html` is still wired in, since it looks removable ( dash-improve-my-llms appears to cover OpenGraph) and is not — its injection runs only on the prerender path, which social scrapers do not take. - `lib/constants.OG_IMAGE_URL` / `_WIDTH` / `_HEIGHT` / `_ALT` — the per-site values a fork changes. ## [1.2.0] - 2026-07-31 **The 2plot network standard, landed on the template.** `2plot.ai` (the network root) and `2plot.dev` (the section hub) shipped this first; satellites are next, and this repo is the one they fork. So the point of this release is not that `boilerplate.2plot.dev` complies — it is that the files a satellite copies verbatim now carry the standard with them. The new [Network Standard](https://boilerplate.2plot.dev/network-standard) page is the per-site checklist. The three obligations below share a shape, and it is worth naming: **every failure they prevent is silent.** Nothing errors, no dashboard turns red, and the damage accumulates for months. That is why each one is now pinned by a test rather than by a convention. ### Added — explicit site identity (`lib/constants.SITE_BRAND`) One constant, `"Dash Documentation Boilerplate — the 2plot network's template"`, now reaches every surface that states what this site is: `Dash(title=)`, `register_page_metadata(path="/", name=…)`, the first line of `pages/home.md`, and `templates/index.html` (`og:site_name`, `og:title`, `twitter:title`, the schema.org `SoftwareApplication.name`, the `<noscript>` heading). What this fixes is not cosmetic. `dash-improve-my-llms` resolves the `/llms.txt` H1 and the llms viewer's brand chip through `resolve_site_title(home_page_name, app.title)`, and given nothing useful it publishes what it finds. On this host that was the `Dash()` constructor's default title: every agent that fetched `boilerplate.2plot.dev/llms.txt` cold was told the site is called **"Dash"**. The page rendered perfectly the whole time. 2.3.4 fixed half of it — generic candidates (`Home`, `Index`, `Dash`) are now skipped rather than served — but a package cannot invent a name; the other half is stating one. Naming rules, from the standard: the brand says what the site *is*; the package name (`dash-documentation-boilerplate`) belongs in the description; "Pip Install Python" is the byline and never the site name. `tests/test_site_identity.py` pins all of it, including the direction that is easy to lose — that `SITE_BRAND` is not itself one of the generic values the package skips. ### Added — the internal-traffic contract, both halves The point of truth is [2plot.ai's satellite-analytics document](https://2plot.ai/docs/satellite-analytics), "Internal traffic": any request whose User-Agent contains `2plot-internal` is network machinery talking to itself and is counted **nowhere**. *Inbound.* `lib/analytics_tracker.track_visit` drops token-carrying requests at write time, **before** `detect_device_type`. The ordering is the whole point: a health sweep and a CI battery both look like bots, so classified first they land in `bot_hits` and get reported to the hub as crawler interest in these docs. `/healthz` and `/health` stopped being stored at all — `lib/traffic_rollup` already filtered them on the way out, but a row that exists and must be discounted is still a row somebody has to know about. *Outbound — the half that was missing here.* Every call this host makes to another network host now sends `INTERNAL_UA`: - `lib/ad_client.py` → `2plot.dev`, **once per docs page view**; - `lib/satellite_reporter.py` → `2plot.ai`, hourly; - `lib/hub_client.py` → `2plot.dev`, per agent-key verify and tier fetch; - `scripts/network_smoke.py`, `scripts/smoke_live.py`, `scripts/audit_links.py`. The ad client is the one that mattered. All of these were arriving as `python-requests/2.x`, which matches the hub's own bot patterns — so this satellite's readers were inflating 2plot.dev's `bot_hits`, once per page view, and had been for as long as the ad slot has existed. The battery scripts keep their Googlebot and Chrome tokens *and* append the internal one: the target still exercises exactly the path under test, it just knows the caller is machinery. The click beacon is the deliberate exception — a browser cannot set a User-Agent, and a click is a real person. `tests/test_internal_traffic.py` proves the exclusion reaches the numbers the hub actually charts (`human_hits` / `bot_hits` in `daily_rollup`), proves the positive case still counts (a rule that drops everything would satisfy the negative assertions), and asserts the outbound header on all three clients and all three scripts. ### Added — `scripts/network_smoke.py`, run in three seats The same named checks against the CI container, against production after a deploy, and in-process from `tests/test_network_smoke.py`, so a failure reads identically wherever it happens. It proves identity (the `/llms.txt` H1 is the brand, verbatim), the deployed artifact (the robots.txt crawler split, which is the only fingerprint visible from outside — pip metadata is not), that no owner-only surface leaks, that a crawler gets prose and not the JavaScript stub, and that agents and browsers get different content types under a `Vary: Accept`. The in-process seat is not redundant: a script that only ever runs in CI and after a deploy is exactly the code that rots, where a typo turns a check into a silent pass. That test also breaks a check on purpose and requires the battery to report it. ### Changed — CI on the network baseline `.github/workflows/ci.yml` is now a template file in its own right: least-privilege `permissions: contents: read`, `timeout-minutes` on every job (the default is six hours, which is how one hung `curl` burns a day of runner minutes), `docker/setup-buildx-action` with a `type=gha` cache, and version fingerprints asserted **inside the built image** rather than in the runner. The container is booted and probed by the battery before anything is allowed to merge. `cd.yml` runs the battery against the live host before `smoke_live.py`. `tests/conftest.py` now boots the app secretless, the way CI's container does: every `CLERK_*`, `CROSS_APP_WEBHOOK_SECRET` and `SESSION_SECRET` is pinned to `""` **before** `run.py` is imported, because `load_dotenv()` runs during that import and a developer's local `.env` would otherwise flip the app into a configured posture and quietly invalidate every fail-closed assertion in `tests/test_access.py`. The analytics ledger moves to a temp dir in the same block — the suite had been appending its own hits to the repo's checked-out `visitor_analytics.json`. Added `.github/dependabot.yml` with a `dash-network` group (a package release lands as one reviewable PR per repo, not five) and an advisory `pip-audit` job. ### Changed — dependency floors - **`dash-improve-my-llms` >= 2.3.4** (from 2.3.2). The network standard; `run.py`'s startup floor and CI's in-image fingerprint both assert it. There is no vendored copy of this package anywhere in the repo — the stale comments in `Dockerfile`, `render.yaml` and `README.md` that still described one are gone. `vendor/` holds `dash_clerk_auth` alone. - **`gunicorn` >= 23.0.0** (from 21.2.0). 21.x carried two HTTP request-smuggling CVEs (CVE-2024-6827, CVE-2024-1135), both fixed in 23.0. `markdown2dash` 0.1.2 declares `gunicorn>=21.2.0,<22.0.0` — a markdown parser pinning a WSGI server — which pip cannot reconcile with that floor, so markdown2dash is installed with `--no-deps` and its real dependencies (`docutils`, `jsonpath`, `mistune`) are listed in `requirements.txt` instead. Every install path does the same two commands: `requirements.txt`, `scripts/dev.sh`, the `Dockerfile`, `render.yaml`'s `buildCommand`, CI, and the README. CI's in-image assert is what keeps the dodge honest. ### Added — `.dockerignore` Found by booting the image locally as part of verifying this release: the Dockerfile ends in `COPY . .`, so a developer's `.env` was being baked into the production image. The container died at boot with `Could not import dash.backends._fastapi` — the local file said `DASH_BACKEND=fastapi` and the image has no FastAPI extra. It never appeared in CI, where the checkout has no `.env`, which is precisely what made it worth a file rather than a lesson: the same `COPY` would carry real Clerk keys and the webhook secret into an image layer on any machine that has them. The ledger, session store, virtualenv and `node_modules` are excluded too. `docs/**/*.md` deliberately is **not** — those files *are* the app. ### Note on versioning 1.1.0 was declared in `README.md` and `lib/constants.APP_VERSION` but never cut here; everything previously sitting under `[Unreleased]` ships as part of 1.2.0. `templates/index.html`'s `softwareVersion` and `APP_VERSION` now agree, which `tests/test_config.py` asserts. --- Previously unreleased, now shipping as part of 1.2.0 — three threads of work: the CI/CD system, network analytics reporting, and the upgrade to `dash-improve-my-llms` 2.2.0. 2.1.0 was assigned during that package's development and never published, so there is no 2.1.0 anywhere and 2.0.0 upgrades straight to 2.2.0. Work described here as "2.1-era" in earlier drafts shipped as part of 2.2.0. ### Changed — dash-improve-my-llms from PyPI (2.3.3); vendored copy removed The four-host verification gate passed, `dash-improve-my-llms` published, and this repo switched from the vendored sdist to the PyPI pin (`dash-improve-my-llms[flask]>=2.3.2`) — the Phase-5 step the vendor block always anticipated. `vendor/dash_improve_my_llms-*.tar.gz` is gone; CI's ASGI legs and the Dockerfile install from PyPI too. `vendor/` still carries `dash_clerk_auth` (not on PyPI, deliberately outside requirements.txt). The floor resolves to 2.3.3, which recategorises the Anthropic crawlers: `ClaudeBot` — the actual *training* crawler — moves to `Disallow`, while the user-triggered and search fetchers `Claude-User` / `Claude-SearchBot` are allowed, matching the intent the OAI-SearchBot fix established for OpenAI. It also strips unexpanded directive lines from resolved prose. The artifact fingerprint in `tests/test_llms_routes.py` and `scripts/smoke_live.py` now asserts the full crawler split, so a host running a stale build fails its post-deploy battery by name. Verifying that fingerprint exposed a real misconfiguration: `run.py` set `block_ai_training=False`, so the training bucket was never emitted and every training crawler was silently allowed — the opposite of the "blocks AI training, allows AI search" policy this project documents, and it would have made 2.3.3's ClaudeBot recategorisation invisible on this host. Now `block_ai_training=True`, matching the documented policy and the rest of the network. ### Changed — production rollout: re-vendor 2.3.2 / 0.9.1, live hub contract Deployment prep for `boilerplate.2plot.dev` (rollout step 4; the hub's auth endpoints are now live in production). - **`dash-improve-my-llms` 2.3.0 → 2.3.2** (vendored). The vendored 2.3.0 was a pre-fix build whose robots.txt disallowed OAI-SearchBot — ChatGPT search's crawler, exactly the audience these surfaces exist for. 2.3.2 allows it. `User-agent: OAI-SearchBot` → `Allow: /` in a live host's `/robots.txt` is the fingerprint that it runs the fixed artifact (pip metadata is invisible from outside); `test_robots_artifact_fingerprint` now asserts it locally so a vendored regression fails CI, not production. 2.3.1 was assigned during development and never published. - **`dash-clerk-auth` 0.9.0 → 0.9.1** (vendored, built from the Dash-Clerk-Auth-Hook working tree). 0.9.0 ships a bug hitting every Clerk satellite forked from this template: clerk-js v5 auto-instantiates from the script tag's `data-*` attributes and reads the *instance* domain, so on a satellite the user button never mounts (dead avatar) while server-side session verification keeps working. 0.9.1 emits `data-clerk-domain="<satellite_domain>"` on the tag when `is_satellite=True`. This app runs no Clerk by design — the bump is for the template's sake. `lib/auth.py`'s fixup #1 guards on the attribute's absence, so it degrades to a no-op under 0.9.1 and stays for forks still on 0.9.0. - **`lib/hub_client.py` aligned with the hub's real contract.** Two functions predated the hub going live. `current_key()` now sends `{"token": <Clerk session token>, "app": ...}` — the hub 401s any caller-asserted identity (`user_id` in the payload is the forgery path) and verifies the token against Clerk's JWKS, minting at `scope=auth`, never admin. Call it on copy-button click, never on page render; `None` degrades to copying the plain URL. `hub_tiers()` is no longer a stub: signed POST `/api/page-tiers` `{"app": ...}` → `{"tiers": {path: tier}, "ttl": s}`, cached for the returned TTL with failures cached 60s — so a down hub costs one timeout per window, not one per request, and resolves to the local tier, which the ceiling rule guarantees can never loosen anything. `verify()` already matched the hub and is untouched. ### Added — AI/LLM surfaces (dash-improve-my-llms 2.2.0) - **`lib/network_directory.py`** — the peer/affiliated/external directory, defined once here and copied verbatim into every satellite. Publishes `<link rel="related">` tags, a `## Network` section in `/llms.txt`, and followed links in the prerendered body, so an agent landing on one satellite can enumerate the rest. Filters the app's own URL out of `peers`. - **Wordmark** — `"2"` + morse(`plot`) + `"ai"`, drawn as columns of dots and dashes in the header of the rendered `llms.txt` view. No period glyph: the morse block already separates the halves, and a literal `.` beside it reads as punctuation dropped into a graphic. The renderer turns a suffix ending in `i` into an upward flourish, so `"ai"` draws as `a` plus that mark, with the real domain in `label` for screen readers and the SVG `<title>`. It lives in the shared module rather than per-app, which is what keeps one mark across the network instead of twelve near-identical ones. - **Page `llms.txt` documents are no longer dead ends.** Each now opens with the site index, the network index one level up the hub chain (`2plot.dev`, correct for a `*.2plot.dev` subdomain), and the sitemap. These documents are usually read in isolation — pasted into a chat, handed to an agent — and an agent fetches a URL rather than crawling from one, so previously its exploration simply stopped there. - **The same URL content-negotiates.** Agents, crawlers and curl get the Markdown byte for byte; browsers get it rendered behind a header carrying the network identity. `?raw=1` and `?format=html` override, both variants send `Vary: Accept`, and the rendered view is `noindex` so it never competes with the page it documents. Verified identical on Flask, FastAPI and Quart. - **`docs/networks/networks.md`** — the guide for satellite authors: the three tiers, why per-host SEO can't express any of this, the wordmark and bulletin conventions, the one-URL-two-audiences contract, and the verification commands. - **Network bulletin left deliberately unwired.** `configure_bulletin()` sits commented next to `add_llms_routes` with a pointer to the contract. `2plot.dev` does not serve `/api/network/bulletin` yet, and pointing at a dead endpoint gains nothing: the client degrades silently and the header renders fine without it — the "Tips for getting started" and "What's new" panels use the package's built-in defaults, which a bulletin only overrides. ### Added — Clerk authentication and llms.txt access control Opt-in, and off in a default clone. This is the template every `*.2plot.dev` subdomain is forked from, so the goal was a pattern good enough to copy rather than a one-off. Requires `dash-improve-my-llms` 2.3.0 (`configure_access`, `configure_viewer_identity`) and the vendored `dash-clerk-auth` 0.9.0, which is deliberately **not** on the active requirements line — a default install should not pull in an auth stack the site does not use. - **`lib/auth.py`** — adapted from `2plot_leaflet/lib/auth.py`, the implementation already sharing authenticated state across `2plot.ai` → `2plot.dev` → `leaflet.2plot.dev` in production. Keeps both satellite fixups for `dash-clerk-auth` 0.9.0 (clerk-js reads `domain` as a *constructor* option from `data-clerk-domain`, and a satellite must `redirectToSignIn()` rather than open a modal that 403s), the `pk_live` auto-enable so production cannot silently boot in primary mode, `DISABLE_CLERK=1`, and call-time env reads. Changed for the template: the satellite domain derives from `APP_BASE_URL`, which every deployment must set anyway — one variable rather than two, and one fewer way to announce another site's domain to Clerk. - **`lib/page_tiers.py`** — `public < auth < admin < hidden`, declared in markdown frontmatter (`tier: admin`) because this template is already frontmatter-driven and marking one page should not require a control board. Two rules: everything except `hidden` falls open when Clerk is unavailable (documentation must not brick over a missing credential), and `effective_tier = more_restrictive(local, hub)` so a satellite may restrict further but never loosen. - **`lib/hub_client.py`** — the client for the hub's `/api/agent-key/current` and `/api/agent-key/verify`. Authenticates the caller with the network's existing `CROSS_APP_WEBHOOK_SECRET` HMAC scheme, the one `lib/satellite_reporter` already uses: it authenticates *who is asking* and derives nothing, which is what keeps "satellites hold no key material" true while still keeping the verify endpoint from being an open key-guessing oracle. Verdicts cached on a SHA-256 fingerprint of the key rather than the key, because that cache is process memory a debugger or error reporter can dump. `allow` cached 900s, `deny` 60s — a brief hub outage must not gate readers who were fine a minute ago, while a revoked key should stop working promptly. - **`lib/access.py`** — the policy, and its ordering is the design: tier → **local Clerk session** → hub, only for `?key=`. A signed-in visitor resolves entirely on this host, so the hub being down gates nothing for them; only the agent path, which arrives with no cookie, needs the hub at all. Reversing it would couple every satellite's availability to one host for no benefit. Kept out of `run.py` so satellites inherit one file. - **`docs/authentication/`** — three layers, so a reader stops at the one they need: the default (nothing to do), a standalone site with its own Clerk, and joining or running a network. Names the two traps: the Clerk token's `iat` is the token's age, not the sign-in's, so wiring it renders a clock that resets every minute; and identity must never travel in the bulletin, which is TTL-cached and shared across every satellite. - **`handoff/`** — kickoff prompts for the two repos this unblocks: an addendum pairing with the `pip-docs+` hub brief, carrying the request shapes and cache TTLs the client already sends, and a per-subdomain port guide. - **`tests/test_access.py`** — 17 tests against a fake hub. The two that justify the design: signed-in browser with the hub unreachable still resolves to `allow`, and a valid key with the hub down degrades to `gated` rather than 500 or prose. One asserts the *ordering* rather than the outcome — a signed-in reader must trigger zero hub calls, since "allowed" could otherwise come from a hub that happened to agree. **Inert until a tier says otherwise.** With the wiring in place, no Clerk keys, and every page public, all 43 surfaces are byte-identical to the build before any of it existed — measured, with a control run to strip out the per-request ids Dash puts in page HTML. ### Changed — dash-improve-my-llms 2.2.0 → 2.3.0 Vendored, as before; 2.3.0 is additive and opt-in. Verified as a no-op on the surfaces that matter: every Markdown document, the root index, `sitemap.xml`, `robots.txt` and the crawler HTML are byte-identical. The HTML viewer variants grow by 192 bytes each — three CSS rules for the identity block that ship whether or not identity is configured. Behaviourally a no-op; not literally byte-identical everywhere, which is worth stating precisely since this baseline is what a later regression gets attributed to. ### Added — CI/CD and tests - **`.github/workflows/ci.yml`** — flake8 (blocking), then the full test suite across a matrix of Python version × backend × Dash version: Flask, FastAPI and Quart on Python 3.12, Python 3.11 and 3.13 on Flask, and the bottom of the `~=4.4.1` range pinned explicitly on Flask and FastAPI so a 4.4.0-only regression cannot hide behind pip resolving to 4.4.1. Asserts the resolved Dash and `dash-improve-my-llms` versions before running anything, boots the app under gunicorn (a page can render under a test client and still fail under a real WSGI worker), and builds and probes the Docker image. - **`.github/workflows/cd.yml`** — runs CI, POSTs the `RENDER_DEPLOY_HOOK_URL` secret, waits for the new instance to be *sustainably* healthy (Render swaps instances rather than restarting in place, so a single 200 from `/healthz` proves nothing), then verifies the live site. Skips the deploy step when the secret is absent instead of failing, so a fork isn't red on day one. - **`tests/`** — a pytest suite that boots `run.py` itself rather than a test app. `conftest.py` normalises the three backends' test clients behind one synchronous `.get()`, including driving Quart's async client from a fixture-owned event loop. Covers page registration and reachability, stub bodies, rendered prose, canonical tags, sitemap/robots/llms.txt, content negotiation in both directions, the navigation block, the banner and its panels, the network directory and wordmark, docs frontmatter and directive targets, heading anchors, and the `BASE_URL` guard. - **`scripts/smoke_live.py`** — post-deploy checks against a live satellite, standard library only. Covers the failures that are silent in production: a canonical on the wrong host, a page serving the JavaScript stub, viewer chrome leaking into an agent's Markdown, a missing `Vary: Accept`, a missing network directory, and dead peer `llms.txt` links. Run in CD and by hand (`python scripts/smoke_live.py https://emojimart.2plot.dev`), and itself tested against the in-process app so a typo can't turn every live check into a silent pass. - **`scripts/dev.sh`** — starts the development server with *this* project's interpreter, resolved from the script's own location rather than from an IDE setting or `PATH`. - **`scripts/audit_links.py`** — walks every page's `llms.txt`, extracts every link, resolves internal paths in-process and checks the rest over the network. A dead link in an `llms.txt` is worse than one on a page: the agent holding that document has no navigation to fall back on and no way to tell a typo from a host that is down. Classified rather than lumped together, because the classes want different responses: `internal` is a real defect, `self-host` is correct once deployed, `network` is a peer awaiting the rollout, `unpushed` is a file that exists locally and 404s only until the branch is pushed, and `external` is someone else's problem to route around. Code spans and fenced blocks are skipped — a URL inside backticks renders as `<code>`, not `<a>` — and a transport failure is retried once, because an audit that cries wolf gets ignored. - **`LICENSE`** — the MIT text the README badge, `pages/home.md` and the Schema.org block have all claimed since 0.1.0 without the file ever existing. - **`render.yaml`** — Render Blueprint for `boilerplate.2plot.dev`: gunicorn, `/healthz` health check, custom domain, and a persistent disk for the analytics ledger (on an ephemeral filesystem a mid-day deploy wipes it and the next hourly report overwrites the day's real total). - **`.flake8`**, **`pytest.ini`**. ### Added — Network analytics reporting to 2plot.ai - **`lib/satellite_reporter.py`** — hourly signed rollup POSTed to `https://2plot.ai/api/satellite/traffic`, so a deployed docs site shows up on the hub's owner-only `/traffic` dashboard. HMAC-SHA256 over `"{timestamp}." + body` with `CROSS_APP_WEBHOOK_SECRET`, matching the network's existing webhook scheme. Off by default: no secret, no reporting. Re-posts yesterday during the first hours of a new day so the final hits of a day aren't left out, and uses a lease file so only one web worker reports per interval instead of every worker racing. `python -m lib.satellite_reporter --dry-run` prints the payload without sending it. - **`lib/traffic_rollup.py`** — derives the reported numbers (`human_hits`, `bot_hits`, `visitors`, `sessions`, `median_session_s`, top pages, countries) using the hub's own definitions, so this app's figures are comparable with every other app on the chart. Infrastructure paths (`/healthz`, `/llms.txt`, `/robots.txt`, `/sitemap.xml`, assets, Dash internals) are excluded from the report but stay in the local ledger. - **`lib/health.py`** — `/healthz` on Flask and Quart, matching the endpoint the FastAPI build already declared. The hub's hourly sweep probes it for up/down + latency, which previously only worked on one of the three backends. - Quart now tracks visitors too; previously only Flask and FastAPI did. ### Changed — dependencies - **`dash-improve-my-llms` 2.0.0 → 2.2.0**, installed from `vendor/` until it is published to PyPI. App 1 of 4 in a staged rollout, first because every satellite documentation site is forked from this repo — a convention set here propagates, and so does a mistake. Page metadata now *merges* instead of assigning, so no later bookkeeping call can erase a page's prose; the prerender reaches every visitor rather than only recognised crawlers; and the Markdown renderer emits real anchors, tables, code fences and rules. Measured on this app: link counts in crawler bodies went from 3 per page to 3–11, code fences from 0 to 5–29 per page, and horizontal rules stopped rendering as literal `---` text. No page serves the crawler stub, before or after — this repo was never affected by the prose-erasure bug, having no bridge loop over `dash.page_registry`. - **Dash pinned to `~=4.4.1`** (was `>=4.4.0`). Verified matrix, from real apps on each backend with the failure reproduced on stock Dash: | Dash | Flask | FastAPI | Quart | |---|---|---|---| | 4.1.0 | ok | n/a — no pluggable backends | n/a | | 4.2.0 | ok | ok | ok | | 4.3.0 | ok | **broken — every non-root page 500s** | ok | | 4.4.0 | ok | ok | ok | | 4.4.1 | ok | ok | ok | 4.3.0 added an early-return path guard to the ASGI middleware that returns before `set_current_request`, while the page catch-all still calls `get_current_request()` — so it raises `RuntimeError: No active request in context`. The catch-all is byte-identical between 4.2.0 and 4.3.0; only the middleware changed. 4.4.0 set the context inside the catch-all as well, so a future middleware guard cannot reintroduce it: 4.4.x is structurally safer, not merely currently-passing. `~=4.4.1` lets patch releases flow without twenty pull requests while blocking 4.5.0, so a minor bump goes through the matrix deliberately. Pinned for the most constrained backend network-wide, **including Flask-only apps** — `DASH_BACKEND` is an env var and this is a shared template, so a Flask deployment becomes a FastAPI deployment with one env change and no code change. - **Dependency floors are enforced at startup, not advised.** A version below the floor stops the boot, names what would degrade, and prints `sys.executable` alongside the expected interpreter. `ALLOW_STALE_DEPS=1` opts out for anyone deliberately testing an older release. The Dash floor is fatal only on FastAPI, where 4.3.0 is an outage rather than a degradation. See *Fixed — environment and tooling* for why this is a hard failure. - **`network_directory.apply()` gates the `wordmark` argument** on the installed signature. During a staged rollout this module reaches satellites before the new package does, and Python raises `TypeError` on an unknown keyword — so passing it unconditionally would turn an older satellite's boot into a crash rather than a missing graphic. Same technique `run.py` uses for Dash's `enable_mcp`. ### Changed — hosts, branding and repo hygiene - **`BASE_URL` moved to `lib/constants.py`** and reads `APP_BASE_URL` from the environment, defaulting to `https://boilerplate.2plot.dev`. `require_owned_base_url()` refuses to boot in production when `APP_BASE_URL` is unset or points at a platform hostname (`*.onrender.com` and friends). This is the template's highest-consequence footgun: a fork that leaves the default in place emits the boilerplate's canonical URL on every one of its pages, which asks Google to deindex it, and nothing about the app looks broken while it happens. - **YouTube links now point at [@2plotai](https://www.youtube.com/@2plotai)**; `plotly.pro` is replaced by `2plot.ai` throughout, and the deployment host by `boilerplate.2plot.dev`. A test fails the build if a live link to `plotly.pro` reappears. - **`.claude/` is untracked and gitignored.** Local session workspace; noise in a template other people fork. - **Dockerfile** copies `vendor/` before the pip layer (the build fails otherwise while the package installs from an sdist), declares a `HEALTHCHECK` against `/healthz`, and no longer leaves apt lists in the image. ### Fixed — SEO and template - **Every page shipped two `<link rel="canonical">` tags.** `templates/index.html` hard-coded one pointing at the site root while the package injected the correct per-page one. A conflicting pair is treated as no signal at all, so the per-page canonicals were doing nothing. The template no longer sets one. - **Two advertised LLM endpoints were 404s.** `<meta name="llms-page-json">` and `llms-architecture` pointed at `/page.json` and `/architecture.txt`, both removed in dash-improve-my-llms 2.0. The `<noscript>` block linked to them too. - **The Open Graph image never existed.** Every share rendered a blank card against `assets/og-image.png`, a file not in the repo. Now points at a real asset with its actual declared dimensions. - **`apple-touch-icon.png` and `site.webmanifest` 404'd on every page load** — both `<link>`ed but neither shipped. Commented out with instructions. - **`piratesbagain.com`** in the navbar (missing `r`) — a dead outbound link on every page. - Placeholder metadata left in the template: `"Your Organization Name"`, `"Your Name or Organization"`, `yourdomain.com`, and a `price` of `"29_000_000"` in the SoftwareApplication schema (not a valid number, and the project is MIT-licensed). ### Fixed — every page shipped the same hard-coded title `templates/index.html` hard-coded a `<title>` and contained no `{%title%}` placeholder anywhere, so the per-page titles `pages/markdown.py` registers were discarded and every page's title depended entirely on `dash-improve-my-llms` rewriting that one element. `LLMSConfig(prerender=False)` — the documented one-argument rollback — silently reverted every page on every satellite to one identical string. Now `<title>{%title%}`, with `app.title` set from a new `constants.APP_TITLE`. Without that second half the placeholder resolves to Dash's default, the bare string `"Dash"`, which is worse than what it replaced. **The trap, for anyone editing that block.** The package finds the element with `re.compile(r".*?", DOTALL | IGNORECASE)` and rewrites the first match: - Delete the element and no closing tag remains to anchor on — nothing is rewritten and no page has a title at all. - Spell the tag name in angle brackets inside a nearby *comment* and the match starts there instead, running to the next closing tag and replacing every line in between. The comment, and any markup after it, vanishes from the served page. With rewriting on it still looks correct, so the damage is only visible in the served bytes. The comment above the element used to contain a literal `` for exactly this reason, and the first attempt at this fix reintroduced it *while explaining it*. The block now describes the tags in words, and three tests pin it: the placeholder is present, the title regex matches nothing but the element itself, and no comment spells the tag in angle brackets. A fourth asserts every page serves a distinct title. ### Fixed — dead links in the llms.txt documents Found by `scripts/audit_links.py` across all 10 documents and 102 links. - **The MIT `LICENSE` file did not exist.** `pages/home.md` and the README both linked to it, and the Schema.org block declared the licence — so the one link a reader follows to check the terms was the one that 404'd. Added. - **The development-server port was wrong.** `pages/home.md` said `http://localhost:8553`; `run.py` binds **8559**. The Docker instruction (8550) was right for the container but rendered as a live link that 404s for anyone not running the image — both are now code spans, so they read as instructions rather than as something to click. - **The `SKILLS.md` link pointed at the wrong path** — `dash-improve-my-llms/blob/main/SKILLS.md`, but the file lives under `docs/`. Fixed to `blob/main/docs/SKILLS.md`. ### Fixed — Markdown rendering - **A heading containing inline code crashed the site at startup.** markdown2dash's renderer does `create_heading_id(text[0])`, and when the first inline token is formatted, `text[0]` is a component rather than a string — `AttributeError` at import, taking every page down. Fixed in `lib/directives/headings.py`. - **TOC anchors pointed at ids that didn't exist.** Even when it didn't crash, the renderer slugged only the *first* inline token (`## Wiring **it** up` → `id="wiring"`) while the `toc` directive slugged the raw markdown (`wiring-**it**-up`). Both now use one `slugify`, so the link and its target agree. Plain headings slug exactly as before, so no existing anchor moved. ### Fixed — MCP wiring - **The MCP server was never enabled.** `run.py` did `from dash import mcp_enabled`, but the symbol lives in `dash.mcp` — the import always raised, and the app printed "MCP not available in dash 4.4.1 (needs >=4.3)" while running 4.4.1. `mcp_enabled` is also the decorator for marking a *function* as an MCP tool, not a server switch. The server is started from Dash's constructor, so `enable_mcp=` / `mcp_path=` is now passed there, and it works on all three backends rather than only FastAPI. Passed as `**kwargs` so naming a 4.3+ keyword can't break the boot on an older Dash. ### Fixed — environment and tooling - **The app booted silently against another project's virtualenv.** An IDE run configuration pointing elsewhere started this app against whatever versions that environment held — on `dash-improve-my-llms` 2.0.0 there is no `llms_viewer.py` at all, so `/<page>/llms.txt` served plain Markdown to every visitor and nothing in the log said why. It cost a debugging session across two repositories, chasing a stale process and a browser cache that were both innocent, and survived a server restart and an incognito window because neither was the variable. Made worse by this repo's own `enable_mcp` fix, which removed the `TypeError` that had been failing loudly on the wrong interpreter — trading a crash for a plausible wrong answer. Warnings were tried first and were not enough: they scroll past above a wall of page-loading output while the app keeps serving. The floors are now fatal (see *Changed — dependencies*), and `scripts/dev.sh` removes the choice of interpreter entirely. A test asserts the same floor, so `pytest` in the wrong environment reports the cause instead of thirty downstream symptoms. - **CI installed a tarball path that no longer existed.** `ci.yml` hardcoded the vendored filename for the FastAPI and Quart legs, so a version bump broke exactly two of the matrix entries. It now globs `vendor/`. - **Header lookups in the test client were case-sensitive.** Werkzeug returns `Content-Type`, httpx returns `content-type`, so the content-negotiation assertions passed on Flask and failed on FastAPI and Quart — reading like a backend bug when the served headers were identical and correct. - **A peer serving its SPA shell counted as a live document.** The peer check asserted only `status == 200`, but a Dash app answers its catch-all with the app shell for *any* unmatched path — `2plot.dev/api/this-endpoint-cannot-exist` returns `200 text/html`, as does `/api/network/bulletin`, which does not exist. A status-only check therefore passes against every host in the network whether or not it publishes anything. `smoke_live.py` now rejects an HTML body for a document URL, and the same reasoning applies to the network-wide check in `ROLLOUT.md`. - **`smoke_live.py` extracted malformed peer URLs.** Its pattern stopped only at whitespace and `)`, and the 2.2.0 navigation block writes links as `[https://host/llms.txt](https://host/llms.txt)` — so it produced `https://2plot.dev](https://2plot.dev/llms.txt`, which would 404 in CD and fail a perfectly good deploy. Invisible locally, because the test shim answers 200 for off-host URLs. - **Viewer-chrome detection keyed on a bare class name.** `docs/networks` legitimately *documents* `dv-banner`, so a substring check failed on the page's own prose. Both the suite and `smoke_live.py` now match rendered markup (`<div class="dv-banner"`), which a Markdown document can never contain — otherwise the check quietly teaches people to stop documenting the viewer. ### Fixed — Analytics accuracy - **AI-search crawlers were not being counted.** The visitor hook was registered after `add_llms_routes`, and the package's bot middleware short-circuits ClaudeBot / ChatGPT-User / PerplexityBot with its own response — so those requests never reached the tracker. The hook is now registered first on Flask/Quart (and last on FastAPI, where Starlette runs the most recently added middleware outermost). - **Every visitor looked like one visitor behind a proxy.** The tracker used `remote_addr`, which on Render/Cloudflare is the proxy. It now reads `CF-Connecting-IP`, `True-Client-IP`, `X-Real-IP` and `X-Forwarded-For` first, and takes the country from Cloudflare's `CF-IPCountry` header when present — free, instant and accurate. - **Concurrent workers overwrote each other's hits.** The ledger was read, modified and rewritten with no lock; under four workers most hits were lost. Writes now take an `flock` and land via an atomic replace. - **Geolocation no longer blocks page views.** The ip-api.com lookup ran inline with a 2s timeout on the first hit from each new IP. It now runs in a bounded background thread and is backfilled into the buffered hit before it is written, so the country is still recorded. Disable with `ANALYTICS_GEO_LOOKUP=0`. - **The ledger is bounded and no longer rewritten on every request.** Hits are buffered (10 hits / 30s) and pruned to `ANALYTICS_RETENTION_DAYS` (45) and `ANALYTICS_MAX_VISITS` (20000); the hub holds the durable history. - The ledger path is now absolute (`TRAFFIC_ANALYTICS_FILE`, else repo root) — a relative default wrote a different file depending on the working directory. - Tablets are no longer counted as mobile (iPads and most Android tablets send a mobile token too, and the mobile test ran first). ## [1.0.0] - 2026-06-14 First stable release. The boilerplate moves to **Dash 4.x** with pluggable backends and **dash-improve-my-llms 2.0**, and retires the experimental TOON format entirely. This is a significant architectural release — see the migration notes at the end of this section. > **Versioning note:** the `0.5.0`–`0.8.0` entries below were the December 2025 > TOON line. That work has been removed (see "Removed" below) and the project > resumes a single, monotonic version line at `1.0.0`. A short-lived second > `0.5.0` (the May 2026 dash-improve-my-llms 2.0 preview) has been folded into > this entry. ### Added — Pluggable backends (Flask / FastAPI / Quart) - **`lib/backend.py`** — single source of truth for backend selection. Reads the `DASH_BACKEND` environment variable (`flask` | `fastapi` | `quart`), falls back to `flask`, and exposes `BackendInfo` (label, color, icon, async flag) so UI components stay in sync with the running backend. - **`run.py`** constructs `Dash(backend=resolve_backend(), ...)` and attaches `app._backend_info` for layout components. - **`components/backend_badge.py`** — a navbar/header badge that shows which backend the site is currently running on. - **`lib/asgi_middleware.py`** and **`lib/asgi_routes.py`** — ASGI middleware and showcase routes (`/healthz`, `/api/backend`, `/api/pages`) that light up on the FastAPI/Quart backends. - New documentation sections: - **Pluggable Backends** (`docs/backends/`) — run the site on any of the three backends with one env var. - **Backend Deep Dive** (`docs/backend-comparison/`) — architecture, strengths/weaknesses, deployment, and best practices for each backend. - **FastAPI Showcase** (`docs/fastapi-showcase/`) — OpenAPI docs, a native JSON API, ASGI middleware, async demo, endpoint explorer, and a stress test, showing what the ASGI backends unlock. ### Added — AI/LLM integration via dash-improve-my-llms 2.0 - **`LLMS_DOC` pattern.** Pages expose a module-level prose string (or call `register_page_metadata(path, llms_doc=...)`); the package serves it verbatim at `/<page>/llms.txt` under whichever backend is active. - `pages/markdown.py` registers the expanded markdown body (with `.. source::` directives inlined) for every markdown-driven page. - `pages/home.py` exports `LLMS_DOC = content` for the root prose. - **Multi-backend AI/LLM surfaces.** `add_llms_routes(app)` auto-detects the backend and serves `/llms.txt`, `/<page>/llms.txt`, `/sitemap.xml`, and `/robots.txt` under Flask, FastAPI, and Quart alike — no `if IS_FLASK:` gate. - **MCP resource bridge.** Each page's prose registers as a `dash.mcp` resource on Dash 4.3+ (a silent no-op on older Dash). ### Changed - **Upgraded Dash 3.2.0 → 4.2.0** and **Dash Mantine Components 2.4.0 → 2.7.0** (Mantine 8.3.6). React 18.2.0. - **`docs/ai-integration/ai-integration.md`** fully rewritten for the 2.0 surface (LLMS_DOC, multi-backend, MCP bridge). - **`requirements.txt`** now pins `dash>=4.1.0`, `dash-mantine-components>=2.7.0`, and `dash-improve-my-llms[flask]>=2.0.0`, with commented `[fastapi]`, `[quart]`, and `[all]` extras plus `uvicorn` for ASGI deployment. - **`docs/example/example.md`** "Highlighting Important Elements" section rewritten around the `LLMS_DOC` pattern. - **`components/header.py`**, **`components/appshell.py`**, and **`components/navbar.py`** updated for the new backend badge and navigation (TOON Format and Handoff entries removed). - **`lib/directives/llms_copy.py`** / **`assets/llms_copy.js`** updated for the 2.0 `/<page>/llms.txt` routing. - `APP_VERSION` and `package.json` bumped to `1.0.0`. ### Removed - **The entire TOON format system** — `lib/toon_generator.py` (~1100 lines), the `docs/toon-format/` page, the TOON Analytics Dashboard (`docs/data-visualization/toon_dashboard.py`), and all `/llms.toon` routes. `dash-improve-my-llms` 2.0 removed TOON from its public API (`TOONConfig`, `toon_encode`, `generate_*_toon` no longer exist). - **`/page.json` and `/<page>/page.json`** routes — dropped in dash-improve-my-llms 2.0; Dash 4.3 MCP exposes layouts as resources natively. - **`/architecture.txt`** — likewise superseded by MCP. - **`mark_important()`** and **`mark_component_hidden()`** — now deprecated no-ops in 2.0. Write the emphasis directly into a page's `LLMS_DOC` markdown. - **`LLMS_INTEGRATION.md`** and the `docs/handoff/` doc (the FastAPI port plan that became 2.0) — superseded by the in-app AI Integration page. ### Migration notes (from any 0.x) 1. **Backend:** the site defaults to Flask, so no change is required. To run on FastAPI or Quart, install the matching extra (`pip install "dash[fastapi]"`) and set `DASH_BACKEND=fastapi`. 2. **AI/LLM prose:** give each page module an `LLMS_DOC = """..."""` string at module scope (or `register_page_metadata(path, llms_doc=...)` when the prose is computed). The startup `UserWarning` from 2.0 names every page still missing prose. 3. **dash-improve-my-llms extra:** pick `[flask]`, `[fastapi]`, `[quart]`, or `[all]` in `requirements.txt`. 4. **Removed APIs:** replace any `mark_important()` / `mark_component_hidden()` calls (now no-ops) with `LLMS_DOC` content, and remove references to TOON, `/page.json`, and `/architecture.txt`. --- ## [0.8.0] - 2025-12-14 ### Added - **TOON v3.3 Format Enhancements** - Major comprehension improvements from ~75-80% to ~95%+ - **New Dataclasses**: - `CodeTip` - Short instructional code snippets with context - `BestPractice` - Numbered best practices with multi-line code examples - `Pattern` - Architectural patterns with implementation code - `Resource` - External resource links with full URLs - **New Extraction Functions**: - `extract_code_tips()` - Finds short code snippets (2-15 lines) with headings - `extract_best_practices()` - Extracts numbered practices from "Best Practices" sections - `extract_patterns()` - Captures pattern implementations from "Common Patterns" sections - `extract_resources()` - Extracts markdown links with full URLs preserved - **New TOON Sections**: - `tips[N]{context,lang,code}:` - Compact code tips with one-line previews - `bestPractices[N]:` - Full multi-line code snippets for each practice - `patterns[N]:` - Pattern descriptions with implementation code blocks - `resources[N]{name,url}:` - External links without URL truncation ### Changed - **Updated TOON format version from toon/3.2 to toon/3.3** - **Enhanced summary line** to include tips, best practices, patterns, and resources counts - **Improved content deduplication** - Tips exclude Best Practices and Patterns sections to avoid duplicate code ### Fixed - **Code block detection in section boundaries** - Headings inside code blocks (like `## My Visualization` in markdown examples) were incorrectly detected as section boundaries - Added code block range detection using `code_block_ranges` list - Added `is_in_code_block()` helper to filter out false headings - Applied fix to `extract_code_tips()`, `extract_best_practices()`, and `extract_patterns()` - **`re.escape()` issue** - `re.escape("Best Practices")` was escaping spaces incorrectly - Changed to custom escaping that only escapes regex special chars but preserves spaces ### Technical Details - Updated `lib/toon_generator.py` (~1100 lines after updates) - Test results for Data Visualization page: - 6 tips (properly deduplicated) - 5 best practices (all with full multi-line code) - 3 patterns (all with implementation code) - 4 resources (with full URLs) - TOON size: 11,444 chars --- ## [0.7.0] - 2025-12-13 ### Added - **Custom Documentation-Aware TOON Generator** (`lib/toon_generator.py`) - Custom TOON route that processes raw markdown from `NAME_CONTENT_MAP` - Achieves **54.7% token reduction** vs llms.txt while preserving all content - Full directive awareness (exec, source, kwargs, toc, llms_copy) - Features: - Section extraction with hierarchical structure (h2-h6) - Directive parsing with option extraction - Source file embedding with smart code compression - Table and list preservation in compact format - Exec component detection with callback markers - Deduplication of code examples and directives - Smart code compression (`compress_code()`) that: - Preserves imports, function/class definitions - Keeps callback decorators and Input/Output patterns - Truncates long files with line count indicator - TOON v3.2 format with optimized output: - Compact section format: `[level] title` - Grouped directives by type - Inline table format with pipe separators - Key lists extraction for substantial bullet points ### Changed - **Custom `/<page>/llms.toon` route** in `run.py` - Overrides default dash-improve-my-llms TOON for markdown pages - Uses raw markdown from NAME_CONTENT_MAP instead of rendered components - Processes source directives to embed actual file content ### Fixed - **TOON content gap issue** - Previous TOON was only capturing 15-20% of documentation content - Root cause: dash-improve-my-llms extracts from rendered Dash components, losing directive context - Solution: Custom route processes raw markdown with full directive awareness - Previous TOON was 185% the size of llms.txt (27,669 chars vs 14,943 chars) - New TOON is 45.3% the size of llms.txt (6,965 chars vs 15,369 chars) ### Technical Details - New module: `lib/toon_generator.py` (698 lines) - `generate_documentation_toon()` - Main entry point - `build_documentation_toon()` - TOON string builder - `extract_sections()` - Hierarchical section parser - `extract_directives()` - Directive extractor with options - `process_source_directive()` - File content reader - `process_exec_directive()` - Component metadata extractor - `compress_code()` - Smart code compression - `compress_section_content()` - Content summarization - `extract_tables()` / `extract_lists()` - Structure extractors --- ## [0.6.0] - 2025-12-13 ### Added - **Enhanced TOON Format v3.1** - Lossless semantic compression with 40-50% token reduction - Application context with related pages and multi-page awareness - Page purpose explanations with human-readable descriptions - Component breakdown with type distribution - Human-readable callback descriptions - Synthesized page summaries - Link categorization (internal vs external) ### Changed - **Upgraded dash-improve-my-llms from v1.0.0 to v1.1.0** - Lossless semantic compression preserves all meaningful content - New content extraction: `extract_markdown_content()`, `parse_markdown_content()` - Smart compression: `compress_code_example()`, `compress_section_content()` - New helper functions: `_generate_page_summary()`, `_format_callback_description()` ### New TOONConfig Options - `preserve_code_examples=True` - Include code snippets from markdown - `preserve_headings=True` - Keep section structure - `preserve_markdown=True` - Extract dcc.Markdown content - `max_code_lines=30` - Max lines per code example - `max_sections=20` - Max sections to include - `max_content_items=100` - Increased from 20 ### Documentation - **Updated AI/LLM Integration Guide** with v1.1.0 TOON enhancements - Added design principle: lossless semantic compression - Updated token efficiency comparison table - Added 6 content gap examples (context, purpose, components, callbacks, summary, navigation) - Updated TOONConfig with new v1.1.0 options ### Improved - Better content preservation in TOON format - Optimal information density vs token reduction balance - Enhanced developer experience with richer TOON output --- ## [0.5.0] - 2025-12-13 ### Added - **TOON Format Support** - Token-Oriented Object Notation for 50-60% fewer tokens - New `/llms.toon` endpoint for token-optimized LLM documentation - New `/architecture.toon` endpoint for token-optimized architecture - New `/<page>/llms.toon` per-page TOON format endpoints - TOON provides tabular arrays and explicit length markers for LLM validation - Ideal for API calls, large apps, and cost-conscious deployments ### Changed - **Upgraded dash-improve-my-llms from v0.3.0 to v1.0.0** - Production-ready release with comprehensive test coverage (88 tests, 98% coverage) - New API exports: `TOONConfig`, `toon_encode`, `generate_llms_toon`, `generate_architecture_toon` - Zero-change migration: existing code works without modifications ### Documentation - **Updated AI/LLM Integration Guide** with comprehensive TOON format documentation - Added TOON format section with benefits comparison table - Added example comparison (markdown vs TOON token usage) - Added TOONConfig configuration examples - Added programmatic TOON generation examples - Updated available routes table with new TOON endpoints - Updated key functions reference with new TOON imports ### Improved - Better AI/LLM documentation organization - Enhanced developer experience with new format options - Cost optimization through token-efficient TOON format --- ## [0.4.0] - 2025-11-10 ### Added - **LLM Copy Button Directive** (`.. llms_copy::`) - New custom directive that adds a "Copy for llm 📋" button to documentation pages - Copies the page's `/llms.txt` URL to clipboard for easy AI assistant sharing - Users can paste the URL into ChatGPT, Claude, or other AI assistants for context-aware help - Features: - Automatic URL construction based on current page path - Visual feedback with "✓ Copied! ✓" confirmation - Fallback clipboard method for non-HTTPS contexts (HTTP development servers) - Works across all modern browsers - Tooltip: "Copy llms.txt URL for AI assistants" - Implementation: - Python directive: `lib/directives/llms_copy.py` - JavaScript handler: `assets/llms_copy.js` - Uses both modern Clipboard API and legacy `execCommand` fallback - Mutation observer for Dash-rendered content detection - Documentation updated in Custom Directives guide - Added to all 5 example documentation pages ## [0.3.0] - 2025-11-09 ### Added - Documentation System - **Comprehensive Getting Started Guide** (385+ lines) - Detailed directive options documentation (`:code: false`, `:defaultExpanded`, `:withExpandedButton`) - Interactive examples with best practices - File structure examples and patterns - **Custom Directives Guide** (476 lines) - Complete documentation for all 4 directives (toc, exec, source, kwargs) - 3 live Python examples (button, counter, form validation) - **Data Visualization Guide** (465+ lines) - 5 chart type examples with full implementations - Plotly template integration guide - Real-time updates and dashboard patterns - **Interactive Components Guide** (569 lines) - 6 callback pattern examples - State management, pattern matching, chained callbacks - Loading states demonstration - **AI/LLM Integration Guide** (577 lines) - Complete dash-improve-my-llms documentation - SEO optimization strategies - Bot management and privacy controls ### Added - Theme System - **DMC Figure Templates Integration** - All Plotly charts now use `dmc.add_figure_templates()` - Theme-aware callbacks for 6 chart examples - Charts dynamically update with light/dark theme toggle - Proper background rendering in both themes - **Code Block Theming** - Theme-aware CSS for markdown code blocks - Proper syntax highlighting in light and dark modes - Inline code and code block styling - **Comprehensive Theme Configuration** - Professional typography hierarchy (h1-h6) - Systematic 4px-based spacing scale - 5-level shadow system - Consistent border radius system - Global component defaults via theme.components - Softer black (#1a1b1e) for better contrast ### Added - UI/UX Enhancements - **Navigation Improvements** - Custom page ordering (Getting Started → Custom Directives → AI/LLM → Interactive → Visualization) - Better visual hierarchy - Organized documentation sections - **Typography System** - Inter font family across application - Optimized line heights (md: 1.55 for body text) - Proper font sizes (16px base) - Font smoothing and text rendering optimization - **Layout Refinements** - Better responsive breakpoints (md for navbar) - Improved spacing consistency - Enhanced mobile experience - Better heading spacing (1.5em top, 0.5em bottom) ### Added - Production Features - **SEO-Ready HTML Template** - Comprehensive meta tags with developer guidance - Open Graph and Twitter Card configuration - Structured data (Schema.org) for Organization and SoftwareApplication - Analytics integration (Google Analytics ready to enable) - Favicon configuration with multiple formats - Performance optimization (preconnect hints) - Search engine verification placeholders - Enhanced noscript fallback with styled content - 297 lines of documentation and configuration ### Improved - **15 Working Python Examples** - Button interactions, counters, form validation - 5 chart types (bar, line, scatter, realtime, dashboard) - Callback patterns and state management - All examples theme-aware and fully functional - **Directive System** - Fixed kwargs directive to parse component specifications (e.g., `dmc.Button`) - Better error handling and fallbacks - Support for directive options - **Code Quality** - Fixed JSON serialization error (removed lambda from theme styles) - Better import statements - Comprehensive inline comments - Fixed DMC 2.4.0 compatibility issues ### Changed - **Better Performance** - Optimized theme switching - Smooth transitions - Better font loading - **Documentation Organization** - Clear learning path - Progressive complexity - Better code examples ### Fixed - Import errors in example files (missing dmc, State imports) - DMC 2.4.0 compatibility (removed unsupported `type` prop from TextInput) - JSON serialization error in theme configuration - Heading ID generation with code blocks in markdown - Theme persistence and switching - Code block rendering in dark mode ## [0.2.0] - 2025-11-09 ### Changed - **BREAKING**: Migrated from Dash 2.5.0+ to Dash 3.2.0 - **BREAKING**: Migrated from dash-mantine-components 0.14.7 to 2.4.0 - **BREAKING**: Updated all Mantine packages from 7.14.1 to 8.3.6 - Updated Flask from 1.0.4+ to 3.1.2 - Updated Plotly from 5.0.0+ to 6.4.0 - Updated `app.run_server()` to `app.run()` (Dash 3.x standard) ### Removed - **BREAKING**: Removed deprecated package imports: - `dash-html-components` (now part of main `dash` package) - `dash-core-components` (now part of main `dash` package) - `dash_table` (now part of main `dash` package) ### Fixed - Replaced deprecated `NotificationProvider` with `NotificationContainer` - Fixed Mantine version mismatch between package.json and DMC version - Added node_modules to .gitignore ### Added - Added package-lock.json for reproducible npm builds - Comprehensive migration documentation (8 detailed guides) - Project analysis and assessment documentation - Persistent theme preference storage using localStorage - Browser color scheme preference detection on first visit - Smooth theme transitions without page flash - AI/LLM & SEO Integration (dash-improve-my-llms v0.3.0) - Automatic llms.txt, page.json, architecture.txt generation - SEO-optimized sitemap.xml with intelligent priority - Bot management (blocks AI training, allows AI search) - Structured data for better search indexing - Privacy controls for sensitive pages ### Improved - Better dependency management with cleaner requirements.txt - Improved code organization with inline comments - Enhanced theme management system - Better performance with latest Dash and DMC versions ## [0.1.0] - 2024-11-30 ### Added - Initial release of Dash Documentation Boilerplate - Markdown-driven documentation system - Support for light and dark themes - Responsive design for mobile and desktop - Docker deployment support - Interactive code examples with syntax highlighting - Custom markdown directives: - `toc` - Table of contents generation - `exec` - Executable Python code blocks - `source` - Source code display with syntax highlighting - `kwargs` - Component props documentation - AppShell layout with header, navbar, and responsive drawer - Search functionality for navigation - Theme toggle with icon indicators - Integration with dash-mantine-components (DMC) - Integration with python-frontmatter for metadata - Custom CSS styling system - Docker and docker-compose configuration ### Documentation - README with getting started guide - Project structure documentation - Example documentation pages --- ## Version History Summary | Version | Date | Dash | DMC | Mantine | Python | Features | |---------|------|------|-----|---------|--------|----------| | 1.0.0 | 2026-06-14 | 4.2.0 | 2.7.0 | 8.3.6 | 3.11+ | Pluggable backends (Flask/FastAPI/Quart), dash-improve-my-llms 2.0, TOON removed | | 0.8.0 | 2025-12-14 | 3.2.0 | 2.4.0 | 8.3.6 | 3.11+ | TOON v3.3, tips/best practices/patterns/resources extraction | | 0.7.0 | 2025-12-13 | 3.2.0 | 2.4.0 | 8.3.6 | 3.11+ | Custom TOON generator, documentation-aware TOON v3.2 | | 0.6.0 | 2025-12-13 | 3.2.0 | 2.4.0 | 8.3.6 | 3.11+ | Enhanced TOON v3.1, dash-improve-my-llms v1.1.0 | | 0.5.0 | 2025-12-13 | 3.2.0 | 2.4.0 | 8.3.6 | 3.11+ | TOON format, dash-improve-my-llms v1.0.0 | | 0.4.0 | 2025-11-10 | 3.2.0 | 2.4.0 | 8.3.6 | 3.11+ | LLM Copy Button directive | | 0.3.0 | 2025-11-09 | 3.2.0 | 2.4.0 | 8.3.6 | 3.11+ | Comprehensive docs, theme system, SEO | | 0.2.0 | 2025-11-09 | 3.2.0 | 2.4.0 | 8.3.6 | 3.11+ | Migration to Dash 3.x, DMC 2.4.0, AI/LLM | | 0.1.0 | 2024-11-30 | 2.5.0+ | 0.14.7 | 7.14.1 | 3.11+ | Initial release | --- ## Migration Guides ### Migrating to 1.0.0 from any 0.x This is the major release that moves the boilerplate to Dash 4.x. See the **Migration notes** under [1.0.0](#100---2026-06-14) for the full checklist. In short: 1. **Backend:** defaults to Flask — no change required. For FastAPI/Quart, `pip install "dash[fastapi]"` (or `[quart]`) and set `DASH_BACKEND=fastapi`. 2. **AI/LLM prose:** add an `LLMS_DOC` string to each page module (or call `register_page_metadata(path, llms_doc=...)`); the 2.0 startup warning lists pages still missing prose. 3. **dash-improve-my-llms extra:** pick `[flask]` / `[fastapi]` / `[quart]` / `[all]` in `requirements.txt`. 4. **Removed APIs:** drop any TOON usage (`TOONConfig`, `toon_encode`, `generate_*_toon`), `/page.json`, `/architecture.txt`, and the now-no-op `mark_important()` / `mark_component_hidden()` calls — move emphasis into `LLMS_DOC` instead. ### Migrating to 0.6.0 from 0.5.0 **Zero changes required!** The upgrade is fully backwards compatible. Key changes: 1. Update `dash-improve-my-llms` in requirements.txt to `>=1.1.0` 2. TOON output now includes richer, lossless semantic content automatically Optional new TOONConfig options: ```python from dash_improve_my_llms import TOONConfig app._toon_config = TOONConfig( # New in v1.1.0: preserve_code_examples=True, # Include code snippets preserve_headings=True, # Keep section structure preserve_markdown=True, # Extract dcc.Markdown content max_code_lines=30, # Max lines per code example max_sections=20, # Max sections to include max_content_items=100, # Increased from 20 ) ``` ### Migrating to 0.5.0 from 0.4.0 **Zero changes required!** The upgrade is fully backwards compatible. Key changes: 1. Update `dash-improve-my-llms` in requirements.txt to `>=1.0.0` 2. New TOON endpoints are automatically available: - `/llms.toon` - Token-optimized LLM docs - `/architecture.toon` - Token-optimized architecture - `/<page>/llms.toon` - Per-page TOON format Optional new features: ```python # Configure TOON output (optional) from dash_improve_my_llms import TOONConfig app._toon_config = TOONConfig( indent=2, delimiter=",", include_metadata=True ) # Programmatic TOON encoding (optional) from dash_improve_my_llms import toon_encode toon_string = toon_encode({"key": "value"}) ``` ### Migrating to 0.3.0 from 0.2.0 Minor updates, mostly additive. Key changes: 1. Documentation content significantly expanded 2. Chart examples now use DMC figure templates 3. Enhanced SEO features in index.html 4. Better theme integration across all components ### Migrating to 0.2.0 from 0.1.0 Major breaking changes. See migration documentation: - **Quick Start**: `MIGRATION_README.md` - **Detailed Guide**: `claude.md` - **Step-by-Step**: `MIGRATION_CHECKLIST.md` - **Code Changes**: `CODE_CHANGES_SUMMARY.md` Key changes to be aware of: 1. Update all imports from `dash_html_components` to `from dash import html` 2. Update all imports from `dash_core_components` to `from dash import dcc` 3. Replace `dmc.NotificationProvider()` with `dmc.NotificationContainer()` 4. Update custom components to use DMC 2.4.0 API 5. Check CSS for any Mantine 8 specific changes --- ## Support - **Issues**: [GitHub Issues](https://github.com/pip-install-python/Dash-Documentation-Boilerplate/issues) - **Discussions**: [GitHub Discussions](https://github.com/pip-install-python/Dash-Documentation-Boilerplate/discussions) - **Dash Community**: [Plotly Community Forum](https://community.plotly.com/) --- [unreleased]: https://github.com/pip-install-python/Dash-Documentation-Boilerplate/compare/v1.0.0...HEAD [1.0.0]: https://github.com/pip-install-python/Dash-Documentation-Boilerplate/compare/v0.8.0...v1.0.0 [0.8.0]: https://github.com/pip-install-python/Dash-Documentation-Boilerplate/compare/v0.7.0...v0.8.0 [0.7.0]: https://github.com/pip-install-python/Dash-Documentation-Boilerplate/compare/v0.6.0...v0.7.0 [0.6.0]: https://github.com/pip-install-python/Dash-Documentation-Boilerplate/compare/v0.5.0...v0.6.0 [0.5.0]: https://github.com/pip-install-python/Dash-Documentation-Boilerplate/compare/v0.4.0...v0.5.0 [0.4.0]: https://github.com/pip-install-python/Dash-Documentation-Boilerplate/compare/v0.3.0...v0.4.0 [0.3.0]: https://github.com/pip-install-python/Dash-Documentation-Boilerplate/compare/v0.2.0...v0.3.0 [0.2.0]: https://github.com/pip-install-python/Dash-Documentation-Boilerplate/compare/v0.1.0...v0.2.0 [0.1.0]: https://github.com/pip-install-python/Dash-Documentation-Boilerplate/releases/tag/v0.1.0 --- <!-- /getting-started — https://llms.2plot.dev/getting-started/llms.txt --> # Getting Started > Install dash-improve-my-llms and mount the whole machine-facing surface of your Dash app in one line — llms.txt, robots.txt, sitemap.xml, a crawler prerender and an MCP bridge. --- ### Install Pick the extra that matches your Dash backend: ```bash pip install "dash-improve-my-llms[flask]" # the default Dash server pip install "dash-improve-my-llms[fastapi]" # Dash 4.4+ FastAPI backend pip install "dash-improve-my-llms[quart]" # Dash 4.4+ Quart backend pip install "dash-improve-my-llms[all]" # all three ``` The package itself has **zero required dependencies** beyond the standard library — the extras only pull in the web framework you already run. That is deliberate: this sits in the request path of every page you serve, so it brings nothing with it that could break. Requires **Dash 4.1+**. MCP resource registration needs **Dash 4.3+**. ### The one line ```python import dash from dash_improve_my_llms import add_llms_routes, LLMSConfig app = dash.Dash(__name__, use_pages=True) # ... register your pages ... add_llms_routes(app, LLMSConfig(warn_missing_llms_doc=True)) ``` That single call mounts every machine-facing surface: | Route | What it serves | |---|---| | `/llms.txt` | the index — every page, with descriptions and links | | `/llms-small.txt` | a compact briefing for a limited context window | | `/llms-full.txt` | the whole corpus in one fetch | | `/<page>/llms.txt` | one page's prose, content-negotiated | | `/robots.txt` | generated from the vendor registry | | `/sitemap.xml` | with priority inferred from your page tree | | `/favicon.ico` | claimed, so Dash's catch-all stops answering it with the app shell | …plus the **prerender** — the fix for the empty `<div>` — and one **MCP resource** per page. `warn_missing_llms_doc=True` is worth keeping on: it prints a line at boot for every page with no prose registered, which is the only way you find out before a crawler does. ### Where prose comes from Three ways, in the order you will reach for them. **1. A module-level `LLMS_DOC` string.** The package finds it automatically. ```python # pages/pricing.py import dash dash.register_page(__name__, path="/pricing") LLMS_DOC = """ # Pricing Three tiers. The free tier has no time limit and no card. """ layout = ... ``` No layout walking, no extraction heuristics, no second copy of your docs to keep in sync with the first. **2. `register_page_metadata()`** for anything that is not a Dash page — the home page, a document assembled at boot, a pseudo-path: ```python from dash_improve_my_llms import register_page_metadata register_page_metadata( path="/", name="My Site — what it is", description="One sentence, used as the site tagline.", llms_doc=open("pages/home.md").read(), schema_type="SoftwareApplication", ) ``` **3. A markdown loader.** If your pages come from `.md` files, feed the expanded body in at registration time — that is what this site does, which is why `/<page>/llms.txt` here serves the *rendered* prose rather than the source with directives still in it. ### Set your public origin ```python app._base_url = "https://your-site.example" ``` This drives `<link rel="canonical">` on every page, the absolute URLs in `sitemap.xml`, and the "this app" entry in `/llms.txt`. It is the single highest-consequence value in the integration: leave it pointing at someone else's host and you are telling Google your entire site is a duplicate of theirs. ### Identity for the crawler document Browsers get whatever your `index.html` declares. Crawlers get the prerendered document — and until you say otherwise, that document has your content signals and none of your identity. One call fixes every crawler surface at once: ```python from dash_improve_my_llms import configure_seo configure_seo( icons=["/assets/favicon/favicon.ico", "/assets/favicon/favicon-32x32.png"], social_image="https://cdn.example/card.png", social_image_alt="My Site", publisher="My Company", same_as=["https://github.com/me/my-package", "https://pypi.org/project/my-package/"], ) ``` See [Configuration](/reference/configuration) for the full signature. ### Check it worked ```bash # the index curl -s https://your-site.example/llms.txt | head # one page, as an agent sees it curl -s https://your-site.example/pricing/llms.txt # the same URL as a browser sees it curl -s -H 'Accept: text/html' https://your-site.example/pricing/llms.txt | head -3 # and the thing that actually matters — is your prose in the initial HTML? curl -s -A 'Googlebot/2.1' https://your-site.example/ | grep -c '<main>' ``` That last command returning `0` means a crawler is still getting a loading spinner. Everything else in this documentation is downstream of it. ### Where to go next - **[Configuration](/reference/configuration)** — every option on `LLMSConfig`, `RobotsConfig` and `configure_seo`. - **[Access & tiers](/reference/access)** — gating pages and their machine twins. - **[The geo guardrail](/reference/geo)** — 451 for whole geographies. - **[The operator panel](/reference/panel)** — read-only live policy. - **[What the crawler sees](/audiences/web-crawlers)** — run the handlers against a real page and watch the output change. --- *Source: /getting-started* --- <!-- /reference/access — https://llms.2plot.dev/reference/access/llms.txt --> # Access & tiers > Gate a page and its machine twin independently — the four verdicts, the two axes, the callable seam, and why the gate document is served instead of a 403. --- ### Two axes, not one Gating documentation has a shape most auth systems get wrong. There are **two** questions, and they have different answers: 1. **Who may read the interactive page?** — a browser question. 2. **Who may read its machine twin** (`/<page>/llms.txt`, the crawler document, the prerender)? — an agent question. Collapse them into one and you get a site that either leaks gated content to every crawler, or hides its documentation from the search engines that would have sent it customers. The package keeps them separate. ### The four verdicts `configure_access(check)` takes a callable that receives a path and returns one of: | Verdict | Meaning | What the machine surfaces get | |---|---|---| | `allow` | public | the prose | | `gated` | needs an account | the **gate document** — a real page saying what is behind the gate and how to get in | | `deny` | not for this reader | 404, unrevealing | | `priced` | payable | with metering **off**, degrades to `gated` | ```python from dash_improve_my_llms import configure_access def check(path: str) -> str: if path.startswith("/internal/"): return "deny" if path.startswith("/premium/"): return "gated" return "allow" configure_access(check) ``` The callable runs **per request**, so it can consult a session, a header, or a store. It must not raise: an exception is caught, logged once per path, and degrades to `gated` — the page's prose is withheld rather than published, which is the safe direction for a broken auth callback. ### Why `gated` serves a document instead of a 403 A 403 to an agent is a dead end. It cannot sign in, and the person who can is usually the one who pasted the URL. So a gated page's `llms.txt` serves a **gate document** — the page's title and description, a statement that it is gated, and the sign-in URL — rather than nothing. ```python configure_access( check, gate_doc=lambda path: f"# {path}\n\nSign in at https://example.com/login\n", link_suffix=lambda: "?key=" + current_agent_key(), ) ``` `link_suffix` is the person→agent handoff: it appends a portable key to the `llms.txt` URLs the site publishes, so a signed-in reader who copies a URL into a chat window does not lose their entitlements at the boundary. That behaviour is deliberate and worth stating plainly: **the gate document publishes the existence and shape of a page, never its content.** ### `deny` versus `hidden` ```python from dash_improve_my_llms import mark_hidden mark_hidden("/admin/control-board") ``` `mark_hidden` is the static version and it **wins over everything** — an application check cannot un-hide a page. It removes the path from `sitemap.xml`, 404s its `llms.txt`, skips the MCP resource, and 404s crawler requests on the page URL. It is deliberately **absent from `robots.txt`**. A `Disallow: /admin` line tells every crawler exactly where your admin surface is; the lesson is old and keeps being relearned. ### Viewer identity ```python from dash_improve_my_llms import configure_viewer_identity configure_viewer_identity(lambda: {"name": "Ada", "signed_in": True}) ``` A zero-argument callable returning the current viewer, used by the rendered `llms.txt` viewer to show who is signed in. Returns `None` for anonymous. ### The 402 seam, and why it ships dark `LLMSConfig(metering=False)` is the default, and with it a `priced` verdict resolves to `gated`. Nothing is published and nothing is charged. That is the whole safety property. A billing bug in a site's own `check()` cannot accidentally publish an offer document or bill anyone, because the priced lane does not exist until an operator turns it on deliberately: ```python add_llms_routes(app, LLMSConfig(metering=True)) configure_access(check, offer_doc=..., payment_headers=...) ``` With metering **on**, a priced page serves an offer document at `402` across every surface, stays listed in the index, and serves the offer at `200` with `noindex` to the crawler column — so a crawler is never shown different content than a payer would get, which is the anti-cloaking rule. **The package never computes a price and never holds a pay-to address.** Both belong to the application. A network bulletin carrying anything shaped like a payment address is refused whole rather than sanitized: a fetched address is a payment-redirection target. ### The rate contract ```python add_llms_routes(app, LLMSConfig(rate_limit_per_minute=30)) ``` Bot traffic over the ceiling on the **corpus routes** gets `429` with `Retry-After`, keyed on the client IP from the edge headers. Three exemptions, each for a reason: - **Humans are never limited.** The stampede this exists for is an agent failure mode. - **`/robots.txt` and `/sitemap.xml` are never limited.** RFC 9309 reads an unreadable `robots.txt` as "no rules at all", so limiting it deletes the very rules it announces. - **It fails open on any error.** Refusing to serve documentation is strictly worse than serving too much of it — this is the one place the package's fail-closed instinct is wrong. Per process: N gunicorn workers means N × the ceiling in aggregate. There is no shared counter, because a limiter that needed Redis would be a dependency in the request path. ### Tiers from the network If you run a family of sites behind a hub, the hub's bulletin can tighten a page's tier network-wide. It can only ever **tighten** — a hub can close a page a site left open, never open one a site chose to close. ### See it resolve The **[policy panel](/showcase/policy-panel)** has a simulator: pick a page, an audience and a country, and it walks the request through the real layer order in-process — geo, then assets, then vendor policy, then the rate contract, then the tier. --- *Source: /reference/access* --- <!-- /reference/configuration — https://llms.2plot.dev/reference/configuration/llms.txt --> # Configuration > Every option on LLMSConfig, RobotsConfig and configure_seo — what each one does, what it defaults to, and the ones with consequences that are not obvious from the name. --- ### `LLMSConfig` — the routes and the documents Passed to `add_llms_routes(app, config)`. Every option is optional; the defaults are what this site runs. ```python LLMSConfig( enabled=True, warn_missing_llms_doc=True, register_mcp_resources=True, prerender=True, llms_nav=True, llms_viewer=True, llms_tiers=True, llms_full_max_bytes=4_000_000, rate_limit_per_minute=None, metering=False, panel=False, panel_path="/llms-policy", panel_token=None, ) ``` | Option | Default | What it does | |---|---|---| | `enabled` | `True` | Master switch. `False` registers nothing | | `warn_missing_llms_doc` | `True` | Boot warning per page with no prose. Keep it on | | `register_mcp_resources` | `True` | One MCP resource per page (Dash 4.3+) | | `prerender` | `True` | The static-HTML injection. **The documented one-argument rollback** | | `llms_nav` | `True` | Sibling-page links in the corpus documents | | `llms_viewer` | `True` | The rendered view browsers get at an `llms.txt` URL | | `llms_tiers` | `True` | Serve `/llms-small.txt` and `/llms-full.txt` | | `llms_full_max_bytes` | `4000000` | Cap on the full corpus | | `rate_limit_per_minute` | `None` | W4's ceiling. `None` = no limiting | | `metering` | `False` | The 402 seam. Off, a `priced` verdict degrades to `gated` | | `panel` | `False` | Register the operator panel | | `panel_path` | `/llms-policy` | Where it lives | | `panel_token` | `None` | Beats the `DIMLL_PANEL_TOKEN` env var | #### The rendered viewer `llms_viewer=True` is what makes an `llms.txt` URL useful to a **person**. An agent asking for that URL gets Markdown; a browser gets the same document rendered, behind a small header carrying the site's brand chip and the network wordmark. If you style it, the header's outer element carries the class `dv-banner` and the wordmark carries `mk-wordmark`. Both are stable. Note what a bare class name is *not*: the string `dv-banner` appearing in a document is prose, not chrome — this paragraph is the proof. Anything checking whether viewer chrome leaked into an agent's copy has to key on the rendered element, not on the name, or documenting the viewer becomes impossible. (Which is why this page states the names in words and never writes the opening tag: doing so would make this very document read as chrome to a naive check.) The rendered view is `noindex` by design, so it never competes with the real page in search results, and both variants send `Vary: Accept` so a CDN cannot hand cached HTML to the next agent that asks. #### Three with non-obvious consequences **`prerender=False`** is the rollback if the injection ever fights your template. It is also the switch that stops per-page `<title>` rewriting — so your `index.html` must keep Dash's `{%title%}` placeholder, or every page reverts to one identical title and nothing looks broken. **`rate_limit_per_minute` is per process.** Under gunicorn with N workers the effective ceiling is N × this number. There is no shared counter, deliberately — a limiter that needed Redis would be a dependency in the request path. **`panel=True` with no token is safe.** The panel 404s for everyone until a token exists, and the token is read *per request*, so it can be rotated or revoked live. Registering it only when a token happens to be set at boot would mean turning it on costs a redeploy — which is when nobody does it. ### `RobotsConfig` — who may crawl Assigned to `app._robots_config`, and read by both `robots.txt` **and** the middleware. ```python RobotsConfig( block_ai_training=True, allow_ai_search=True, allow_traditional=True, crawl_delay=None, custom_rules=None, disallowed_paths=None, block_ai_training_docs=False, vendor_policy=None, default_unknown_ai="allow", ) ``` | Option | Default | What it does | |---|---|---| | `block_ai_training` | `True` | GPTBot, ClaudeBot, CCBot, Google-Extended, … | | `allow_ai_search` | `True` | Claude-User, ChatGPT-User, PerplexityBot, OAI-SearchBot | | `allow_traditional` | `True` | Googlebot, Bingbot, Slurp, DuckDuckBot | | `crawl_delay` | `None` | Seconds, rendered inside the `User-agent: *` group | | `custom_rules` | `None` | Raw lines appended verbatim | | `disallowed_paths` | `None` | Paths inside the `*` group | | `block_ai_training_docs` | `False` | Close the corpus to blocked training crawlers | | `vendor_policy` | `None` | Per vendor: `allow` / `block` / `meter`. Map **or callable** | | `default_unknown_ai` | `"allow"` | Posture for an AI crawler not in the registry | #### `block_ai_training=False` does not mean "balanced" With it `False` the training bucket is **not emitted at all**, which silently *allows* every training crawler. If you want a permissive posture, say so per vendor rather than by deleting the group. #### `block_ai_training_docs` is the one most people want to leave alone Off by default, and that default is load-bearing: your documentation exists to get the package used, and blocking a training crawler on your *pages* while leaving `/llms.txt` open is a coherent bargain. Turning this on closes the corpus too. #### `vendor_policy` takes a callable ```python app._robots_config = RobotsConfig( vendor_policy={"perplexitybot": "block", "gptbot": "meter"}, ) ``` Keys are **registry keys** (lowercase — `gptbot`, `claudebot`, `googlebot`), not display names. An unrecognised key is logged and ignored, so a typo is a policy that silently does nothing. Pass a **zero-argument callable** instead of a dict and it is evaluated on every request — that is the seam a writable control board wires a persisted store through: ```python app._robots_config = RobotsConfig(vendor_policy=my_store.vendor_policy) ``` `meter` renders as `Allow` in `robots.txt` and behaves as allow until the rate limiter consumes it — a `Disallow` would kill the funnel the meter exists for. Try every combination on the **[bot policy sandbox](/showcase/robots-sandbox)**. ### `configure_seo` — identity on the crawler document ```python configure_seo( icons=None, social_image="", social_image_alt="", social_image_width="", social_image_height="", twitter_site="", twitter_card="summary_large_image", publisher="", logo="", same_as=None, root_icons=True, ) ``` `icons` accepts plain paths or dicts with `href` / `sizes` / `rel`: ```python configure_seo(icons=[ "/assets/favicon/favicon.ico", {"href": "/assets/favicon/favicon-32x32.png", "sizes": "32x32"}, {"href": "/assets/favicon/apple-touch-icon.png", "rel": "apple-touch-icon", "sizes": "180x180"}, ]) ``` `root_icons=True` also claims `/favicon.ico` — Google's fallback — which Dash's page catch-all would otherwise answer with the app shell. `same_as` becomes JSON-LD `sameAs`. For a package's documentation site, list the GitHub repo and the PyPI project: three properties pointing at each other is the strongest available statement of which URL is a package's canonical docs home. **Declared image dimensions must match the file.** A size that disagrees is worse than declaring none, because the platform reserves that box and crops into it. ### Page-level helpers ```python from dash_improve_my_llms import mark_hidden, mark_important, register_page_metadata mark_hidden("/admin/control-board") ``` `mark_hidden` removes a path from `sitemap.xml`, 404s its `llms.txt`, skips the MCP resource, and returns 404 to crawler requests on the page URL. It is deliberately **not** added to `robots.txt`: a `Disallow` line publishes the path it is trying to protect. ```python register_page_metadata( path="/pricing", name="Pricing", description="Three tiers.", llms_doc="# Pricing\n\n...", title="My Site | Pricing", image_url="https://cdn.example/card.png", schema_type="TechArticle", lastmod="2026-08-22", ) ``` `lastmod` is emitted **verbatim** into the sitemap, and omitted entirely when absent. Set it when the content genuinely changes; never script it from file mtimes, which reset on every container build and re-invent the daily-lie sitemap. ### The network directory ```python from dash_improve_my_llms import register_network register_network( name="My Network", hub_url="https://hub.example", peers=[{"name": "sibling", "url": "https://sibling.example"}], ) ``` Emits `<link rel="related">` tags, a `## Network` section in `/llms.txt`, and followed links in the prerendered body — so an agent that lands on one site can enumerate the rest. ### Everything else - **[Access & tiers](/reference/access)** — `configure_access`, `configure_viewer_identity`, the four verdicts. - **[The geo guardrail](/reference/geo)** — `configure_geo`. - **[The operator panel](/reference/panel)** — the token gate and what it shows. --- *Source: /reference/configuration* --- <!-- /reference/geo — https://llms.2plot.dev/reference/geo/llms.txt --> # Geo guardrail > configure_geo — opt-in 451 on every surface for whole geographies, the header trust model, the fail-open postures, and the callable seam a control board writes through. --- ### What it is Opt-in denial of whole geographies. Every listed country receives **HTTP 451 Unavailable For Legal Reasons on every surface** — pages, client-side navigation, assets, the `llms.txt` family, `sitemap.xml`, `robots.txt`, root icon redirects — humans and bots alike. The application does not exist for that geography. Unconfigured is a **strict no-op**: with no `configure_geo()` call, or an empty denylist, every response is byte-identical to a build without the feature. ### And what it honestly is not **A compliance guardrail and a uniform-response layer. Not a security boundary.** The country of a request comes from an edge header, and an edge header is exactly as trustworthy as the edge in front of your origin. Behind Cloudflare, `CF-IPCountry` is set by Cloudflare and client-supplied copies are stripped — the header is reliable. A client that reaches your **origin directly** — no proxy, or a leaked origin hostname — can spoof or omit any header. There is no trusted-proxy validation here, deliberately: half of one would imply a promise the header model cannot keep. If the block matters adversarially, enforce it **at the edge as well** (a Cloudflare country WAF rule) and treat this as the layer that makes the origin's answer uniform across every surface — which an edge rule alone cannot do, because it does not know that `/_dash-update-component` carries page navigation or that `/llms-full.txt` is the corpus. ### Verify your host before trusting it Geo requires the host to be **edge-proxied** (or an app-side `resolver=`). A DNS-only host with no proxy in front resolves every request "unknown" and, under the default `unknown="allow"`, the feature ships **inert** — configured, tested, and blocking nobody. The live check is the [operator panel](/reference/panel)'s line: > this request resolved to: `DE` (via `cf-ipcountry`) If that says "unknown" for a request you *know* came through your edge, the header is not being forwarded. Fix that before trusting a denylist. ### Usage ```python from dash_improve_my_llms import configure_geo configure_geo(deny_countries=["RU", "CN", "IR"]) ``` Full signature: ```python configure_geo( deny_countries=["RU"], # or a zero-argument callable unknown="allow", # "allow" (default) | "deny" resolver=None, # (headers) -> "US" | None exempt_paths=("/healthz", "/health", "/livez", "/readyz"), body=None, # override the one-line 451 body policy_url="", # emitted as Link: rel="blocked-by" (RFC 7725) ) ``` ### `deny_countries` — the reloadable seam A **static sequence** is validated at config time — `ValueError` on anything that is not an ISO 3166-1 alpha-2 code. A **zero-argument callable** is evaluated on **every request**. This is the seam a writable control board wires a persisted store through: ```python # lib/policy_store.py import json, pathlib _STORE = pathlib.Path("/var/data/policy_overrides.json") def geo_deny(): try: return json.loads(_STORE.read_text()).get("geo_deny", []) except FileNotFoundError: return [] # run.py configure_geo(deny_countries=geo_deny) ``` A store edit takes effect on the **next request in every worker** — no restart, no redeploy. That is what dissolves the multi-worker problem: mutate module state and you change one worker; mutate a file every worker re-reads and you change all of them. This site's control board does exactly this. Here is the whole store it writes through — flock-guarded, validated on write, atomic on replace, and fail-open on every read failure: ```python # File: lib/policy_store.py """The writable policy store — this site's half of the 2.7.0 callable seam. `dash-improve-my-llms` 2.7.0 accepts a **zero-argument callable** wherever it accepts a static policy value, and evaluates it on **every request**: configure_geo(deny_countries=policy_store.geo_deny) RobotsConfig(vendor_policy=policy_store.vendor_policy) That callable is the entire integration surface. The package stays zero-dep and never learns what a disk is; this module owns persistence, validation and concurrency, and hands back a plain list or dict. Flip a country on the control board and the next request in every worker answers 451 — no restart, no redeploy, no signal, no shared cache server. Why a separate store from :mod:`lib.page_visibility` ---------------------------------------------------- The inherited board's store answers "who may READ this page" and is consulted by :mod:`lib.access` at render time. This one answers "who may reach this ORIGIN at all" and is consulted by the package's middleware before routing. Different blast radius, different validation rules, different failure posture — and one corrupt JSON file should not take out both. They share a directory and nothing else. The three hard-won behaviours, inherited from the template's store and kept deliberately: * **Cross-worker reconciliation, on the request itself.** gunicorn runs N workers; a board POST mutates one. Every read re-stats the file and re-parses only when ``(mtime_ns, size)`` moved. That is one ``os.stat`` per request — deliberately NOT the template's 1s throttle, because the geo seam's whole promise is "the NEXT request", and a throttle makes that "the next request, probably, within a second". A page-visibility toggle can afford to land a second late; a compliance block cannot. * **Both persistence guards.** Boot warns when ``POLICY_STORE_FILE`` is unset, and when it points under ``/var/`` at a directory that is not actually a mount (a bare ``mkdir`` behaves identically until the next deploy wipes it). * **Fail-open reads, validated writes.** A missing, unreadable, malformed or wrong-shaped store yields the empty policy and logs once — matching the contract docs/GEO.md states for a raising callable, because from the package's side that is exactly what this is. Writes validate first and refuse to persist garbage, so the fail-open path stays a bug-tolerance measure rather than a routine one. Concurrency: writes take an exclusive ``flock`` for the whole read-modify-write and land via ``os.replace``, so a reader never sees a torn file and two workers toggling different countries cannot lose one another's edit. """ from __future__ import annotations import errno import fcntl import json import logging import os import threading import time from pathlib import Path from typing import Any logger = logging.getLogger(__name__) _STORE_PATH = Path(os.environ.get("POLICY_STORE_FILE") or "policy_overrides.json") # The shape this module persists. Anything else in the file is preserved on # write but ignored on read, so a future key (2.8's bot x country matrix) # added by a newer worker survives an older worker's toggle. _GEO_DENY = "geo_deny" _GEO_UNKNOWN = "geo_unknown" _VENDOR_POLICY = "vendor_policy" VENDOR_ACTIONS = ("allow", "block", "meter") UNKNOWN_POSTURES = ("allow", "deny") _lock = threading.Lock() _cache: dict[str, Any] = {} _cache_stamp: tuple[int, int] | None = None _warned: set[str] = set() def path() -> Path: """Where this process reads and writes policy.""" return _STORE_PATH def _warn_once(key: str, message: str, *args) -> None: """One line per distinct failure, however many requests hit it. The seam runs inside every request. A store that goes unreadable would otherwise write a log line per request forever, which is how a degradation becomes an outage of its own. """ if key in _warned: return _warned.add(key) logger.warning(message, *args) # --------------------------------------------------------------------------- # Validation — the same ISO rules the package applies to a static list # --------------------------------------------------------------------------- def normalize_country(code: Any) -> str | None: """``"de"`` -> ``"DE"``; anything that is not two ASCII letters -> None. Mirrors the package's own ``_normalize_codes``. Validating here as well is not redundant: the package's static path raises ``ValueError`` at config time, but a CALLABLE that returns junk is treated as an empty denylist and fails OPEN. An invalid code that reaches the store would therefore not block anything and not say why. Refusing it at the write is the only place the operator finds out. """ if not isinstance(code, str): return None code = code.strip().upper() if len(code) != 2 or not code.isascii() or not code.isalpha(): return None # The package treats these as "unknown", never as a country; storing one # would create a denylist entry that can never match. if code in ("XX", "T1"): return None return code def _clean_deny(raw: Any) -> list[str]: if not isinstance(raw, (list, tuple)): return [] seen: list[str] = [] for item in raw: code = normalize_country(item) if code and code not in seen: seen.append(code) return sorted(seen) def known_vendors() -> dict[str, str]: """``{registry key: display name}`` from the package, or ``{}``. Best-effort by design: the registry is a 2.7.0 surface, and this module has to keep importing on the 2.6.1 floor requirements.txt still pins. """ try: from dash_improve_my_llms.vendors import VENDORS return {v.key: v.display for v in VENDORS} except Exception: return {} def normalize_vendor(name: Any) -> str | None: """``"ClaudeBot"`` / ``"claudebot"`` -> the registry KEY, or None. The package's ``vendor_policy`` map is keyed on the registry key, and an unrecognised key is logged and IGNORED — so a board that stored the display name would show an override that quietly does nothing. Same reasoning as :func:`normalize_country`: refuse at the write, because the write is the only place the operator finds out. """ if not isinstance(name, str) or not name.strip(): return None candidate = name.strip().lower() registry = known_vendors() if not registry: return candidate # pre-2.7.0: nothing to validate against if candidate in registry: return candidate # Accept a display name too — it is what the board's table shows. for key, display in registry.items(): if display.lower() == candidate: return key return None def _clean_vendor_policy(raw: Any) -> dict[str, str]: if not isinstance(raw, dict): return {} out: dict[str, str] = {} for name, action in raw.items(): key = normalize_vendor(name) if key is None: continue if not isinstance(action, str) or action.lower() not in VENDOR_ACTIONS: continue out[key] = action.lower() return out # --------------------------------------------------------------------------- # Reading — per request, cheap, and never raising # --------------------------------------------------------------------------- def _read() -> dict[str, Any]: """The current store contents, re-parsed only when the file moved. Never raises. Every failure path returns the empty policy, because this runs inside the request path of a package that promises to fail open. """ global _cache, _cache_stamp try: st = _STORE_PATH.stat() except OSError as exc: if exc.errno != errno.ENOENT: _warn_once("stat", "policy store %s unreadable (%s) — empty policy", _STORE_PATH, exc) # No file is the normal unconfigured state, not a failure: an app that # has never opened the board has no store, and must behave exactly # like an app with no board at all. if _cache_stamp is not None: with _lock: _cache, _cache_stamp = {}, None return {} stamp = (st.st_mtime_ns, st.st_size) if stamp == _cache_stamp: return _cache with _lock: if stamp == _cache_stamp: # another thread just reloaded return _cache try: with _STORE_PATH.open("r", encoding="utf-8") as fh: fcntl.flock(fh.fileno(), fcntl.LOCK_SH) try: loaded = json.load(fh) finally: fcntl.flock(fh.fileno(), fcntl.LOCK_UN) if not isinstance(loaded, dict): raise ValueError(f"top level is {type(loaded).__name__}, not an object") _cache = loaded _warned.discard("parse") except Exception as exc: _warn_once("parse", "policy store %s malformed (%s) — empty policy", _STORE_PATH, exc) _cache = {} _cache_stamp = stamp return _cache # --------------------------------------------------------------------------- # THE SEAM — the callables handed to the package # --------------------------------------------------------------------------- def geo_deny() -> list[str]: """``configure_geo(deny_countries=policy_store.geo_deny)``. Zero-argument, called by the package on every request, and contractually forbidden from raising or doing I/O beyond this one local stat+read. """ return _clean_deny(_read().get(_GEO_DENY)) def geo_unknown() -> str: """Posture for requests whose country will not resolve. NOT a callable seam — ``configure_geo(unknown=...)`` takes a plain string read once at config time. The board writes it, and run.py reads it at boot only, so the UI says so. """ value = _read().get(_GEO_UNKNOWN) return value if value in UNKNOWN_POSTURES else "allow" def vendor_policy() -> dict[str, str]: """``RobotsConfig(vendor_policy=policy_store.vendor_policy)``. Read per request by the middleware AND per render by robots.txt, which is what keeps the published promise and the enforced behaviour identical. """ return _clean_vendor_policy(_read().get(_VENDOR_POLICY)) # --------------------------------------------------------------------------- # Writing — validated, flocked, atomic # --------------------------------------------------------------------------- def _write(mutate) -> dict[str, Any]: """Read-modify-write under an exclusive lock, landing atomically. ``mutate`` receives the current document and edits it in place. The lock is held across the whole cycle so two workers toggling different countries cannot lose one another's edit; ``os.replace`` is what stops a concurrent READER seeing half a file. """ global _cache, _cache_stamp _STORE_PATH.parent.mkdir(parents=True, exist_ok=True) lock_path = _STORE_PATH.with_suffix(_STORE_PATH.suffix + ".lock") with _lock: with lock_path.open("a+") as lock_fh: fcntl.flock(lock_fh.fileno(), fcntl.LOCK_EX) try: try: doc = json.loads(_STORE_PATH.read_text(encoding="utf-8")) if not isinstance(doc, dict): doc = {} except (OSError, ValueError): doc = {} mutate(doc) tmp = _STORE_PATH.with_suffix(_STORE_PATH.suffix + ".tmp") tmp.write_text(json.dumps(doc, indent=2, sort_keys=True), encoding="utf-8") os.replace(tmp, _STORE_PATH) finally: fcntl.flock(lock_fh.fileno(), fcntl.LOCK_UN) # Invalidate rather than adopt: the file we just wrote is the truth, # and re-reading it through the normal path keeps exactly one parser. _cache, _cache_stamp = {}, None _warned.discard("parse") return _read() def set_geo_deny(codes) -> list[str]: """Replace the denylist. Returns what was actually stored.""" cleaned = _clean_deny(codes) def mutate(doc): doc[_GEO_DENY] = cleaned _write(mutate) return cleaned def toggle_country(code: str) -> tuple[list[str], bool]: """Flip one country. Returns ``(denylist, now_denied)``. Invalid codes are refused rather than silently dropped — see :func:`normalize_country` for why a bad code is worse than no code. """ normalized = normalize_country(code) if normalized is None: raise ValueError( f"{code!r} is not an ISO 3166-1 alpha-2 country code. " "The package treats anything else as 'unknown', so this entry " "would never match a request and the block would silently do " "nothing." ) state = {} def mutate(doc): current = _clean_deny(doc.get(_GEO_DENY)) if normalized in current: current.remove(normalized) state["denied"] = False else: current.append(normalized) state["denied"] = True doc[_GEO_DENY] = sorted(current) _write(mutate) return geo_deny(), state["denied"] def set_geo_unknown(posture: str) -> str: if posture not in UNKNOWN_POSTURES: raise ValueError(f"unknown posture must be one of {UNKNOWN_POSTURES}") def mutate(doc): doc[_GEO_UNKNOWN] = posture _write(mutate) return posture def set_vendor_action(vendor: str, action: str) -> dict[str, str]: """Set one vendor's policy. ``action='allow'`` removes the override.""" key = normalize_vendor(vendor) if key is None: raise ValueError( f"{vendor!r} is not a known bot vendor. The package keys " "vendor_policy on its registry key and IGNORES anything else " "with a log line, so this override would silently do nothing. " f"Known: {', '.join(sorted(known_vendors())) or '(registry unavailable)'}" ) if action not in VENDOR_ACTIONS: raise ValueError(f"action must be one of {VENDOR_ACTIONS}") def mutate(doc): current = _clean_vendor_policy(doc.get(_VENDOR_POLICY)) current[key] = action doc[_VENDOR_POLICY] = current _write(mutate) return vendor_policy() def clear_vendor_action(vendor: str) -> dict[str, str]: key = normalize_vendor(vendor) def mutate(doc): current = _clean_vendor_policy(doc.get(_VENDOR_POLICY)) current.pop(key or (vendor or "").strip(), None) doc[_VENDOR_POLICY] = current _write(mutate) return vendor_policy() # --------------------------------------------------------------------------- # What the board shows about itself # --------------------------------------------------------------------------- def status() -> dict[str, Any]: """Store health for the board's footer. ``pid`` and ``mtime`` together are the cross-worker diagnostic: refresh twice, and a pid that changes while mtime holds steady proves every worker is reading the same store. """ try: st = _STORE_PATH.stat() exists, mtime, size = True, st.st_mtime, st.st_size except OSError: exists, mtime, size = False, None, 0 return { "path": str(_STORE_PATH), "exists": exists, "mtime": mtime, "mtime_text": (time.strftime("%Y-%m-%d %H:%M:%S UTC", time.gmtime(mtime)) if mtime else "never written"), "size": size, "pid": os.getpid(), "degraded": "parse" in _warned or "stat" in _warned, "persistent": _persistent(), } def _persistent() -> bool: configured = os.environ.get("POLICY_STORE_FILE") if not configured: return False p = Path(configured) if str(p).startswith("/var/") and len(p.parts) > 2: return os.path.ismount(str(Path("/") / p.parts[1] / p.parts[2])) return True def persistence_warning() -> None: """Loud at boot when board writes would not survive a redeploy. Same two shapes lib/page_visibility.py guards, and for the same reason: both were observed live on the pilot host. A geo denylist that silently resets on deploy is worse than a page-visibility toggle that does, because nothing on the site looks different afterwards — the block simply stops. """ configured = os.environ.get("POLICY_STORE_FILE") if not configured: print( "[policy] WARNING: POLICY_STORE_FILE unset — geo and vendor " "policy are writing to the app directory and will NOT survive a " "redeploy. Set POLICY_STORE_FILE=/var/data/policy_overrides.json " "on the service (render.yaml declares it, but only a Blueprint " "sync or a dashboard add makes the disk live)." ) return if not _persistent(): anchor = Path(configured) anchor = Path("/") / anchor.parts[1] / anchor.parts[2] \ if len(anchor.parts) > 2 else anchor.parent print( f"[policy] WARNING: {anchor} is not a mounted disk on this " "instance — the geo denylist will vanish on the next deploy. " "Attach the render.yaml disk (Blueprint sync, or add it in the " "dashboard)." ) def reset_for_tests() -> None: """Drop the in-process cache and the warn-once ledger.""" global _cache, _cache_stamp with _lock: _cache, _cache_stamp = {}, None _warned.clear() ``` Two details in there are worth lifting out. It re-stats the file on **every** call rather than throttling: the seam's promise is "the next request", and a throttle downgrades that to "probably, within a second" — fine for a page toggle, not for a compliance block. And it preserves keys it does not understand, so a newer worker writing a shape an older one cannot read does not lose that policy on the older worker's next write. Callable failures degrade the safe way: a raising callable is logged **once** and treated as an empty denylist (fail-open). It can never take down the request path. ### `unknown` — the posture for unresolvable countries `"allow"` is the default and deliberately fail-open. It keeps three real traffic classes working on a geo-enabled host: platform health checks (origin-internal, no country header), internal monitoring sweeps, and direct-to-origin fetches. `"deny"` is for operators whose edge guarantees the header on every real request. Under it, health checks survive **only** via `exempt_paths` — so confirm your platform's actual health path first. There are **no User-Agent exemptions**, and there will not be. A UA is trivially spoofable, so an "allow our monitoring bot" rule would be a hole, not a feature. ### Resolution order 1. Your `resolver(headers)`, if configured. For apps with their own geo-IP database. **Never do network I/O in it** — this runs inside every request. 2. `CF-IPCountry` (Cloudflare) 3. `CloudFront-Viewer-Country` (AWS, when the distribution forwards it) 4. `X-Vercel-IP-Country` (Vercel) 5. `Fastly-Geo-Country`, then `X-Country-Code` `XX` (Cloudflare's unknown), `T1` (Tor), empty, and anything that is not two ASCII letters all mean **unknown**, not a country. Network lookups are not supported and will not be. An IP-geolocation call in the request path is a third-party dependency between your visitor and your page. ### Exempt paths match EXACTLY `/healthz` is exempt. `/healthz-evil`, `/healthz/`, `/healthz/anything` and `/HEALTHZ` are **not** — a prefix match would be a bypass anyone could spell. Without the exemption, a geo-enabled host would 451 its own platform health check and be reported down. ### The 451 response One line of `text/plain`, status 451, `Cache-Control: no-store`, plus `Link: <policy_url>; rel="blocked-by"` when a policy URL is configured. **`no-store` is load-bearing.** The response varies by country and **no `Vary` token exists for edge geo headers** — a shared cache storing one country's 451 would serve it to the world. The corollary for *allowed* responses: origin `Vary` cannot express country either, so if you cache at a CDN, cache-key on country there. ### Two accepted consequences - **The discovery floor bends here, once.** `robots.txt`, `sitemap.xml` and the root `llms.txt` are otherwise public everywhere, always. Geo denial is a deliberate exception: compliance, not monetization. Do not cite it as precedent for gating those surfaces anywhere else. - **A session established before the block 451s on its next navigation.** The pages-router POST is covered — that is the point. Total block means total. ### Watch it work The **[policy panel](/showcase/policy-panel)** shows the live denylist as a world map and simulates a request from any country against the real logic. --- *Source: /reference/geo* --- <!-- /reference/panel — https://llms.2plot.dev/reference/panel/llms.txt --> # Operator panel > A read-only, token-gated page showing the live effective policy of every surface the package governs — and why it displays but never writes. --- ### What it is One token-gated route — default `/llms-policy` — rendering the **live effective policy** of every surface the package governs. Opt-in: ```python add_llms_routes(app, LLMSConfig(panel=True)) ``` Token via `LLMSConfig(panel_token=...)` or the `DIMLL_PANEL_TOKEN` environment variable. `panel=False` is the default and registers nothing — the path falls through to your app exactly as before. ### What it shows | Section | Source | The point | |---|---|---| | Identity | package version, base URL, page count | which build answered | | Vendor policy | the **same fold** `robots.txt` renders from | it cannot drift from `robots.txt` | | Bot policy flags | the attached `RobotsConfig`, or the defaults | says == does, even unconfigured | | Tier documents | `LLMSConfig` | corpus posture | | Access control | callback **qualnames only** | who gates, without running the gate | | Geo guardrail | the policy **plus** "this request resolved to `DE` via `cf-ipcountry`" | the live per-host check | | Rate limiting | `rate_limit_per_minute` | the ceiling in force | | Network | directory, bulletin state, hub tightenings | what the hub has tightened | Every section ends with the copy-paste call that would change it. Two details worth knowing. The vendor table renders from `vendors.effective_policies` — the identical function `robots.txt` renders from — so the panel and your published policy are one statement rather than two things to keep in sync. And the access section shows callback **qualnames** and never invokes them: a request-scoped check must not run outside a request. ### The gate - Token compared with `hmac.compare_digest`. - Transported via the **`X-LLMS-Panel-Token`** header (preferred) or `?token=` (convenient in a browser; **it lands in access logs**). - The env var is read **per request** — rotate it and the old token dies on the next request, no redeploy. - **Unset token ⇒ 404, unconditionally.** Production fails closed. - **Wrong token ⇒ 404 with an unrevealing body.** The panel never advertises its own existence. - Deliberately absent from `robots.txt` (a `Disallow` line publishes the path — the `/admin` lesson), from the sitemap, and from the llms index. - Success carries `X-Robots-Tag: noindex, nofollow` and `Cache-Control: private, no-store`. - The [geo guardrail](/reference/geo) 451s the panel too. Intended: "451 on everything" includes the operator standing in a denied country. ```bash curl -s -H 'X-LLMS-Panel-Token: <token>' https://your-site.example/llms-policy ``` ### Read-only, and why that is not timidity Package configuration is **per-process module state**. Under gunicorn's N workers, a panel that *mutated* config would change one worker and lie on the next refresh — a nondeterministically lying control plane, which is worse than no control plane. And a write-capable endpoint behind a single shared token is a remote policy override. So this panel displays and never writes. The footer shows the serving worker's **pid and boot time**. Values that flip between refreshes mean different workers booted with different code or env — a deployment diagnostic, not a panel bug. ### The writable layer above it Your site's own control board, wired through the **callable seams** the panel's hints show: ```python configure_geo(deny_countries=store.geo_deny) RobotsConfig(vendor_policy=store.vendor_policy) ``` Both are read per request by every worker — which is exactly what dissolves the multi-worker problem the read-only decision guards against. Mutate module state and you change one worker; mutate a file every worker re-reads and you change all of them. This site runs that pattern: a flock-guarded JSON store on a mounted disk, written by an admin-gated control board whose every write callback re-checks the gate server-side, and read back through the seams on every request. The **[policy panel showcase](/showcase/policy-panel)** is the public, read-only view of the same state — no token required, because it shows only what this site already publishes about itself. --- *Source: /reference/panel* --- <!-- /showcase/policy-panel — https://llms.2plot.dev/showcase/policy-panel/llms.txt --> # Policy Panel > This host's live effective policy — tiers, vendor verdicts, the hub ceiling, and the country guardrail as a world map — plus a simulator that resolves a hypothetical request through the real in-process logic. --- ### Showcase C — the policy panel Every layer this package adds makes a decision about a request. Individually each is simple; stacked, "why did that visitor get a 451 rather than a sign-in card?" becomes a question nobody can answer from the code. This panel answers it for **this host, right now**. ```python # Live component, rendered above on the browser lane. # Source: docs/policy_panel/policy_panel.py """Showcase C — this host's live effective policy, read-only, plus geo. Read-only in the same sense the package's operator panel is: it displays and never writes. The WRITABLE layer is /admin/control-board, which is admin-gated and re-checks the gate server-side in every write callback. The geo section is real — 2.7.0 is installed, so the choropleth reflects the live denylist through `lib.policy_store`, the same callable `configure_geo` reads on every request. Exec-module rules: ids all start `plcy-`; no import-time registry walk. """ from dash import Input, Output, callback, dcc, html, no_update import dash_mantine_components as dmc from dash_iconify import DashIconify import plotly.graph_objects as go ID = "plcy" # ISO 3166-1 alpha-2 -> alpha-3, for the plotly choropleth's `locations`. # Only the countries a fleet operator is plausibly going to name; anything # else falls back to a plain list so the map is never the only readout. ALPHA2_TO_ALPHA3 = { "AF": "AFG", "AR": "ARG", "AU": "AUS", "AT": "AUT", "BE": "BEL", "BR": "BRA", "BY": "BLR", "CA": "CAN", "CH": "CHE", "CL": "CHL", "CN": "CHN", "CO": "COL", "CU": "CUB", "CZ": "CZE", "DE": "DEU", "DK": "DNK", "EG": "EGY", "ES": "ESP", "FI": "FIN", "FR": "FRA", "GB": "GBR", "GR": "GRC", "HK": "HKG", "HU": "HUN", "ID": "IDN", "IE": "IRL", "IL": "ISR", "IN": "IND", "IQ": "IRQ", "IR": "IRN", "IT": "ITA", "JP": "JPN", "KE": "KEN", "KP": "PRK", "KR": "KOR", "MX": "MEX", "MY": "MYS", "NG": "NGA", "NL": "NLD", "NO": "NOR", "NZ": "NZL", "PE": "PER", "PH": "PHL", "PK": "PAK", "PL": "POL", "PT": "PRT", "RO": "ROU", "RU": "RUS", "SA": "SAU", "SE": "SWE", "SG": "SGP", "SY": "SYR", "TH": "THA", "TR": "TUR", "TW": "TWN", "UA": "UKR", "US": "USA", "VE": "VEN", "VN": "VNM", "ZA": "ZAF", } component = html.Div( [ dmc.Group( [ DashIconify(icon="tabler:shield-lock", width=22), dmc.Text("Live effective policy for this host", fw=600), dmc.Badge("read-only", color="gray", variant="light"), ], gap="sm", ), dmc.Space(h="sm"), html.Div(id=f"{ID}-summary"), dmc.Space(h="lg"), dmc.Text("Country guardrail", fw=600, size="sm"), dmc.Text( "Denied countries receive 451 on every surface — pages, assets, " "the corpus, robots.txt, even the favicon. The list is read from " "the policy store on every request, so the control board changes " "it with no restart.", size="xs", c="dimmed", ), dmc.Space(h="sm"), dcc.Graph(id=f"{ID}-map", config={"displayModeBar": False}, style={"height": "420px"}), html.Div(id=f"{ID}-geo-detail"), dmc.Space(h="lg"), dmc.Text("Simulate a request", fw=600, size="sm"), dmc.Group( [ dmc.Select(id=f"{ID}-sim-page", label="Page", searchable=True, w=280, placeholder="Loading…"), dmc.Select( id=f"{ID}-sim-audience", label="Audience", w=240, data=[ {"value": "human", "label": "A person in a browser"}, {"value": "search", "label": "AI search fetcher"}, {"value": "training", "label": "AI training crawler"}, {"value": "traditional", "label": "Googlebot"}, ], value="human", ), dmc.TextInput(id=f"{ID}-sim-country", label="CF-IPCountry", placeholder="e.g. DE (blank = unknown)", w=180), ], gap="md", align="flex-start", ), dmc.Space(h="sm"), html.Div(id=f"{ID}-sim-verdict"), dcc.Interval(id=f"{ID}-boot", interval=250, max_intervals=1), dcc.Interval(id=f"{ID}-poll", interval=5000), ] ) @callback( Output(f"{ID}-sim-page", "data"), Output(f"{ID}-sim-page", "value"), Input(f"{ID}-boot", "n_intervals"), ) def _pages(_tick): import dash options = sorted( ({"value": e["path"], "label": e.get("name") or e["path"]} for e in dash.page_registry.values() if not e["path"].startswith("/admin/")), key=lambda o: o["label"], ) return (options, "/") if options else (no_update, no_update) @callback( Output(f"{ID}-summary", "children"), Output(f"{ID}-map", "figure"), Output(f"{ID}-geo-detail", "children"), Input(f"{ID}-poll", "n_intervals"), ) def _policy(_tick): """Re-read on a poll so a board toggle shows up here without a refresh.""" from lib import policy_store denied = policy_store.geo_deny() summary = _summary_cards(denied) return summary, _choropleth(denied), _geo_detail(denied) def _summary_cards(denied): import dash from dash_improve_my_llms import __version__ as pkg_version from dash_improve_my_llms.vendors import effective_policies try: policies = effective_policies(getattr(dash.get_app(), "_robots_config", None)) except Exception: # The fold reads a live config object; a public showcase degrades to # zeroes rather than throwing an error toast at a reader. policies = {} blocked = sum(1 for value in policies.values() if value == "block") metered = sum(1 for value in policies.values() if value == "meter") cards = [ ("Package", pkg_version, "blue"), ("Vendors blocked", blocked, "red"), ("Vendors metered", metered, "yellow"), ("Countries denied", len(denied), "grape"), ] return dmc.Group( [ dmc.Paper( dmc.Stack( [ dmc.Text(str(value), size="24px", fw=700, c=color), dmc.Text(label, size="xs", c="dimmed", tt="uppercase"), ], gap=2, align="center", ), withBorder=True, radius="md", p="md", style={"minWidth": "140px"}, ) for label, value, color in cards ], gap="sm", ) def _choropleth(denied): """Allow / deny, as a world map. Read-only — the board owns the toggles.""" codes = [ALPHA2_TO_ALPHA3.get(code) for code in denied] codes = [code for code in codes if code] figure = go.Figure( go.Choropleth( locations=codes or ["ATA"], z=[1] * len(codes) if codes else [0], locationmode="ISO-3", colorscale=[[0, "rgba(0,0,0,0)"], [1, "#e03131"]], showscale=False, marker_line_color="rgba(128,128,128,0.35)", marker_line_width=0.4, hovertemplate="%{location} — denied (451)<extra></extra>", ) ) figure.update_geos( showcoastlines=True, coastlinecolor="rgba(128,128,128,0.35)", showland=True, landcolor="rgba(128,128,128,0.12)", showframe=False, projection_type="natural earth", bgcolor="rgba(0,0,0,0)", ) figure.update_layout( margin={"r": 0, "t": 0, "l": 0, "b": 0}, paper_bgcolor="rgba(0,0,0,0)", geo_bgcolor="rgba(0,0,0,0)", ) return figure def _geo_detail(denied): if not denied: return dmc.Alert( "No countries are denied. The guardrail is wired and configured — " "an empty denylist is a strict no-op, so every response is " "byte-identical to a build without it.", color="teal", variant="light", icon=DashIconify(icon="tabler:world-check"), ) return dmc.Group( [dmc.Badge(code, color="red", variant="filled") for code in denied] + [dmc.Text("→ 451 on every surface", size="xs", c="dimmed")], gap="xs", ) @callback( Output(f"{ID}-sim-verdict", "children"), Input(f"{ID}-sim-page", "value"), Input(f"{ID}-sim-audience", "value"), Input(f"{ID}-sim-country", "value"), ) def _simulate(path, audience, country): """Resolve a hypothetical request through the real in-process logic.""" if not path: return None from lib import access as site_access from lib import policy_store steps = [] denied = policy_store.geo_deny() resolved = (country or "").strip().upper() known = len(resolved) == 2 and resolved.isalpha() and resolved not in ("XX", "T1") if not known: steps.append(("Country", "unknown → allowed (the fail-open default)", "teal")) elif resolved in denied: steps.append(("Country", f"{resolved} is denied → 451, and nothing else runs", "red")) return _verdict_list(steps, "451 Unavailable For Legal Reasons", "red") else: steps.append(("Country", f"{resolved} is not denied → continue", "teal")) ua_class = audience or "human" if ua_class == "human": steps.append(("Vendor policy", "humans are never subject to bot policy", "teal")) else: policy = _policy_for_class(ua_class) colour = {"allow": "teal", "block": "red", "meter": "yellow"}[policy] steps.append(("Vendor policy", f"{ua_class} class → {policy}", colour)) if policy == "block" and not path.endswith("llms.txt"): return _verdict_list(steps, "403 Forbidden", "red") try: tier = site_access.local_tier(path) except Exception: tier = "public" steps.append(("Page tier", f"{tier}", "teal" if tier == "public" else "blue")) if tier == "public": return _verdict_list(steps, "200 OK — the page is served", "teal") return _verdict_list(steps, f"gated ({tier}) — the sign-in card is served", "blue") def _policy_for_class(ua_class): import dash try: from dash_improve_my_llms.vendors import VENDORS, effective_policies policies = effective_policies(getattr(dash.get_app(), "_robots_config", None)) except Exception: return "allow" for vendor in VENDORS: if vendor.cls == ua_class: return policies.get(vendor.key, "allow") return "allow" def _verdict_list(steps, outcome, colour): return dmc.Stack( [ dmc.Timeline( [ dmc.TimelineItem( title=label, children=dmc.Text(detail, size="xs", c="dimmed"), ) for label, detail, _c in steps ], active=len(steps), bulletSize=14, lineWidth=2, ), dmc.Alert(outcome, color=colour, variant="light", icon=DashIconify(icon="tabler:flag-check")), ], gap="sm", ) ``` ### The order the layers run in The simulator above walks them in the order the middleware does, and the order is load-bearing: 1. **Geo.** First, before everything — before the asset short-circuit (which would wave through `/assets/*` and the `POST /_dash-update-component` that carries client-side navigation) and before the bot gate (which would exempt humans). "451 on all surfaces" is one enforcement point with nothing to drift. 2. **Assets.** Static files are not policy-gated. 3. **Vendor policy.** Bots only. Humans are never subject to it. 4. **Rate contract.** Bot traffic on the corpus routes, over the ceiling. 5. **Page tier.** `public` / `auth` / `admin` / `hidden`, with the control board's override winning over frontmatter, and the hub's ceiling applying on top of both. ### The map is read-only. The board is not. The choropleth reflects `lib/policy_store.geo_deny()` — the **same callable** `configure_geo()` reads on every request: ```python # run.py configure_geo(deny_countries=policy_store.geo_deny) ``` That is the 2.7.0 seam, and it is the whole reason this site can have a control board at all. The package's own operator panel (`/llms-policy`, token-gated and 404 to everyone without the token) is read-only by decision: package config is per-process module state, so under N gunicorn workers a *mutating* panel would change one worker and lie on the next refresh. Routing every write through a file the workers re-read per request dissolves that problem — and that is what the control board does. ### What "451 on every surface" means For a denied country, all of these answer `451` with `Cache-Control: no-store`: `/` · any docs page · `/assets/*` · `POST /_dash-update-component` · `/llms.txt` · `/llms-small.txt` · `/llms-full.txt` · `/<page>/llms.txt` · `/robots.txt` · `/sitemap.xml` · `/favicon.ico` · the operator panel itself. `no-store` is not decoration. The response varies by country and **no `Vary` token exists for edge geo headers**, so a shared cache holding one country's 451 would serve it to the world. Exempt paths — `/healthz`, `/health`, `/livez`, `/readyz` — match **exactly**. `/healthz-evil` is blocked. Without the exemption, a geo-enabled host would fail the hub's hourly health sweep and be reported down. ### It is a compliance guardrail, not a security boundary The country comes from an edge header. Behind Cloudflare, `CF-IPCountry` is set by Cloudflare and client-supplied copies are stripped, so it is trustworthy. A client reaching the origin **directly** can say anything. If a block matters adversarially, add the edge WAF rule as well and treat this as the layer that makes the origin's answer *uniform* across every surface — which an edge rule alone cannot do, because it does not know that `/_dash-update-component` carries page navigation or that `/llms-full.txt` is the corpus. On a DNS-only host with no proxy in front, every request resolves "unknown", and under the default `unknown="allow"` the feature ships **inert**. The live per-host check is the operator panel's line — *"this request resolved to: DE (via cf-ipcountry)"*. Verify it before trusting a denylist. ### The hub may only tighten The network bulletin can make this host **more** restrictive — block a vendor, lower the rate ceiling, tighten a page tier. It can never loosen one. A compromised or misconfigured hub can refuse traffic; it cannot open a site that chose to close. A bulletin carrying anything shaped like a payment address is refused **whole**, not sanitized: a fetched pay-to address is a payment-redirection target, and a partially-accepted payload is the exploitable shape. --- *Source: /showcase/policy-panel* --- <!-- /showcase/robots-sandbox — https://llms.2plot.dev/showcase/robots-sandbox/llms.txt --> # Bot Policy Sandbox > Build a RobotsConfig with switches and watch the robots.txt it would generate, plus the per-vendor verdict table — against a throwaway config that never touches what this site serves. --- ### Showcase B — the bot-policy sandbox `robots.txt` is a file most projects write once, by hand, and never check again. This package generates it from a **vendor registry**, which means it can also show you what any configuration *would* produce before you ship it. Move the switches. The document on the right is generated by the same function that serves [`/robots.txt`](/robots.txt) on this host. ```python # Live component, rendered above on the browser lane. # Source: docs/robots_sandbox/robots_sandbox.py """Showcase B — build a RobotsConfig, see the robots.txt it would generate. THE INVARIANT, and it has a test (tests/test_showcase.py): this module never assigns `app._robots_config`. Dash callbacks are global on a shared server, so a sandbox that mutated the live config would let any visitor rewrite what every other visitor — and every crawler — is served. Every render here builds a THROWAWAY config and passes it to `generate_robots_txt(config=...)`. Exec-module rules: ids all start `rbsx-`; no import-time registry walk. """ from dash import Input, Output, callback, html import dash_mantine_components as dmc from dash_iconify import DashIconify ID = "rbsx" SITEMAP = "https://llms.2plot.dev/sitemap.xml" BASE_URL = "https://llms.2plot.dev" _POLICY_COLOR = {"allow": "teal", "block": "red", "meter": "yellow"} def _vendor_rows(): """(key, display, operator, class) for every vendor in the registry.""" from dash_improve_my_llms.vendors import VENDORS return [(v.key, v.display, v.operator, v.cls) for v in VENDORS] component = html.Div( [ dmc.Alert( "This sandbox builds a throwaway config on every render. It never " "touches what this site actually serves — open /robots.txt in " "another tab and it will not change.", title="Nothing here is live", color="blue", variant="light", icon=DashIconify(icon="tabler:shield-check"), ), dmc.Space(h="md"), dmc.Grid( [ dmc.GridCol( dmc.Stack( [ dmc.Text("Coarse flags", fw=600, size="sm"), dmc.Switch(id=f"{ID}-training", checked=True, label="block_ai_training"), dmc.Switch(id=f"{ID}-search", checked=True, label="allow_ai_search"), dmc.Switch(id=f"{ID}-traditional", checked=True, label="allow_traditional"), dmc.Switch(id=f"{ID}-docs", checked=False, label="block_ai_training_docs"), dmc.NumberInput(id=f"{ID}-delay", label="crawl_delay", value=10, min=0, max=120, w=140), dmc.Space(h="xs"), dmc.Text("Per-vendor override (W2)", fw=600, size="sm"), dmc.Select( id=f"{ID}-vendor", label="Vendor", data=[{"value": key, "label": f"{display} ({cls})"} for key, display, _op, cls in _vendor_rows()], searchable=True, clearable=True, placeholder="none", ), dmc.SegmentedControl( id=f"{ID}-action", data=[{"value": "allow", "label": "allow"}, {"value": "meter", "label": "meter"}, {"value": "block", "label": "block"}], value="block", fullWidth=True, ), ], gap="sm", ), span={"base": 12, "md": 4}, ), dmc.GridCol( dmc.Tabs( [ dmc.TabsList([ dmc.TabsTab("Generated robots.txt", value="robots"), dmc.TabsTab("Per-vendor verdicts", value="verdicts"), ]), dmc.TabsPanel(html.Div(id=f"{ID}-robots"), value="robots", pt="sm"), dmc.TabsPanel(html.Div(id=f"{ID}-verdicts"), value="verdicts", pt="sm"), ], value="robots", ), span={"base": 12, "md": 8}, ), ], gutter="lg", ), ] ) @callback( Output(f"{ID}-robots", "children"), Output(f"{ID}-verdicts", "children"), Input(f"{ID}-training", "checked"), Input(f"{ID}-search", "checked"), Input(f"{ID}-traditional", "checked"), Input(f"{ID}-docs", "checked"), Input(f"{ID}-delay", "value"), Input(f"{ID}-vendor", "value"), Input(f"{ID}-action", "value"), ) def _render(training, search, traditional, docs, delay, vendor, action): from dash_improve_my_llms import RobotsConfig from dash_improve_my_llms.robots_generator import generate_robots_txt # THROWAWAY. Never assigned to the app — see this module's docstring. from dash_improve_my_llms.vendors import effective_policies config = RobotsConfig( block_ai_training=bool(training), allow_ai_search=bool(search), allow_traditional=bool(traditional), block_ai_training_docs=bool(docs), crawl_delay=int(delay) if delay else None, vendor_policy={vendor: action} if vendor else None, ) robots = generate_robots_txt(sitemap_url=SITEMAP, base_url=BASE_URL, config=config) policies = effective_policies(config) robots_panel = dmc.Paper( dmc.Code(robots, block=True, style={"maxHeight": "560px", "overflow": "auto", "display": "block"}), withBorder=True, radius="md", p="sm", ) verdicts = dmc.Table( [ dmc.TableThead(dmc.TableTr([ dmc.TableTh("Vendor"), dmc.TableTh("Operator"), dmc.TableTh("Class"), dmc.TableTh("Effective policy"), ])), dmc.TableTbody([ dmc.TableTr([ dmc.TableTd(display), dmc.TableTd(dmc.Text(operator, size="xs", c="dimmed")), dmc.TableTd(dmc.Text(cls, size="xs")), dmc.TableTd(dmc.Badge(policies.get(key, "?"), color=_POLICY_COLOR.get(policies.get(key), "gray"), variant="light")), ]) for key, display, operator, cls in _vendor_rows() ]), ], striped=True, highlightOnHover=True, withTableBorder=True, ) return robots_panel, verdicts ``` ### One invariant, and it has a test The sandbox builds a **throwaway** `RobotsConfig` on every render and passes it as `generate_robots_txt(config=...)`. It never assigns `app._robots_config`. That is not fastidiousness. Dash callbacks are **global on a shared server**: a sandbox that mutated the live config would let any visitor rewrite the policy every other visitor — and every crawler — is served. `tests/ test_showcase.py` asserts the module contains no such assignment, because this is exactly the kind of thing a later refactor makes "simpler". ### What the switches do | Knob | Effect | |---|---| | `block_ai_training` | the training bucket (GPTBot, ClaudeBot, CCBot, …). With it `False` the bucket is **not emitted at all**, which silently *allows* training — not "balanced" | | `allow_ai_search` | Claude-User, ChatGPT-User, PerplexityBot, OAI-SearchBot — the fetchers that send readers | | `allow_traditional` | Googlebot, Bingbot, Slurp, DuckDuckBot. `False` emits real `Disallow` groups | | `block_ai_training_docs` | closes the documentation surfaces to blocked training crawlers. **Off by default** — the corpus exists to get the package used | | `crawl_delay` | rendered inside the `User-agent: *` group, where RFC 9309's strict parsers expect it | | per-vendor override | 2.7.0's `vendor_policy`: one vendor, `allow` / `block` / `meter` | ### `meter`, and why it renders as Allow `meter` means *"fetchable under the rate contract"*. A `Disallow` would kill the funnel the meter exists for, so robots.txt renders it as `Allow` and the middleware treats it as allow until the limiter consumes it. ### One fold, two consumers The verdict table and the generated document are both rendered from `vendors.effective_policies(config)` — one function. That is the design that makes "what the site says" and "what the site does" the same statement rather than two things to keep in sync. ```python from dash_improve_my_llms.vendors import effective_policies effective_policies(app._robots_config) # {'gptbot': 'block', 'googlebot': 'allow', ...} ``` The package's read-only operator panel (`/llms-policy`) renders its vendor table from this same call, which is why it cannot drift from `robots.txt`. --- *Source: /showcase/robots-sandbox*