The Staging robots.txt That Indexed Fake Checkout Data

Postmortem excerpt: staging lived at staging.shop.example. Someone copied production’s robots.txt during a Cloudflare Pages promote. Googlebot found test SKUs, fake customer emails in HTML fixtures, and a “Place order” flow that talked to a sandbox processor. Leadership asked why robots did not “block Google.” Because robots.txt is not a lock.

What went wrong (mechanics)

1. Staging was publicly reachable (no basic auth).
2. robots.txt either allowed crawling (prod copy) or was ignored by a non-Google bot that still mirrored pages.
3. Internal Slack links and a partner’s QA bookmark gave Google discovery paths.
4. Disallow was misunderstood as “delete from the index.”

Even a correct Disallow: / only asks polite crawlers to skip fetches. URLs can still be indexed with limited information. Malicious scrapers ignore the file entirely.

Layered protection for non-prod

Control	Role
Network / Basic auth / SSO gate	Real access control
X-Robots-Tag: noindex, nofollow	Indexing signal on every response
Separate hostname + no prod backlinks	Reduces discovery
robots.txt Disallow	Soft crawl budget hint
Synthetic data only	Limits blast radius if indexed

Ship noindex via edge headers so a forgotten meta tag in one template cannot open a hole. Keep staging credentials out of the HTML. Prefer ephemeral preview URLs over long-lived public staging.

Production rules worth generating carefully

A robots.txt generator helps you avoid syntax mistakes (Disallow vs Allow order, wildcards, sitemap lines). Useful patterns:

• Allow marketing and docs; disallow search result URLs with endless query params
• Do not block assets required for rendering if you care about search appearance
• Reference Sitemap: https://example.com/sitemap-index.xml
• Keep a different artifact for staging in CI — never promote the prod file blindly

Environment-specific config (Workers, Netlify headers, nginx) should inject staging rules. Treat “same robots as prod” as a deploy smell.

Incident cleanup when Google already visited

• Add auth + noindex immediately
• Use Search Console removals / temporary hide for sensitive URLs
• Rotate any credentials that appeared in fixtures
• Audit who linked to staging from public pages
• Add a CI check: fail the build if staging hostname serves 200 without noindex

Generators vs tribal knowledge

Hand-written robots files accumulate myths (“Disallow secret folders,” “crawl-delay fixes SEO”). Generate from intent: which user-agents, which path prefixes, which sitemap. Review the output like code. One wrong Disallow: / on production is as famous as the staging leak — traffic cliffs look like “Google punished us” when we asked crawlers to leave.

Sitemap and robots interaction

A production Sitemap: line that accidentally points at staging, or a staging sitemap submitted in Search Console, recreates the leak through a different door. Keep Search Console properties environment-scoped. Generators should emit sitemap URLs from config, not from a hard-coded prod hostname pasted into every file.

Also watch Allow/Disallow order with wildcards. A broad Allow can reopen paths you thought you closed. Validate with Google’s robots testing tools on the exact production URL after deploy — syntax that “looks fine” locally can still surprise at the edge if two robots.txt files exist (apex vs www, or Workers overriding Pages).

Checklist before promoting config

• Staging hostname serves auth or IP allowlist
• Staging responses include noindex
• Staging robots.txt is not a copy of production without review
• Production robots.txt does not Disallow critical render assets
• CI fails if Disallow: / appears on the production branch artifact

robots.txt is a polite note on the door. Staging needs a lock, a “no photos” sign, and furniture that is not real customer data.