User-agent: *
Allow: /

# The app surfaces are NOT listed here on purpose. Two reasons, both learned
# from Google Search Console on 2026-08-17:
#
#  1. robots.txt cannot deindex anything. It blocks crawling, not indexing — a
#     blocked URL that is linked still gets a URL-only entry, which is exactly
#     what "Indexed, though blocked by robots.txt" means. Worse, blocking it
#     guarantees Google never sees the noindex that WOULD deindex it.
#  2. /studio, /batch and the other product surfaces are linked site-wide and
#     browsable signed out on purpose ("someone who cannot see the product
#     cannot decide to buy it", App.tsx). Blocking what you advertise is a
#     contradiction Google reports back to you every time.
#
# So: product surfaces are crawlable and indexable. Genuinely private routes
# (/dashboard, /presets, /affiliate/dashboard, /developer, /admin/*) stay crawlable but
# send `X-Robots-Tag: noindex` — see _NOINDEX_ROUTES in api/main.py.
#
# Only opaque subtrees with nothing to index are still blocked here. Note these
# are PREFIX matches: an unanchored rule blocks every path that starts with it,
# which is how `Disallow: /affiliate` also blocked /affiliates and
# /affiliate-terms. Anchor with $ if you ever add a single-page rule back.
Disallow: /api/
Disallow: /_umami/

# ── AI assistants ─────────────────────────────────────────────────────────────
# RETRIEVAL IS ALLOWED, TRAINING IS NOT. The split is deliberate and it is the
# same split the edge enforces (see the note below), so this file and Cloudflare
# now say the same thing.
#
# Retrieval crawlers index the site in order to CITE it, and that is already the
# single biggest signup source: chatgpt.com sent 22 of the first 94 accounts,
# more than Google and Bing combined (measured 2026-08-29 from
# users.signup_source). For an unknown brand that is free distribution.
#
# Training crawlers take the content into model weights and return nothing. An
# earlier version of this file allowed them too, arguing that being in the
# corpus is free permanent advertising. That is a real argument and it is why
# the reversal is written down rather than just made: Petar's call, 2026-08-30,
# is that Audexum's content is not to be scraped for training.
#
# NOTE ON THE FORMAT, because it bites: a crawler that matches a named
# User-agent group obeys ONLY that group and ignores `User-agent: *` entirely.
# So every named group below repeats the two Disallow lines. Deleting them from
# one group silently opens /api/ to that crawler alone.
#
# THIS FILE IS NOT THE BINDING CONTROL, the edge is. Since 2026-10-22 (Petar's
# call) the audexum.com zone allows ALL AI bots: AI bot policies Search, Agent
# and Training = Allow, and the WAF custom rule "Block mixed-purpose AI training
# crawlers" (GPTBot/ClaudeBot) is DISABLED, not deleted. Verified live: GPTBot,
# ClaudeBot, CCBot, PerplexityBot, OAI-SearchBot, ChatGPT-User, Claude-SearchBot,
# Perplexity-User, Googlebot all 200. Training is allowed on purpose: Audexum
# sells a service, not its words, so a model that learns the name is free reach.
# Check the zone before trusting this file:
#   GET /zones/{zone_id}/bot_management
# A crawler that ignores this file gets a 403 anyway; one that honours it is
# told the same thing politely, which is the only reason to keep both in sync.

# Retrieval — these put audexum.com INTO an assistant's answer. Allowed.
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: PerplexityBot
User-agent: Perplexity-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: Bingbot
Allow: /
Disallow: /api/
Disallow: /_umami/

# Training — allowed since 2026-10-22. The /api/ and /_umami/ lines still apply
# (Allow / is overridden by the longer Disallow match), and every named group
# must repeat them, see above.
#
# Google-Extended and Applebot-Extended are not crawlers, they are consent
# tokens: they gate whether Gemini and Apple Intelligence may TRAIN on the site.
# They sit here for that reason. Google-Extended is the one line with a real
# cost — it also gates Gemini grounding, so refusing it gives up the same kind
# of citation traffic ChatGPT provides. Move it back up one group to reverse
# that single trade; Googlebot and plain web search are unaffected either way.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Amazonbot
User-agent: Meta-ExternalAgent
User-agent: Bytespider
User-agent: Google-Extended
User-agent: Applebot-Extended
Allow: /
Disallow: /api/
Disallow: /_umami/

Sitemap: https://audexum.com/sitemap.xml