# Everything public is crawlable. The app (/o/) and the OAuth hops (/auth/) # sit behind login. The legal pages are indexable on purpose: they carry # the company details, refund and cancellation terms that search engines # and answer engines use as trust signals. /email_preferences keeps itself # out of the index with a robots meta tag. User-agent: * Disallow: /o/ Disallow: /auth/ Allow: / # Explicit welcome for search and answer-engine crawlers, same rules as *. # Training crawlers are allowed too: being in the training data is what # makes a small brand show up in un-grounded answers. User-agent: GPTBot User-agent: OAI-SearchBot User-agent: ChatGPT-User User-agent: ClaudeBot User-agent: Claude-SearchBot User-agent: Claude-User User-agent: PerplexityBot User-agent: Perplexity-User User-agent: Google-Extended User-agent: Applebot User-agent: Applebot-Extended User-agent: meta-externalagent User-agent: Amazonbot User-agent: DuckAssistBot User-agent: bingbot User-agent: CCBot User-agent: Bytespider Disallow: /o/ Disallow: /auth/ Allow: / Sitemap: https://postcrest.com/sitemap.xml