# As a condition of accessing this website, you agree to abide by the following # content signals: # (a) If a Content-Signal = yes, you may collect content for the corresponding # use. # (b) If a Content-Signal = no, you may not collect content for the # corresponding use. # (c) If the website operator does not include a Content-Signal for a # corresponding use, the website operator neither grants nor restricts # permission via Content-Signal with respect to the corresponding use. # The content signals and their meanings are: # search: building a search index and providing search results (e.g., returning # hyperlinks and short excerpts from your website's contents). Search does not # include providing AI-generated search summaries. # ai-input: inputting content into one or more AI models (e.g., retrieval # augmented generation, grounding, or other real-time taking of content for # generative AI search answers). # ai-train: training or fine-tuning AI models. # use: how AI systems may consume the content (immediate, reference, or full). # ANY RESTRICTIONS EXPRESSED VIA CONTENT SIGNALS ARE EXPRESS RESERVATIONS OF # RIGHTS UNDER ARTICLE 4 OF THE EUROPEAN UNION DIRECTIVE 2019/790 ON COPYRIGHT # AND RELATED RIGHTS IN THE DIGITAL SINGLE MARKET. # BEGIN Cloudflare Managed content User-agent: * Content-Signal: search=yes,ai-train=no,use=reference Allow: / # Training crawlers User-agent: Amazonbot Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: Bytespider Disallow: / User-agent: CCBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: Diffbot Disallow: / User-agent: Google-Extended Disallow: / User-agent: GPTBot Disallow: / User-agent: omgili Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Claude-Web Disallow: / User-agent: cohere-ai Disallow: / User-agent: MistralAI-Training Disallow: / User-agent: meta-externalagent Disallow: / User-agent: GoogleOther Disallow: / User-agent: Baiduspider Disallow: / User-agent: PetalBot Disallow: / User-agent: AwarioSmartBot User-agent: AwarioRssBot Disallow: / User-agent: Google-CloudVertexBot Disallow: / User-agent: QualifiedBot Disallow: / User-agent: Cotoyogi Disallow: / User-agent: ICC-Crawler Disallow: / User-agent: atlassian-bot Disallow: / User-agent: FishBot Disallow: / User-agent: BorderxBot Disallow: / User-agent: NavuBot Disallow: / User-agent: SemrushBot-SWA Disallow: / User-agent: WARDBot Disallow: / User-agent: magpie-crawler Disallow: / User-agent: KimiBot Disallow: / User-agent: CitibotSiteCrawler Disallow: / # END Cloudflare Managed Content # Coen Construction robots.txt # https://coenconstruction.com # Last updated: 2026-07-19 # Google - full access to public content User-agent: Googlebot Allow: / Allow: /api/functions/serveSitemap Disallow: /api/ Disallow: /admin/ Disallow: /estimator/ Disallow: /website/ Disallow: /snow/ Disallow: /login Disallow: /register Disallow: /forgot-password Disallow: /reset-password Disallow: /my-projects Disallow: /project Disallow: /shared-design Disallow: /customer-portal Disallow: /subcontractor-portal Disallow: /sub-bid-portal Disallow: /sub-onboarding Disallow: /sub-doc-upload Disallow: /employee-onboarding Disallow: /payroll-approval Disallow: /estimate-approval Disallow: /vendor/ Disallow: /field Disallow: /staff/ # Bing - full access to public content User-agent: Bingbot Allow: / Allow: /api/functions/serveSitemap Disallow: /api/ Disallow: /admin/ Disallow: /estimator/ Disallow: /website/ Disallow: /snow/ Disallow: /login Disallow: /register Disallow: /forgot-password Disallow: /reset-password Disallow: /my-projects Disallow: /project Disallow: /shared-design Disallow: /customer-portal Disallow: /subcontractor-portal Disallow: /sub-bid-portal Disallow: /sub-onboarding Disallow: /sub-doc-upload Disallow: /employee-onboarding Disallow: /payroll-approval Disallow: /estimate-approval Disallow: /vendor/ Disallow: /field Disallow: /staff/ # Other search engines User-agent: Slurp Allow: / Allow: /api/functions/serveSitemap Disallow: /api/ Disallow: /admin/ Disallow: /estimator/ Disallow: /website/ Disallow: /snow/ User-agent: DuckDuckBot Allow: / Allow: /api/functions/serveSitemap Disallow: /api/ Disallow: /admin/ Disallow: /estimator/ Disallow: /website/ Disallow: /snow/ User-agent: Baiduspider Allow: / Allow: /api/functions/serveSitemap Disallow: /api/ Disallow: /admin/ Disallow: /estimator/ Disallow: /website/ Disallow: /snow/ # AI / LLM crawlers - allowed on public marketing pages (for AI search # visibility), blocked from the admin/customer/vendor app same as everyone # else. Same Allow/Disallow pattern as Googlebot/Bingbot above. User-agent: GPTBot Allow: / Allow: /api/functions/serveSitemap Disallow: /api/ Disallow: /admin/ Disallow: /estimator/ Disallow: /website/ Disallow: /snow/ Disallow: /login Disallow: /register Disallow: /forgot-password Disallow: /reset-password Disallow: /my-projects Disallow: /project Disallow: /shared-design Disallow: /customer-portal Disallow: /subcontractor-portal Disallow: /sub-bid-portal Disallow: /sub-onboarding Disallow: /sub-doc-upload Disallow: /employee-onboarding Disallow: /payroll-approval Disallow: /estimate-approval Disallow: /vendor/ Disallow: /field Disallow: /staff/ User-agent: ChatGPT-User Allow: / Allow: /api/functions/serveSitemap Disallow: /api/ Disallow: /admin/ Disallow: /estimator/ Disallow: /website/ Disallow: /snow/ Disallow: /login Disallow: /register Disallow: /forgot-password Disallow: /reset-password Disallow: /my-projects Disallow: /project Disallow: /shared-design Disallow: /customer-portal Disallow: /subcontractor-portal Disallow: /sub-bid-portal Disallow: /sub-onboarding Disallow: /sub-doc-upload Disallow: /employee-onboarding Disallow: /payroll-approval Disallow: /estimate-approval Disallow: /vendor/ Disallow: /field Disallow: /staff/ User-agent: CCBot Allow: / Allow: /api/functions/serveSitemap Disallow: /api/ Disallow: /admin/ Disallow: /estimator/ Disallow: /website/ Disallow: /snow/ Disallow: /login Disallow: /register Disallow: /forgot-password Disallow: /reset-password Disallow: /my-projects Disallow: /project Disallow: /shared-design Disallow: /customer-portal Disallow: /subcontractor-portal Disallow: /sub-bid-portal Disallow: /sub-onboarding Disallow: /sub-doc-upload Disallow: /employee-onboarding Disallow: /payroll-approval Disallow: /estimate-approval Disallow: /vendor/ Disallow: /field Disallow: /staff/ User-agent: anthropic-ai Allow: / Allow: /api/functions/serveSitemap Disallow: /api/ Disallow: /admin/ Disallow: /estimator/ Disallow: /website/ Disallow: /snow/ Disallow: /login Disallow: /register Disallow: /forgot-password Disallow: /reset-password Disallow: /my-projects Disallow: /project Disallow: /shared-design Disallow: /customer-portal Disallow: /subcontractor-portal Disallow: /sub-bid-portal Disallow: /sub-onboarding Disallow: /sub-doc-upload Disallow: /employee-onboarding Disallow: /payroll-approval Disallow: /estimate-approval Disallow: /vendor/ Disallow: /field Disallow: /staff/ User-agent: Claude-Web Allow: / Allow: /api/functions/serveSitemap Disallow: /api/ Disallow: /admin/ Disallow: /estimator/ Disallow: /website/ Disallow: /snow/ Disallow: /login Disallow: /register Disallow: /forgot-password Disallow: /reset-password Disallow: /my-projects Disallow: /project Disallow: /shared-design Disallow: /customer-portal Disallow: /subcontractor-portal Disallow: /sub-bid-portal Disallow: /sub-onboarding Disallow: /sub-doc-upload Disallow: /employee-onboarding Disallow: /payroll-approval Disallow: /estimate-approval Disallow: /vendor/ Disallow: /field Disallow: /staff/ User-agent: Google-Extended Allow: / Allow: /api/functions/serveSitemap Disallow: /api/ Disallow: /admin/ Disallow: /estimator/ Disallow: /website/ Disallow: /snow/ Disallow: /login Disallow: /register Disallow: /forgot-password Disallow: /reset-password Disallow: /my-projects Disallow: /project Disallow: /shared-design Disallow: /customer-portal Disallow: /subcontractor-portal Disallow: /sub-bid-portal Disallow: /sub-onboarding Disallow: /sub-doc-upload Disallow: /employee-onboarding Disallow: /payroll-approval Disallow: /estimate-approval Disallow: /vendor/ Disallow: /field Disallow: /staff/ User-agent: FacebookBot Allow: / Allow: /api/functions/serveSitemap Disallow: /api/ Disallow: /admin/ Disallow: /estimator/ Disallow: /website/ Disallow: /snow/ Disallow: /login Disallow: /register Disallow: /forgot-password Disallow: /reset-password Disallow: /my-projects Disallow: /project Disallow: /shared-design Disallow: /customer-portal Disallow: /subcontractor-portal Disallow: /sub-bid-portal Disallow: /sub-onboarding Disallow: /sub-doc-upload Disallow: /employee-onboarding Disallow: /payroll-approval Disallow: /estimate-approval Disallow: /vendor/ Disallow: /field Disallow: /staff/ User-agent: cohere-ai Allow: / Allow: /api/functions/serveSitemap Disallow: /api/ Disallow: /admin/ Disallow: /estimator/ Disallow: /website/ Disallow: /snow/ Disallow: /login Disallow: /register Disallow: /forgot-password Disallow: /reset-password Disallow: /my-projects Disallow: /project Disallow: /shared-design Disallow: /customer-portal Disallow: /subcontractor-portal Disallow: /sub-bid-portal Disallow: /sub-onboarding Disallow: /sub-doc-upload Disallow: /employee-onboarding Disallow: /payroll-approval Disallow: /estimate-approval Disallow: /vendor/ Disallow: /field Disallow: /staff/ User-agent: PerplexityBot Allow: / Allow: /api/functions/serveSitemap Disallow: /api/ Disallow: /admin/ Disallow: /estimator/ Disallow: /website/ Disallow: /snow/ Disallow: /login Disallow: /register Disallow: /forgot-password Disallow: /reset-password Disallow: /my-projects Disallow: /project Disallow: /shared-design Disallow: /customer-portal Disallow: /subcontractor-portal Disallow: /sub-bid-portal Disallow: /sub-onboarding Disallow: /sub-doc-upload Disallow: /employee-onboarding Disallow: /payroll-approval Disallow: /estimate-approval Disallow: /vendor/ Disallow: /field Disallow: /staff/ User-agent: Applebot-Extended Allow: / Allow: /api/functions/serveSitemap Disallow: /api/ Disallow: /admin/ Disallow: /estimator/ Disallow: /website/ Disallow: /snow/ Disallow: /login Disallow: /register Disallow: /forgot-password Disallow: /reset-password Disallow: /my-projects Disallow: /project Disallow: /shared-design Disallow: /customer-portal Disallow: /subcontractor-portal Disallow: /sub-bid-portal Disallow: /sub-onboarding Disallow: /sub-doc-upload Disallow: /employee-onboarding Disallow: /payroll-approval Disallow: /estimate-approval Disallow: /vendor/ Disallow: /field Disallow: /staff/ # All other crawlers - public content allowed, private areas blocked User-agent: * Allow: / Allow: /api/functions/serveSitemap # API routes are not public pages, except the sitemap endpoint allowed above. Disallow: /api/ # Admin and internal tools Disallow: /admin/ Disallow: /estimator/ Disallow: /website/ Disallow: /snow/ Disallow: /login Disallow: /register Disallow: /forgot-password Disallow: /reset-password # Authenticated user areas. /start and /budget-estimator are public lead-gen pages. Disallow: /my-projects Disallow: /project Disallow: /shared-design # Vendor, customer, subcontractor, and employee portals Disallow: /vendor/ Disallow: /estimate-approval Disallow: /customer-portal Disallow: /subcontractor-portal Disallow: /sub-bid-portal Disallow: /sub-onboarding Disallow: /sub-doc-upload Disallow: /employee-onboarding Disallow: /payroll-approval Disallow: /field Disallow: /staff/ # Duplicate-content query parameters Disallow: /*?ref= Disallow: /*?utm_source= Disallow: /*?utm_medium= Disallow: /*?utm_campaign= Disallow: /*?utm_term= Disallow: /*?utm_content= Disallow: /*?fbclid= Disallow: /*?gclid= Disallow: /*?sessionid= # Throttle aggressive unknown crawlers. Crawl-delay: 10 # This file is coenconstruction.com's, and only coenconstruction.com's. # # coen.builders is a SEPARATE property — a different domain, a different brand, # its own Search Console property, its own sitemap. It is served from the same # Cloudflare Pages project, but nothing here speaks for it: functions/_middleware.js # serves coen.builders its own robots.txt (with its own Sitemap: line) on its own # hostname, and scripts/prerender-builders.mjs writes its sitemap. # # A Sitemap: line for coen.builders used to sit below. Google does not honour a # cross-host sitemap declaration unless the hosts are cross-verified, so it bought # nothing, and it is the one place this file named the other brand. Declare # coen.builders' sitemap in coen.builders' own Search Console property instead. Sitemap: https://coenconstruction.com/sitemap.xml Sitemap: https://coenconstruction.com/api/functions/serveSitemap