# Block known scrapers and data harvesters. # Amazonbot is NOT here on purpose (GEO audit 2026-09-08): it feeds Alexa+ and # Rufus, which are answer surfaces, and the standing rule is that no AI answer # or training crawler is blocked. It is throttled with the training bots below. User-agent: AhrefsBot Disallow: / User-agent: SemrushBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: DotBot Disallow: / User-agent: DataForSeoBot Disallow: / User-agent: PetalBot Disallow: / User-agent: BLEXBot Disallow: / User-agent: MegaIndex Disallow: / User-agent: SEOkicks Disallow: / User-agent: SplitSignalBot Disallow: / User-agent: Bytespider Disallow: / # Every block below repeats the SAME disallow list. A crawler that matches a # specific User-agent block ignores the `*` block entirely, so a rule stated # only under `*` applies to nobody named here. # # /signin was the original entry (17,937 of 90,763 AI-crawler hits in the 30 # days to 2026-08-18, a login form eating 19.8% of the crawl budget). The GEO # audit of 2026-09-08 found the same leak on every other noindex surface: # /journal, /settings, /studio, /trips/new, /e/ (event pages), /vote/. They # carry `noindex` in metadata, which stops indexing but not fetching, so they # are blocked here as well. /api/ is internal JSON and is now disallowed in # every block (it used to be disallowed only under `*`, so every named AI bot # was allowed into it); /api/dataset/ stays open because it is the public, # CC BY 4.0 dataset endpoint that the open-data landing page advertises. User-agent: * Allow: / Disallow: /quiz Disallow: /signin Disallow: /journal Disallow: /settings Disallow: /studio Disallow: /trips/new Disallow: /e/ Disallow: /vote/ Allow: /api/dataset/ Disallow: /api/ User-agent: Googlebot Allow: / Disallow: /quiz Disallow: /signin Disallow: /journal Disallow: /settings Disallow: /studio Disallow: /trips/new Disallow: /e/ Disallow: /vote/ Allow: /api/dataset/ Disallow: /api/ User-agent: Bingbot Allow: / Disallow: /quiz Disallow: /signin Disallow: /journal Disallow: /settings Disallow: /studio Disallow: /trips/new Disallow: /e/ Disallow: /vote/ Allow: /api/dataset/ Disallow: /api/ # AI bots: search/assistant fetchers (answering user questions, citing Spill) # crawl fast; model-training crawlers stay throttled at 10s. User-agent: GPTBot Allow: / Disallow: /quiz Disallow: /signin Disallow: /journal Disallow: /settings Disallow: /studio Disallow: /trips/new Disallow: /e/ Disallow: /vote/ Allow: /api/dataset/ Disallow: /api/ Crawl-delay: 10 User-agent: OAI-SearchBot Allow: / Disallow: /quiz Disallow: /signin Disallow: /journal Disallow: /settings Disallow: /studio Disallow: /trips/new Disallow: /e/ Disallow: /vote/ Allow: /api/dataset/ Disallow: /api/ Crawl-delay: 1 User-agent: ChatGPT-User Allow: / Disallow: /quiz Disallow: /signin Disallow: /journal Disallow: /settings Disallow: /studio Disallow: /trips/new Disallow: /e/ Disallow: /vote/ Allow: /api/dataset/ Disallow: /api/ Crawl-delay: 1 User-agent: ClaudeBot Allow: / Disallow: /quiz Disallow: /signin Disallow: /journal Disallow: /settings Disallow: /studio Disallow: /trips/new Disallow: /e/ Disallow: /vote/ Allow: /api/dataset/ Disallow: /api/ Crawl-delay: 10 User-agent: Claude-SearchBot Allow: / Disallow: /quiz Disallow: /signin Disallow: /journal Disallow: /settings Disallow: /studio Disallow: /trips/new Disallow: /e/ Disallow: /vote/ Allow: /api/dataset/ Disallow: /api/ Crawl-delay: 1 User-agent: Claude-User Allow: / Disallow: /quiz Disallow: /signin Disallow: /journal Disallow: /settings Disallow: /studio Disallow: /trips/new Disallow: /e/ Disallow: /vote/ Allow: /api/dataset/ Disallow: /api/ Crawl-delay: 1 User-agent: anthropic-ai Allow: / Disallow: /quiz Disallow: /signin Disallow: /journal Disallow: /settings Disallow: /studio Disallow: /trips/new Disallow: /e/ Disallow: /vote/ Allow: /api/dataset/ Disallow: /api/ Crawl-delay: 10 User-agent: CCBot Allow: / Disallow: /quiz Disallow: /signin Disallow: /journal Disallow: /settings Disallow: /studio Disallow: /trips/new Disallow: /e/ Disallow: /vote/ Allow: /api/dataset/ Disallow: /api/ Crawl-delay: 10 User-agent: PerplexityBot Allow: / Disallow: /quiz Disallow: /signin Disallow: /journal Disallow: /settings Disallow: /studio Disallow: /trips/new Disallow: /e/ Disallow: /vote/ Allow: /api/dataset/ Disallow: /api/ Crawl-delay: 1 User-agent: Perplexity-User Allow: / Disallow: /quiz Disallow: /signin Disallow: /journal Disallow: /settings Disallow: /studio Disallow: /trips/new Disallow: /e/ Disallow: /vote/ Allow: /api/dataset/ Disallow: /api/ Crawl-delay: 1 User-agent: Google-Extended Allow: / Disallow: /quiz Disallow: /signin Disallow: /journal Disallow: /settings Disallow: /studio Disallow: /trips/new Disallow: /e/ Disallow: /vote/ Allow: /api/dataset/ Disallow: /api/ Crawl-delay: 10 User-agent: Amazonbot Allow: / Disallow: /quiz Disallow: /signin Disallow: /journal Disallow: /settings Disallow: /studio Disallow: /trips/new Disallow: /e/ Disallow: /vote/ Allow: /api/dataset/ Disallow: /api/ Crawl-delay: 10 User-agent: Applebot Allow: / Disallow: /quiz Disallow: /signin Disallow: /journal Disallow: /settings Disallow: /studio Disallow: /trips/new Disallow: /e/ Disallow: /vote/ Allow: /api/dataset/ Disallow: /api/ Crawl-delay: 1 User-agent: Applebot-Extended Allow: / Disallow: /quiz Disallow: /signin Disallow: /journal Disallow: /settings Disallow: /studio Disallow: /trips/new Disallow: /e/ Disallow: /vote/ Allow: /api/dataset/ Disallow: /api/ Crawl-delay: 10 User-agent: meta-externalagent Allow: / Disallow: /quiz Disallow: /signin Disallow: /journal Disallow: /settings Disallow: /studio Disallow: /trips/new Disallow: /e/ Disallow: /vote/ Allow: /api/dataset/ Disallow: /api/ Crawl-delay: 10 User-agent: DuckAssistBot Allow: / Disallow: /quiz Disallow: /signin Disallow: /journal Disallow: /settings Disallow: /studio Disallow: /trips/new Disallow: /e/ Disallow: /vote/ Allow: /api/dataset/ Disallow: /api/ Crawl-delay: 1 Sitemap: https://www.spillthe.wine/sitemap_index.xml Sitemap: https://www.spillthe.wine/sitemap/static.xml Sitemap: https://www.spillthe.wine/sitemap/wineries.xml Sitemap: https://www.spillthe.wine/sitemap/browse.xml Sitemap: https://www.spillthe.wine/sitemap/lists.xml Sitemap: https://www.spillthe.wine/sitemap/community-lists.xml Sitemap: https://www.spillthe.wine/sitemap/trips.xml Sitemap: https://www.spillthe.wine/sitemap/regions.xml Sitemap: https://www.spillthe.wine/sitemap/amenity-regions.xml