User-agent: relemindbot Disallow: / User-agent: YandexBot Disallow: / User-agent: ia_archiver Disallow: /samples User-agent: 008 Disallow: / User-agent: Bytespider Disallow: / User-agent: GPTBot Disallow: / User-agent: ChatGPT-User Disallow: / User-agent: BoardReader Disallow: / User-agent: anthropic-ai Disallow: / User-agent: Yeti Disallow: / User-agent: PetalBot Disallow: / User-agent: MJ12bot Disallow: / user-agent: arquivo-web-crawler disallow: / user-agent: Sogou web spider disallow: / user-agent: trendictionbot disallow: / user-agent: SemrushBot disallow: / user-agent: Riddler disallow: / user-agent: DotBot disallow: / user-agent: GrapeshotCrawle disallow: / user-agent: MauiBot disallow: / user-agent: yacybot disallow: / user-agent: CCBot disallow: / user-agent: BLEXBot disallow: / user-agent: AhrefsBot disallow: / User-agent: DataForSeoBot disallow: / Crawl-delay: 5.0 User-agent: Slurp Crawl-delay: 3.0 user-agent: YandexBot Disallow: /astro-databank Disallow: /swisseph Disallow: /forarch user-agent: bingbot Crawl-delay: 3.0 User-agent: baiduspider* Crawl-delay: 5.0 Disallow: /swisseph user-agent: FreeFind disallow: /*.pdf$ disallow: /cgi/chart.cgi user-agent: OpenindexSpider disallow: /*.pdf$ disallow: /cgi/chart.cgi Disallow: /astro-databank/Category: user-agent: Mediapartners-Google Crawl-delay: 5.0 user-agent: Googlebot Crawl-delay: 5.0 user-agent: CriteoBot/0.1 Crawl-delay: 5.0 User-agent: * Allow: / Crawl-delay: 2.0 Disallow: /apache Disallow: /adb/ Disallow: /design Disallow: /hfot Disallow: /h/ Disallow: /htm/ Disallow: /license Disallow: /sstat Disallow: /stats Disallow: /tmpd Disallow: /trans Disallow: /webp Disallow: /cgi/sub.cgi Disallow: /cgi/bugrep.cgi Disallow: /cgi/ovi.cgi Disallow: /cgi/ade.cgi Disallow: /astrowiki/de/Benutzer Disallow: /astrowiki/de/Diskussion Disallow: /astrowiki/en/User Disallow: /astrowiki/en/Talk Disallow: /astro-databank/Private: Disallow: /wiki/*/index.php Sitemap: https://www.astro.com/sitemap_index.xml