# Disallow all crawlers access to certain pages. User-agent: * Disallow: /admin # Google sitemap Sitemap: http://www.studentenwerke-bw.de/sitemap.xml User-agent: SlySearch Disallow: / User-agent: SmartDownload Disallow: / User-agent: SpankBot Disallow: / User-agent: spanner Disallow: / User-agent: SuperBot Disallow: / User-agent: SuperHTTP Disallow: / User-agent: Surfbot Disallow: /