# Robots.txt file from http://www.searchengineworld.com # # Built from text file #http://info.webcrawler.com/mak/projects/robots/active/all.txt # # This restricts access to only known and registered robots. # User-agent: * Disallow: / User-agent: Googlebot User-agent: msnbot User-agent: Slurp Disallow: User-agent: * Disallow: /_app/ Disallow: /_images/ Disallow: /secure/ Disallow: /xsecure/ Disallow: /adminarea/ Disallow: /_assets/ Disallow: /_swf/ Disallow: /_screenshots/ Disallow: /misc/ Disallow: /mobile/ Disallow: /cgi-bin/ Disallow: /go/