User-agent: * Disallow: /global_inc/ # While there is nothing to hide in the engine dir crawlers can publish confusing links # in their search results so we exclude the big guys from crawling it. Certain crawlers # do need access to this so we only exclude the big ones User-agent: *Google* Disallow: /global_engine/ User-agent: *googlebot* Disallow: /global_engine/ User-agent: *Slurp* Disallow: /global_engine/ User-agent: *sitecheck* Disallow: /global_engine/ User-agent: *mercator* Disallow: /global_engine/ User-agent: *ia_archiver* Disallow: /global_engine/ User-agent: *webcrawler* Disallow: /global_engine/ User-agent: *msnbot* Disallow: /global_engine/ User-agent: *scooter* Disallow: /global_engine/