User-agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT; MS Search 4.0 Robot) Disallow: / # exclude some access-controlled areas User-agent: * Disallow: /dns/inc/ Disallow: /mail/inc/ Disallow: /web/inc/