How to Write Robots.txt for Your Website?
To exclude all robots from the entire server User-agent: * Disallow: / To allow all robots complete access User-agent: * Disallow: (or just create an empty “/robots.txt” file, or don’t use one at all) To exclude all robots from part of the server User-agent: * Disallow: /cgi-bin/ Disallow: /tmp/ Disallow: /junk/ To exclude a single … Read more