How to Write Robots.txt for Your Website?

To exclude all robots from the entire server

To allow all robots complete access

(or just create an empty “/robots.txt” file, or don’t use one at all)

To exclude all robots from part of the server

To exclude a single robot

To allow a single robot

To exclude all files except one

This is currently a bit awkward, as there is no “Allow” field. The easy way is to put all files to be disallowed into a separate directory, say “stuff”, and leave the one file in the level above this directory:

Alternatively you can explicitly disallow all disallowed pages: