Categories: How to

How to Write Robots.txt for Your Website?

To exclude all robots from the entire server
User-agent: *
Disallow: /

To allow all robots complete access
User-agent: *
Disallow:

(or just create an empty “/robots.txt” file, or don’t use one at all)

To exclude all robots from part of the server
User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/
To exclude a single robot
User-agent: BadBot
Disallow: /
To allow a single robot
User-agent: Google
Disallow:

User-agent: *
Disallow: /
To exclude all files except one

This is currently a bit awkward, as there is no “Allow” field. The easy way is to put all files to be disallowed into a separate directory, say “stuff”, and leave the one file in the level above this directory:

User-agent: *
Disallow: /~joe/stuff/

Alternatively you can explicitly disallow all disallowed pages:

User-agent: *
Disallow: /~joe/junk.html
Disallow: /~joe/foo.html
Disallow: /~joe/bar.html

Comments

comments

Shayed

Share
Published by
Shayed

Recent Posts

Step-by-Step Guide to Setting Up a Secure Private Cloud Server for Your Startup

If you’re ready to take control of your organization’s data by setting up a private… Read More

3 days ago

Step-by-Step Guide to Setting Up a Secure Private Cloud Server for Your Organization.

Building a private cloud involves creating a virtualized environment where you can manage, store, and… Read More

3 days ago

Flex AI: The Future of Flexible Artificial Intelligence

In the rapidly evolving landscape of artificial intelligence, Flex AI stands as a transformative force,… Read More

1 month ago

iPhone 16 Pro and Pro Max: A Comprehensive Overview of Apple’s Latest Flagships

Apple is set to once again make waves in the smartphone market with the iPhone… Read More

2 months ago

How to Save a Water Damaged iPhone

Act quickly! The sooner you take action, the better your chances of saving your water… Read More

3 months ago

Exciting Innovations in Electric Vehicles for 2024

Introduction The electric vehicle (EV) market continues to grow rapidly, driven by technological advancements and… Read More

5 months ago

This website uses cookies.