Categories: How to

How to Write Robots.txt for Your Website?

To exclude all robots from the entire server
User-agent: *
Disallow: /

To allow all robots complete access
User-agent: *
Disallow:

(or just create an empty “/robots.txt” file, or don’t use one at all)

To exclude all robots from part of the server
User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/
To exclude a single robot
User-agent: BadBot
Disallow: /
To allow a single robot
User-agent: Google
Disallow:

User-agent: *
Disallow: /
To exclude all files except one

This is currently a bit awkward, as there is no “Allow” field. The easy way is to put all files to be disallowed into a separate directory, say “stuff”, and leave the one file in the level above this directory:

User-agent: *
Disallow: /~joe/stuff/

Alternatively you can explicitly disallow all disallowed pages:

User-agent: *
Disallow: /~joe/junk.html
Disallow: /~joe/foo.html
Disallow: /~joe/bar.html

Comments

comments

Shayed

Share
Published by
Shayed

Recent Posts

How to Create a Winning Facebook Ad Campaign for Your Business

Introduction Creating a successful Facebook ad campaign is essential for small and medium-sized businesses looking… Read More

6 days ago

Why Small Businesses Need Facebook Marketing in 2025

Introduction In 2025, Facebook remains one of the most powerful platforms for small and medium-sized… Read More

7 days ago

Summary of Key Trading Rules Followed by Expert Pro Traders

Expert traders have honed their craft over years of market observation, trial, and error. Their… Read More

7 days ago

Master Google Ads: Top 10 PPC Strategies for 2025 Success

Overview As we step into 2025, mastering Google Ads PPC strategies is essential for businesses… Read More

2 months ago

How TF-IDF Impacts SEO?

Introduction to TF-IDF: A Beginner's Guide with Real-World Examples Search engines like Google aim to… Read More

2 months ago

How to Save $500 a Year with These Energy-Efficient Smart Devices

Introduction In today’s world, rising energy costs are a concern for many households. But what… Read More

2 months ago

This website uses cookies.