Creating Robots.txt File in a WordPress Blog

Robots.txt is a way through which search engines can identify which pages of your blog are allowed to be spidered or indexed. It is different from meta name=”robots” tag, which tell search engines or robots only about those individual pages on which these tags are placed. There is a huge difference between these two.

You can find tons of information about the robots.txt file at RobotsTXT.Org, if interested or else I’ll be explaining it here in simple terms.

General rule of thumb is to have your robots.txt file at the root of your domain i.e., domainname.com/robots.txt.

Example of a Robots.txt File

A perfect example of WordPress blog robots.txt file is as below

sitemap: http://www.9to5blogger.com/sitemap.xml
User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-content/
Disallow: /wp-includes/
Disallow: /go/
Disallow: /comments/feed/
Disallow: /trackback/
Disallow: /index.php
Disallow: /xmlrpc.php
Disallow: *?wptheme
Disallow: ?comments=*
Disallow: /search?
Disallow: /?p=*
Disallow: /ko/
Disallow: /recommended/

User-agent: Mediapartners-Google*
Allow: /

User-agent: Googlebot-Image
Allow: /wp-content/uploads/

User-agent: Adsbot-Google
Allow: /

User-agent: Googlebot-Mobile
Allow: /

You can generate this robots.txt file using a notepad or by using Meta Robots WordPress plugin. In case you wanted to create robots.txt file manually, use the Robots generator tool.

Check What Matt Cutts is Saying about Robots.Txt File

Last Updated on by Radha Krishna under Wordpress Category 1 Comment

Tags:

 

Share This Article

Get Free Email Alerts

Stay Updated with Useful Blogging Tips. Get Free Email Newsletter Updates..

And, Next Confirm Your Email Subcription

   

One Comment to “Creating Robots.txt File in a WordPress Blog”

  1. Denisa says:

    Thx for this tutorial.

Leave your comment here:

CommentLuv badge

Unable to load the Are You a Human PlayThru™. Please contact the site owner to report the problem.

 
We Use & Recommend
[Bluehost - Review | Installation | Discount ] Bluehost Coupon
Free Email Alerts
Facebook Twitter Google Plus RSS Feed Email
Rate Us

Like US