Creating Robots.txt File in a WordPress Blog

Robots.txt is a way through which search engines can identify which pages of your blog are allowed to be spidered or indexed. It is different from meta name=”robots” tag, which tell search engines or robots only about those individual pages on which these tags are placed. There is a huge difference between these two.

You can find tons of information about the robots.txt file at RobotsTXT.Org, if interested or else I’ll be explaining it here in simple terms.

General rule of thumb is to have your robots.txt file at the root of your domain i.e., domainname.com/robots.txt.

Example of a Robots.txt File

A perfect example of WordPress blog robots.txt file is as below

sitemap: http://www.9to5blogger.com/sitemap.xml
User-agent: *
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-content/
Disallow: /wp-includes/
Disallow: /go/
Disallow: /comments/feed/
Disallow: /trackback/
Disallow: /index.php
Disallow: /xmlrpc.php
Disallow: *?wptheme
Disallow: ?comments=*
Disallow: /search?
Disallow: /?p=*
Disallow: /ko/
Disallow: /recommended/

User-agent: Mediapartners-Google*
Allow: /

User-agent: Googlebot-Image
Allow: /wp-content/uploads/

User-agent: Adsbot-Google
Allow: /

User-agent: Googlebot-Mobile
Allow: /

You can generate this robots.txt file using a notepad or by using Meta Robots WordPress plugin. In case you wanted to create robots.txt file manually, use the Robots generator tool.

Check What Matt Cutts is Saying about Robots.Txt File

1 Comment

Leave a Reply

Your email address will not be published.


*


Unable to load the Are You a Human PlayThru™. Please contact the site owner to report the problem.