Robots.txt is a way through which search engines can identify which pages of your blog are allowed to be spidered or indexed. It is different from meta name=”robots” tag, which tell search engines or robots only about those individual pages on which these tags are placed. There is a huge difference between these two.
You can find tons of information about the robots.txt file at RobotsTXT.Org, if interested or else I’ll be explaining it here in simple terms.
General rule of thumb is to have your robots.txt file at the root of your domain i.e., domainname.com/robots.txt.
A perfect example of WordPress blog robots.txt file is as below
Check What Matt Cutts is Saying about Robots.Txt File