Importance of Robots.txt File

Even some good ranking sites wont be having robots.txt file. Is it important to have robots.txt file. Or does it any way related to ranking? Here is a detailed analysis on why to have a robots.txt file on your website.

What’s the need of robots.txt file:

It's there for you to tell the spiders / crawlers shouldn’t crawl / spider. If you have pages you don't want to be indexed, use it to stop that happening. That saves bandwidth. Lack of a robots.txt file will mean that the spiders will try and index everything your website have. (Mean all the pages including admin folder etc). Similarly, if there are search engines or other automated processes that you don't want indexing your site (like the way back machine for example) then robots.txt is useful.

How to I create robots.txt file:

It’s a plain text file that need to be uploaded in the root directory of your website. The easiest way to create a robots.txt file is to use the robots.txt generator tool in Webmaster Tools. If you want to create the file yourself, you can use any text editor. It should be an ASCII-encoded text file, not an HTML file. The filename should be lowercase.

SyntaxThe simplest robots.txt file uses two rules:
User-agent: the robot the following rule applies to
Disallow: the pages you want to block

These two lines are considered a single entry in the file. You can include as many entries as you want. You can include multiple Disallow lines and multiple user-agents in one entry. If you would like any additional information, look for the following post Robots.txt tips and tricks.