A robots.txt is a simple text file, which contains standard set of instructions for search engine crawlers. It contains inline commands that will request the crawlers to ignore certain directories or files on your web site.

Advantages of using robots.txt file:

• Prevent search engine crawlers from accessing private, confidential and important information.
• Prevent search engine crawlers from consuming unnecessary bandwidth.
• Provides specific file or directory level commands for particular search engine robot

There are number of tools for robots.txt which will validate your file and create them with standard instructions.