How to create robots.txt file for Google demonstrates the simplicity of creating a robots.txt file for Google. Created by https://goo.gl/4UFEae seo specialist this video tutorial session shows quick and easy to understand methods for creating a robots.txt file and also explains how to use it for giving directives to user-agents like Googlebot. You can learn more about specification for this file here:
https://developers.google.com/webmast...
I strongly encourage you to include the name of the Google search engine's user-agent when using this file. If for web search results (typical) specify the name of user-agent as "Googlebot" this will ensure that Google does in fact obey your directive. Typically you would use this file to block certain portions of your website (for example: private folder, or, sensitive file/s) that being the case we know that Google will fetch this document first, and thus, we can tell it to go and visit our sitemap for efficient crawl process. To learn more about robots.txt file simply visit:
https://support.google.com/webmasters...