Creating and Using a robots.txt File

Because robots.txt file on your website, you can expect, you can see the results in this article, a number of tools that can help you is essential.

What is robots.txt file?
Page robots.txt file server, you tell me the section of the site being crawled and indexed by search robots can be. If you, or index, an index for a search engine are registered, it's a different matter, one that the index can be closed, to prevent parts of your site's direction.
A simple text file in Notepad file itself can be made. Here's a folder you need for your home or index page for the site is stored in the root directory.
Why?
All search engines, or at least all major, now a spider or a bot on your site right now is obtained by robots.txt. Because the current robots.txt file, even if the robot is not part of the field is still a good idea, an invitation to your site can look at as a problem.
If not, their position, number or website as all spiders.
If you search on a particular page or an incomplete one site to create
I is important for all of it without adequate privacy password and knows when to protect bored, do not appear in search engines.
Most people are not included - if, for example, you really need something in the library cgi - bin directory recommended? One or two folders, thank you or error page.
Case, each search engine (same page) on the main page, and is used to configure the robot to provide all answers. Spam is important to prevent the formation of parties to be punished in many search engines.
Some robots or spiders, search engine submit email addresses collected the main purpose.
It is true that searching for them on sites I am looking for is a good reason. In those days, you can get access to site statistics? If your data "," No, he is the head file, forget it here on your site, many search robots do not take admission in favor of the robots.txt files mentioned will not help.
Robots.txt File
What is the main effort by robots.txt. This is what is your favorite word processing would be created in Notepad. Each topic has two lines:
User Agent: tattoo BOT scheme method name Disclaimer: file or directory
This line is in a folder, or replicated outside of a spider can, or you want to Bottofairu.
Some examples.
1 clarity. People of different files search engine
You, "is, Google and many works have one index file, folder name, privatefile.htm to make. If I have a hotel and a spider that Google is Google." These lines, robots.txt file is associated with:
User Agent: Robot Google Liability: / private / privatefile.htm
Let robot, spiders removed from all parts of the Sites
Newsection 'site for new category in the list did not, they are ready can be constructed, if any, will be rejected. Robot all use, if you want to see is', '*' wildcards only.
User Agent: responsibility: newsection
At the end of folder name begins with a slash, it is a form, which means that all files in the folder mentioned use.
Three. All spiders, all indications Nomimashou
Again, you can use all boilerplate except spiders. Secondly, in a vacuum, Safe Harbor, which easily leave blank lines.
User Agent: responsibility
Four. Spiders crawl your site in many
Above command, a small change - Beware!
User Agent: rejected
Living on your site and a site to remember next time you use the command!