What is Robots.txt file and How to Check it Correctly

page_watches 295097 page_stars
img-border Robots txt test

Robots.txt file serves to provide valuable data to the search systems scanning the Web. Before examining of the pages of your site, the searching robots perform verification of this file. Due to such procedure, they can enhance the efficiency of scanning. This way you help searching systems to perform the indexation of the most important data on your site first. But this is only possible if you have correctly configured robots.txt.

Just like the directives of robots.txt file generator, the noindex instruction in the meta tag robots is no more than just a recommendation for robots. That is the reason why they cannot guarantee that the closed pages will not be indexed and will not be included in index. Guarantees in this concern are out of place. If you need to close for indexation some part of your site, you can use a password to close the directories.


Table of contents



Main Syntax


User-Agent: the robot to which the following rules will be applied (for example, “Googlebot”)

Disallow: the pages you want to close for access (when beginning every new line you can include a large list of the directives alike)
Every group User-Agent / Disallow should be divided with a blank line. But non-empty strings should not occur within the group (between User-Agent and the last directive Disallow).

Hash mark (#) can be used when needed to leave commentaries in the robots.txt file for the current line. Anything mentioned after the hash mark will be ignored. When you work with robot txt file generator, this comment is applicable both for the whole line and at the end of it after the directives.
Catalogues and file names are sensible of the register: the searching system accepts «Catalog», «catalog», and «CATALOG» as different directives.

Host: is used for Yandex to point out the main mirror site. That is why if you perform 301 redirect per page to stick together two sites, there is no need to repeat the procedure for the file robots.txt (on the duplicate site). Thus, Yandex will detect the mentioned directive on the site which needs to be stuck.

Crawl-delay: you can limit the speed of your site traversing which is of great use in case of high attendance frequency on your site. Such option is enabled due to the protection of robot.txt file generator from additional problems with an extra load of your server caused by the diverse searching systems processing information on the site.

Regular phrases: to provide more flexible settings of directives, you can use two symbols mentioned below:
* (star) – signifies any sequence of symbols,
$ (dollar sign) – stands for the end of the line.


Main examples of robots.txt generator usage


Ban on the entire site indexation

User-agent: *
Disallow: /

This instruction needs to be applied when you create a new site and use subdomains to provide access to it.
Very often when working on a new site, Web developers forget to close some part of the site for indexation and, as a result, index systems process a complete copy of it. If such mistake took place, your master domain needs to undergo 301 redirect per page. Robot.txt generator can be of great use!


The following construction PERMITS to index the entire site:

User-agent: *


Ban on the indexation of particular folder

User-agent: Googlebot
Disallow: /no-index/


Ban on a visit to the page for the certain robot

User-agent: Googlebot
Disallow: /no-index/this-page.html


Ban on the indexation of certain type of files

User-agent: *
Disallow: /*.pdf$


To allow a visit to the determined page for the certain web robot

User-agent: *
Disallow: /no-bots/block-all-bots-except-rogerbot-page.html
User-agent: Yandex
Allow: /no-bots/block-all-bots-except-Yandex-page.html


Website link to sitemap

User-agent: *
Sitemap: http://www.example.com/none-standard-location/sitemap.xml

Peculiarities to take into consideration when using this directive if you are constantly filling your site with unique content:

because a great many unfair webmasters parse the content from other sites but their own and use them for their own projects.


Which is better robots txt generator or noindex?


If you don’t want some pages to undergo indexation, noindex in meta tag robots is more advisable. To implement it, you need to add the following meta tag in the section of your page:

<meta name=”robots” content=”noindex, follow”>

Using this approach, you will:

Robots txt file generator serves better to close such types of pages:


Which tools and how can help you check out the robots.txt file?


When you generate robots.txt, you need to verify if they contain any mistakes. The robots.txt check of the searching systems can help you cope with this task:


Google Webmasters

Sign in to account with the current site confirmed on its platform, pass to Crawl and then to robots.txt Tester.


Robots.txt tester in Google Search Consoleimg-close


This robot txt test allows you to:


Yandex Webmaster

Sign in to account with the current site confirmed on its platform, pass to Tools and then to Robots.txt analysis.


Robobts txt generator for Yandeximg-close


This tester offers almost equal opportunities for verification as the one described above. The difference resides in:

1 Star2 Stars3 Stars4 Stars5 Stars (13 votes, average: 4.92 out of 5)

Interesting now

Google Cache: Practical Guide Of Its Usage

If you have heard about Google Cache but still don`t know to use it efficiently, this guide is totally for you. Here you will find a practical guide on how to view cached pages using Google Cache and

How to Conduct SEO Content Analysis Properly?

Every text on the web is created for some purpose. An effective text is the one that performs the necessary tasks. To achieve your goals, we recommend performing SEO content analysis at the several

E-commerce SEO: Best Practices for Your Store

E-commerce retailers are generally skeptical when they are told that they can double sales and traffic to their site by applying SEO best practices.

How to Use Meta Robots Tag Correctly?

Meta robots tags are used to pass instructions to search engines on how they should index or crawl sections of a website. This article gives an overview of the different values that can be used in a

What Is JSON-LD Markup and Why Is It Better than Schema Markup?

If you understand how to use schema.org, but do not dare to mark pages up because of the complexity of the process, this article is for you. There is an effective and easy-to-use alternative - the

Causes of Duplicate Content and How to Solve the Problem

All search engines, including Google, have problems with duplicate content. When the same content is shown at numerous locations on the internet, a search engine can’t determine which URL should be

Show more


The password link was sent to you by email

The password changed