How to Use Robots txt for SEO

Whenever we go on exploring a new place then we need guidance for convenience and save time! The same way the web robots like that of any search engine take the help of Robots.txt file to get an idea about how to crawl pages of a particular website.

By the way, the behaviour of such crawlers to move all over the internet, access, index, and serve the content to the targeted users are based on a group of web standards known as REP or Robots exclusion protocol that includes robots.txt as well.

In a simple way, we can understand and remember robots.txt as a mix of two terms Robot and Txt. So, it is a txt or text file that is intended to be used by the web Robots possible that of the search engines.

It can also help webmasters if the website to control the crawling behaviour of a user agent but it has to be done carefully, since disallowing the important or all pages of your site from a search engine like Google can be highly dangerous.

The webmasters of a website can use robots.txt to instruct the web-crawling software or user agents to what all parts to crawl and what not of the site. It can be done by using “allow” or “disallow” instructions inside the robots.txt file for some or all crawler user agents

A search engine is mainly responsible for two main jobs to get its job done. First one is to discover the content from the web by crawling everywhere and indexing the updates. The next job is to look for the related information into its indexed directory to serve the right content as per a search query.

So, Robots txt what is it?

Search engines follow the links and go from one website to another, the process is also called as “spidering”. Whenever the bot or web crawler reaches a new website then before start spidering the same it first looks for the robots.txt file. If it get one then it will read it to gain information about how to crawl the website, especially what to access and what not! In case of absence of robots.txt file, the user-agents can start crawling the other information available on the website.

The file should consist of at least the following two elements;

User-agent: (Name of the user-agent)

Disallow: (URL string that must not be crawled)

Together the above two lines can be considered as a discrete set of user-agent directives and are separated from other sets using a line break (/).

If a single rule is specified in the file for more than one user-agents then the crawler will first read and follow the directives that are mentioned in a separate group of instructions.

Anyone can look at the content of the robots.txt present on a website by simply using the browser method.

You need to add robots.txt after the main URL like https://demo.com/robots.txt or its subdomain like https://shop.demo.com/robots.txt.

It is mandatory that the robots.txt file should be available after the root domain. So, you can mention the same on the browser.

If you will not find any .txt page on the output then this means there is no (live) robots.txt page currently present on the website.

There should be separate robots.txt files for the root domain(demo.com/robots.txt) and its every subdomain(blog.demo.com/robots.txt).

All the instructions present in the file must be read from top to the bottom either by a human or any software bot! It can be possible that a robot or a user agent would not read the robots.txt file of a website. It is usually possible with email address scrapers or malware robots type of nefarious crawlers.

There are many advantages of using robots.txt in a website. Such as;

– To ask search engines to do not index certain files like PDFs, images, etc. on your website. Meta directives can also use as an alternative to robots.txt to avoid indexing of the pages but do not work for resource files.

– A webmaster can ensure efficient crawling of a website by providing helpful tips to its bots.

– To avoid search engines to show any internal search results page on the public SERP.

– By blocking certain unimportant or unnecessary pages of the website you can maximize your crawl budget on the required pages.

– To be used like meta-robots to avoid duplicate content to be displayed in SERPs.

– You can take its help to not index the internal search results or broken web pages of your website.

– To prevent overloading of the web servers that is possible when crawlers load multiple contents at a time by adding some crawl delay.

– If you do not want people to land on any page that is at its staging version that can impact the impression especially of a first-time visitor of a website.

– To help user agents easily access the location of the sitemap(s).

A webmaster can keep a particular section of a website (especially under construction or incomplete ones) completely private from the crawling bots.

It is necessary to create the robots.txt file if the number of indexed URLs goes more than expectations.

It is best to use any word editor like notepad or wordpad to create a simple text file compatible with the rules to make a robots.txt.

Just include the basic directives like “User agent:” and “Disallow: /” to create a basic file for the website.

Anyone can include the rules by following the compatible syntax inside the robots.txt file.

The best way is to first generate the sitemaps of your website and include its URLs at the bottom to make it more effective.

The common terms that are used inside a robots.txt file are:

– Crawl-delay – It indicates for how much time do a specified crawler needs to wait before accessing the content of a page. The command will not work for the Googlebot, however the crawl rate can be set from the Google Search Console to get the same job done.

– User-agent – It mentions a specific web crawler or the user agent (generally a search engine) to which a webmaster wants to give crawl instructions. There are technical names for search engines like Googlebot for Google and so on.

– Allow (used by Google) – It is a useful syntax to instruct the Googlebot to crawl a subfolder or a page that is present inside any parent subfolder or a page that might be disallowed.

– Disallow – It is to instruct a web bot to not access any specific URL. The command should not be allowed twice for any URL.

-Sitemap – Any compatible user-agent like Yahoo, Ask, Bing, or Google can access this command to find the location of the mentioned XML sitemaps based on a URL.

Note: The regular expressions like a dollar sign ($) and asterisk (*) can be used by SEO to help user agents of Bing and Google in identifying the subfolders or pages. Here * is the pattern-matching syntax to cover all the kind of possible URL ending options and * is to represent a different sequence of characters, working as a simple wildcard.

It can be done by blocking or disallowing the web bots by specifying the directives for each or all to not access a page or subfolder of a website.

Here are some directives commonly used in the robots.txt file to instruct their user-agents or web crawlers;

1) Allowing every web crawler to find all the content

Syntax: User-agent: * Disallow:

2) Disallowing a particular web crawler to access a folder

Syntax: User-agent: Googlebot Disallow: /extra-subfolder/

(Above instruction is asking the Google’s crawler to do not access any pages of the location www.site-name.com/extra-subfolder/)

3) Disallowing all the web crawlers to access any content

Syntax: User-agent: * Disallow: /

(You can use the simple instruction as a solution to How to block bots Robots txt?)

4) Disallowing a particular web crawler to access a specific web page

Syntax: User-agent: Googlebot Disallow: /extra-subfolder/useless-page.html

The popular search engine uses many spider software that revolves all over the web and scans the websites. The highlighted ones are Googlebot, Googlebot-images (used for images), and Googlebot-news (to index and serve the information about the news to the users).

Use a text editor that can create a standard UTF-8 text file. Creating the file using a word processor might add any unexpected character like curly quotes and can save it in any proprietary format that might arise issues for crawlers to understand the instructions. Comments can be added after specifying # character or mark.

Here are some suggestions on creating the file especially for Google user agents;

1) The file should follow the Robots Exclusion Standard.

2) It can include one or more rules for allowing or blocking the access to the specified crawler to a particular path of a site.

3) A webmaster should be familiar with almost all the syntax of robots.txt file to understand the subtle behaviour of each syntax.

4) The site cannot be having more than one robots.txt file.

5) The file supports both subdomains (like http://website.demo.com/robots.txt or any non-standard port like (http://demo:8181/robots.txt).

6) If you do not know or having the access to the root folder of your website then it is best to reach the web hosting service provider to keep the robots.txt file inside the same. In case you can’t access to the website root then use meta tags as alternative blocking method.

7) More than one group directives or rules (mentioned one per line) can be included in the robots.txt file.

8) It supports only ASCII characters.

9) A group provides information about to whom it is applied for (user agent) and what all files or directories that an agent cannot/can access. The directives are processed from top to bottom. A web bot associated itself to only one rule set that can be specified separately or comes first.

10) As per the default assumption a bot can crawl any directory or page by a “Disallow:” syntax.

11) The directives used in the file are case-sensitive, like Disallow: /one.xml doesn’t apply to ONE.xml.

12) It applies to the full domain of a website consisting of either https or http protocol.

Usually, the user agents of Bing and Google go with a specific group of directives but by default, first, matching rules are preferable since different search engine web bots interpret the directives in a different manner.

It is also suggested for webmasters to avoid using the crawl-delay syntax as much as possible in their robots.txt file so to reduce the total crawl time of the search engine bots.

You can take the help of robots.txt Tester tool available on the Google’s webmaster console to check whether the Google’s bot’s are able to crawl the URL that you had already blocked from its Search. It can also show the logic errors and syntax warning if there are any in your robots.txt. You can edit there and retest it.

Once everything is fine, you can cope with the changes and update your main file located on your website server. Similarly, you can use different tools to check in advance the crawling behavior of search engine after reading the robots.txt of your website.

You can also check how the robots.txt in your website is performing by using the ‘Blocked URLs; feature inside the ‘Crawl’ section provided on left section of the Google Webmaster Tools. However, it might not show the current or updated version of robots.txt but can be used for testing purposes.

Try to regularly check your robots.txt file using any tool about whether everything is valid in it and the file is operating in the right manner as expected! By the way, it might take many days or even few weeks for a search engine to identify a disallowed URL by reading about the same from the robots.txt and remove its indexing.

After including all the rule sets in the file and naming it with robots.txt it needs to be saved in the main or root folder of the website in the server. A root level folder can be a “www’ or “htdocs” that helps robots.txt to appear next to your domain name.

It is always suggested to keep a reasonable size of robots.txt by avoiding unwanted directives from mentioning in the file. It is because years before John Mueller of Google has already clarified the fact that Googlebot will only access the first 500kB of a robot.txt file. A giant file can be truncated in an undesired way to form a line that might be interpreted as an incomplete rule.

It is also known as Robots exclusion protocol or robots exclusion standard that is used by websites to communicate with web robots or crawlers. Search engines use their robots to categorize the websites.

Webmasters use robots.txt files to instruct or guide such robots to get better indexing of their websites. You do not need a robots.txt file if you do not want to control user-agent access to any area of your website. One can find more details about robots.txt from any advanced topic like How to Create a Search Engine Bot?

For better search engine rankings, it is a best SEO practice to allow its crawlers to reach and access your site with ease. Our website generally consists of lots of unwanted pages than our expectations, and when search engine bots crawl ever page of your site then it will surely consume more time and this will surely be going to negatively affect its ranking.

Google uses the crawl budget (divided into two parts, crawl rate limit and crawl demand) for every website to decide the number of URLs it wants or can scan. So, if you help such bots or user agents to access and index only the most valuable content of your website robots.txt is a must!

An SEO never wants any sections or content to be blocked of a website that is necessary to be crawled.

– A search engine like Google can have multiple user-agents like Googlebot-Image (to search the images) and Googlebot (for organic search). Many user agents that belongs to the same search engine can follow the same rules so many webmasters skip to specify directives for each of these crawlers. An SEO can take advantage of this by mentioning different instructions to each of the crawlers even if they long to one search engine to better control their crawling behaviour.

– For better SEO it is necessary that the disallowed links or pages must not include any further links that need to be followed. So, the blocked page should not passant link equity to the link destination or it is better to use any other blocking mechanism. They must also not be linked with other pages accessible by the search engines i.e. webpages that are not disallowed by meta robots, robots.txt, or else. Otherwise, the important linked resources will not be accessed and indexed by the search engines.

– It is best to submit the robots.url URL directly on the Google after any updates done on the file to ensure its quick access by the targeted user agent. Generally, a search engine updates the cached robots.txt contents once in a day at least.

It is good to mention the location of all or any sitemaps based on the website’s domain at the bottom part of its robots.txt file. By the way, sitemaps are XML files that contain detailed information about the pages of a website like their URL with the related metadata like its importance, its update interval, and the last update.

All such information can be used by search engine bots to intelligently crawl a website. So, in this way webmasters can help the user agents that support Sitemaps to know and access all the URLs from the sitemap and know more about them in their process of discovering pages from one link to another within one or from another site.

For example;

Browser address: https://www.demo.com/robots.txt

Output:

User-agent: *

Disallow: *.dl.html

Allow: /*.html$

Sitemap: https://www.demo.com/en-au/sitemap.xml

Sitemap: https://www.demo.com/en-se/sitemap.xml

Sitemap: https://www.demo.com/en-us/sitemap.xml

(The above directives are to call more than one sitemaps via robots.txt file.)

There are security risks associated with robots.txt since many malicious bots cannot follow it as well as one can use it to know all the disallowed links and directly access them. So as a solution, you can password protect the area of your website that contains private content so that an intruder can’t access it even after knowing its location.

To present sensitive data from indexing or get appeared in the SERPs (either directly or indirectly i.e. through liked pages) it is best to use any other method than disallowing the same from the robots.txt to block the page. It can be either no index meta directive or password protection methods.

WordPress generally makes a virtual default robots.txt file in the root directly for its websites that can’t be seen on the directory. So, it is always best to create a new file that overlaps with any default settings especially to disallow the login or signup page that doesn’t matter to a search engine!

Many people are usually confused about How to remove Robots txt in WordPress or other platforms. However, the process is the same for all! The robots.txt file needs to be saved on the top-level directory of the website i.e. the root domain or main directory so to help the bots to find it with ease. So, all you need is to delete the file directly from that particular folder or location.

Try not to include the instructions to hide confidential user information inside the robots.txt file. It is because the file is a publically accessible file, one can see its directives by adding /robots.txt at the end of the root domain.

In this manner, anyone can come to know what all pages is allowed by the webmaster of the site to be crawled or not by all or specific web bots. The file must be saved with “robots.txt” name only since it’s case sensitive so no other combination will be accepted by any user agent!

Lastly, you might be confused between x-robots, meta robots, and robots.txt that sounds similar terms. Among them, x-robots and meta are meta directives but robots.txt is a text file and they are used to apply different functions.

To be specific, x-robots and meta are to dictate the indexing behaviour at page element (or individual page) level, whereas robots.txt is to prove information about the director or site-side crawl behaviour.

There are higher chances that the search engine bots can index and display the content of your website on SERPs in the better way and make it more visible by spending its crawl budget well while scanning the same site. By using the robots.txt can also block the crawling of auto-generated WordPress tag pages and prevent any more duplicate content.

Overall, you need to take much care while dealing with what to include in the robots.txt file. After all, a small mistake inside the robots.txt file could make your entire website get deindexed.