The robots.txt record is basically used to indicate which parts of your site ought to be slithered by insects or web crawlers. It can indicate various principles for various insects.
Googlebot is a case of a creepy crawly. It's conveyed by Google to slither the Internet and record data about sites so it realizes how high to rank various sites in query items.
Utilizing a robots.txt document with your site is a web standard. Arachnids search for the robots.txt document in the host registry (or primary organizer) of your site. This content document is constantly named "robots.txt". You can discover your robots.txt document by going to:
yourwebsite.com/robots.txt
Most standard insects conform to headings indicated in robots.txt records yet loathsome creepy crawlies may not. The substance inside robot.txt documents are freely accessible. You can endeavor to boycott undesirable arachnids by altering the .htaccess document related with your site.
It's significant that advertisers check their robots.txt document to ensure web search tools are welcome to creep significant pages. On the off chance that you ask web search tools to not slither your site, at that point your site won't show up in query items.
You can likewise utilize the robots.txt document to show arachnids where to discover a sitemap of your site, which can make your substance increasingly discoverable.
You can likewise indicate a slither postponement, or how long robots should hold up before gathering more data. A few sites may need to utilize this setting if bots are gobbling up transfer speed and making your site load more slow for human guests.
An Example Robots.txt File
Here is the thing that may show up in a robots.txt document:
User-agent: *
Disallow:/ebooks/*.pdf
Disallow:/staging/
User-agent: Googlebot-Image
Disallow:/images/
Here is the thing that each line implies in plain English.
User-agent: * — The principal line is clarifying that the principles that follow ought to be trailed by all web crawlers. The reference mark implies all bugs in this specific circumstance.
Disallow:/ebooks/*.pdf — related to the main line, this connection implies that all web crawlers ought not creep any pdf documents in the ebooks envelope inside this site. This implies web search tools wo exclude these direct PDF joins in list items.
Disallow:/staging/ — related to the main line, this line asks all crawlers not to slither anything in the staging organizer of the site. This can be useful in case you're running a test and don't need the organized substance to show up in query items.
User-agent: Googlebot-Image — This clarifies the principles that follow should just be trailed by only one explicit crawler, the Google Image crawler. Every arachnid utilizes an alternate "user-agent" name.
Disallow:/images/ — related to the line quickly over this one, this asks the Google Images crawler not to slither any images in the images organizer.
.png)