What is Robots.txt?
Robots.txt is a text file webmaster create to instruct web robots how to crawl pages on their website.
The robots.txt file is a part of the robots exclusion protocol,
a group of web standards that regulate how robots crawl the web, access and index content, and serve that content to visitors.