What What this means is is that the robots.txt instructs the search engine spiders, robots which parts of a web site they need to not pay a visit to. A straightforward, user friendly robots txt generator can be used to put these instructions in an internet site. Find the robots.txt https://rankingseotoolz.com/robots-txt-generator