Submitted by Stefan on Sat, 07/05/2014 - 3:26pm

Commands in a robots.txt file are requests, not commands, to web crawlers, which they hopefully choose to honor.

However malicious crawlers will actually look for the locations of hidden content in robots.txt - which they would otherwise not have discovered if the robots.txt file was not present.

That is why I believe it is better not to specify the location of hidden pages using robots.txt, but instead with a meta robots tag with a noindex value (or other solutions f you have access to server configuration.)

The content of this field is kept private and will not be shown publicly.
About text formats

Plain text

  • Lines and paragraphs break automatically.
  • Web page addresses and email addresses turn into links automatically.