Find “robot.txt” Files [read for more info]
September 20th, 2006
The robots.txt file serves as a set of instructions for web crawlers. The “disallow” tag tells a web crawler where NOT to look, for whatever reason. Hackers will always go to those places first!
"robots.txt" "Disallow:" filetype:txt
