# Dork: inurl:"robots" | "robot" intext:"Disallow:" | "Allow:" ext:txt
# Files Containing Juicy Info
# Date: 15/09/2021
# Exploit Author: Aftab Alam
Description: This Dork shows all web pages that have a publicly disclosed “robots.txt” file, which contains a list of pages on the particular web server that should not be crawled to be indexed by search engines. By having access to this file, someone could possibly:
1. Know the pages that the web server owner intends to hide from search engine results
2. Know the pages that exist on the web server and are poorly hidden using this technique
3. Gain access to pages with privileged login portals (administrator, webmaster, etc.)
