xctf_Robots

Robots
[Target]
Robots Protocol
[Environment]
Windows
[Tools]
Browser
[Analysis]
1. According to the topic, you can see the robots protocol, so the first thing that comes to mind is robots.txt

2. Disallow can see that this is a crawler protocol:
User— Agent: Define the search engine type (define which search engine the robots are effective for)
User—Agent: BaiduSpider (define that the protocol is only effective for Baidu search engines)
User—Agent: * (define the protocol that the protocol is effective for all search engines)
Disallow: Definition Search engines prohibit crawling of the included address
Disallow:/ (representing that the entire site is prohibited from crawling by search engines, / represents the root directory of the site), if the entire site is prohibited from crawling by search engines in the early stage, the spiders will not be used for a long time Come to the website to fetch content.
Disallow: /data/ (represents the prohibition of spiders from crawling all pages after the data directory)
Disallow: /d (represents the webpage behind the data file is blocked)
3. Search for this php file to

get the flag