Test point: robots file
There’s nothing to say, it just depends on whether you know the robots file.
Baidu Encyclopedia: Robots is an agreement between the website and the crawler. It uses a simple and direct txt format text to tell the corresponding crawler the permissions allowed. In other words, robots.txt is the first file to be viewed when accessing the website in the search engine. . When a search spider visits a site, it will first check whether robots.txt exists in the root directory of the site. If it exists, the search robot will determine the scope of access based on the contents of the file; if the file does not exist, all The search spiders will be able to access all pages on the website that are not password protected.
Append botos.txt after the url, as shown below:
The fl0g.php page will not be crawled by crawlers. direct interview.
