Apache is prohibited web crawler, set many times before, but always does not work, turned out to be are just plain wrong, and can not be written in Dirctory, Location to be written in SetEnvIfNoCase User-Agent "spider" bad_bot BrowserMatchNoCase bingbot bad_bot BrowserMatchNoCase googlebot bad_bot Order Deny, Allow # the following are prohibited soso reptile Deny from 124.115.4. 124.115.0. 64.69.34.135 216.240.136.125 218.15.197.69 155.69.160.99 58.60.13. 121.14.96. 58.60.14. 58.61.164 . 202.108.7.209 Deny from env = bad_bot SetEnvIfNoCase User-Agent "spider" bad_bot it is banned all reptiles include spider characters. If you want to ban targeted reptiles, reptiles changed to exactly match the string, if bingbot, Googlebot etc.
Reproduced in: https: //my.oschina.net/766/blog/210883