Processing of Subdomain Data in Vulnerability Mining

0x01 Preface

Today, I tested the domain name of an e-commerce business with the sub-domain scanner of Master Lijiejie. The result is almost 70,000+ sub-domain names. When I manually visit some websites, I have the following results

This result is caused by the fact that there is no index in the web directory of the tengine server, and this kind of server generally does not have loopholes, there is no web service, and most of the open ports are 80 443 ports, we need to discard

So how to find the open domain name of the web service among so many subdomains, you may need the help of (python)

0x01 80_open_detect_V1.0 write

The original idea was to use a single thread, read the subdomain name in each line, and then use the requests module to access it, and judge whether it is a domain name with services through features.

80_open_detect_V1.0.py

#coding=utf-8
# Author : MrSm1th
import requests 
import re
import os
def main(file,newfile):
    i=0
    j=0
    f = open(file,"r")
    f1 = open(newfile,"a+")
    while 1:
        j+=1
        line = f.readline()
        if not line:
            print "[*]done!Found:   "+str(i)
            break
        url = "http://"+"".join(re.findall("(.*).com",line))+".com"
        print str(j)+":"+url
        try:
            if "403 Forbidden" and "Powered by Tengine" in requests.get(url).content:
                continue
            else:
                i+=1
                print "[*]found:"+url
                f1.write(url+"\n")
        except:
            print "[-]connect fail:"+url
    f1.close()
    f.close()
if __name__ == '__main__':
    file = raw_input("enter file you want to detect:")
    if(os.path.exists(file)==False):
        exit("[-]file not exists")
    newfile = file+"_80_open_detect.txt"
    if(os.path.exists(newfile)==True):
        exit("[-]newfile exists")
    main(file,newfile)

But after testing it found that the script is very slow. Immediately found that you need to run with multi-threading.

0x01 80_open_detect_V2.0 effect

Record it. .