How to Develop Your Own Vulnerability Scanning Tool

The core of the vulnerability scanning tool is the scanner, and the design idea of ​​the scanner is: flexible, easy to expand, and easy to modify. Flexible means that it can perform specific vulnerability scanning alone, and can also execute all integrated vulnerability detection modules in batches; easy to expand It means that the new vulnerability detection module can be clearly and simply integrated into the scanner; it is easy to modify, and the detection logic can be modified for each vulnerability scanning module according to special circumstances. Below we will use an open source scanning tool on the Internet to explain. Of course, there may be pits in the deployment process and use, as long as they are solved one by one:

Scanner source code

Scanner download address:  https://gitee.com/samllpig/SafeTool-51testing

Detailed installation tutorial of the tool:  http://quan.51testing.com/pcQuan/lecture/117

Software Architecture Diagram

Install and deploy

• Install python 3.6 or above environment (if you are afraid of trouble, install 3.6, because installing 3.8 and other higher versions still needs to adjust a little code compatibility problem, but there are not many problems)
• install redis
• Install wxPython==4.0.7

pip install wxPython==4.0.7

#If the installation fails, execute a few more edits, mainly because the installation fails due to network timeout
#You can also directly download the relevant module package, such as downloading wxPython-4.0.7-cp36-cp36m-win_amd64.whl, go to the official website to download, be sure to Download the package that matches the python version #Install the
specified module by installing the package

pip install wxPython-4.0.7-cp36-cp36m-win_amd64.whl

#If the installation process prompts what package is missing, continue to download what package 

 You can also install a higher version, such as wxPython==4.1.1

• install openssl

There will be problems in general installation, you can download it directly from the official website  http://slproweb.com/products/Win32OpenSSL.html

After local installation, you also need to copy the library files to the python directory so that they can be recognized, such as:

1. Copy the openssl installation directory: C:\Program Files\OpenSSL-Win64\lib to D:\Python\libs

2. Copy the openssl installation directory: C:\Program Files\OpenSSL-Win64\include to d:\Python\include

• Check requirements.txt

In the root directory of the source code of this file, the modules and versions that need to be installed are configured. We need to confirm whether our python version matches it. For example, Python3.8 requires changing lxml to lxml==4.6.3

• pip install module package

pip install -r requirements.txt

# 如果安装失败，多执行几编，主要是因为网络超时导致安装失败，也可以到官网找模块包下载后来安装
# 如果安装提示版本问题，就需要替换版本，一般Python3.8会遇到

• Start the services in the following order

Start the redis database
Start the server myproxy.bat
Start the client python consoleMain.py 

abnormal modification

After installation and deployment, we may still encounter some problems, which are related to the inconsistency of the original development environment of the code.

1. Compatibility between Python 3.8 and 3.6

Python3.8 has removed the time.clock() method, but it is still used in this source code, so if you encounter related errors, you need to modify it manually, such as to get the system time, you can use time.perf_counter() instead

2. The problem of paths with spaces

This code has not been fully considered in this regard. If the deployment path has spaces, it will report a path error, such as the "D:\Program Files" path. We either don't deploy it in the path with spaces, or directly change his code. ,for example:

setUp = "python " + path
#可以将path路径用引号全圈起来
setUp = "python '" + path + "'"

 3. wt.exe cannot be found error at startup time

I don't know the origin of wt.exe. This misunderstanding sometimes does not affect the startup, because the startup file consoleMain.py has made relevant judgments, but in order not to call wt.exe at all, we can also put the relevant information in consoleMain.py The code is directly changed:

# wtSetUp = "wt.exe python " + path
# 把以上调用改为直接调PowerShell.exe
wtSetUp = "start cmd /k PowerShell.exe python '" + path + "'"

After this change, there is another advantage, that is, the cmd window will not be closed immediately when the code is abnormal, so that you can see the following specific errors, which is helpful for debugging and analyzing code errors.

 4. Chinese report utf-8 encoding error

For example, when executing exec audit during web scanning, an error is reported:

By reporting the error, we can see the utf-8 encoding error, and the error is the resp.read().decode method of request.py. We change the encoding to the following (specifically what encoding to change, you can try it yourself):

self.content = resp.read().decode('gbk')

After the encoding is successfully changed, there is no such error when calling through the python consoleMain.py entry.

5. Report list index out of range error

This low-level error is usually caused by improper use, but it also shows that the robustness of the code is insufficient. For example, the execution of exec attacks.xss reports the following error:

 We can see the specific line of the params.py file that reported the error, which is self.url.split("?"). The problem lies in this question mark, because I started to set the scan path to: set url http://172.16.1.63, This is wrong and should be set as follows:

set url http://172.16.1.63/?u=admin

It can be seen at a glance, you have to add a question mark to indicate that there are parameters. You can directly change his code for this problem, and add a judgment. If there are no parameters, you will be prompted to reset instead of reporting an error. In addition, this section also shows that further development is needed. Normally, scanning tools should actively scan and capture links and places that can perform XSS cross-site scripting attacks, instead of manually setting URLs to simulate attacks.

6. Execute myproxy.bat and cannot find mitmdump

Obviously we have installed mitmdump, why can't we find it? This is because when pip is installed, it is installed locally or globally. Normally, mitmdump is installed in the Scripts in the python directory, such as D:\Tools\Python\Scripts. If If you don't find it, then you need to search in the code path, whether it is installed in the code directory, for example: SafeTool-51testing\venv\Scripts

This is because we use the PyCharm Community Edition development tool to install and deploy possible problems. We either move mitmdump, or directly change myproxy.bat and change the calling path:

"D:\Program Files\Project\SafeTool-51testing\venv\Scripts\mitmdump" -q -s myproxy.py -p 8000

7. Problems with generating reports

There is also a problem with this piece of open source code, and a simple modification method will also be mentioned below.

Simple to use

First open our scanner to see the interface:

 Scanner interface:

Well, the above is our scanner, all of which are operated using commands, which are easy to remember, and you can just follow along.

• help command use

help: list integrated plugin commands and descriptions.

Before we start scanning, we need to do some basic settings. Enter the help set command to see which parameters we need to set. The ones marked with * are required options.

• info command

info : Displays detailed vulnerability detection modules

The usage format of the info command is: info [plugin name], and the plugin name is obtained by entering the help command in the above figure.

Enter info attacks:

Enter the info audit command:

It can be seen that there are too few detection modules under the audit plugin. Later, we need to add scanning modules such as tomcat, nginx, weblogic, etc.

Enter the info brute command:

Enter info disclosure:

• set command:

Before scanning, you need to perform basic settings. Use the help set command to view the options that can be set. If you set it through the set command, the options with * in the options displayed in the help set are required options, and the others are set as needed.

set command format: set [options] [parameters]

Example: set url  http://192.168.16.132/wordpress/?s=11

• check command:

Enter check argv to display the parameter value set in set

The agent parameter, that is, the version information, will be randomly obtained by default.

• exec command

Execute detection command, command format: exec plugin name[.module name]

If you only enter the plugin name [attacks,audit,brute,disclosure], all vulnerability detection modules under the current plugin will be executed. If you enter the plugin name.module name, for example: attacks.xss, only the modules specified under the current plugin will be executed. name.

Enter exec audit : execute all modules under the detection middleware plugin

Enter exec attacks.xss: Use the Detect xss vulnerability module in the injection plugin:

Enter exec attacks.blindsqli : Detect blind SQL injection vulnerabilities

• report command:

Generate test report command

Command parameters: report [report name]

Note: The report can only be generated if at least one complete plugin detection is performed, not the special vulnerability detection, that is, the exec plugin name, not the exec plugin name. Module name

Enter the command: report webscan

Open the test report according to the directory displayed by the prompt. The report format is html:

Well, the above is all the commands and the complete execution process of using the scanner (this report executes two modules (plug-ins) of audit and attacks, but the results are classified as the first plug-in module for the following reasons).

Note: Regarding generating the report, the original code should be problematic. If only the plug-in submodule is executed, such as exec attacks.xss, and then report webscan is executed to generate the report, it will fail. This is because the code only executes the plug-in. The total module exec attacks will call the report assignment (the function that calls the submodule is startup_spec_attacks, and the report value is not reassigned, so the exec submodule will cause the report data to be merged into the previous plug-in general module), see the code as follows:

#执行attacks子模块函数，没有对REPORT进行再赋值
def startup_spec_attacks(attack:str):
    if attack in attacks_info.keys():
        plugins = spec_attacks_plugins(attack)
        startup_plugins(plugins)
    else:
        warn("模块不存在!")

def startup_full_attacks():
    global resultJson
    if not REPORT['startTime']:
        REPORT['startTime'] = strftime("%Y/%m/%d at %H:%M:%S")
    execmod.append("attacks")
    plugins = attacks_plugins()
    if resultJson:
        resultJson = {}
    startup_plugins(plugins)
    REPORT['attacks'] = resultJson  #调用总的attacks模块，才对报告结果赋值

This problem needs attention. If you want to change it simply, add the REPORT assignment directly to the startup_spec_attacks function (requires repeated judgment):

def startup_spec_attacks(attack:str):
    global resultJson
    if not REPORT['startTime']:
        REPORT['startTime'] = strftime("%Y/%m/%d at %H:%M:%S")
    if resultJson and not REPORT['attacks']:
        resultJson = {}
    if attack in attacks_info.keys():
        execmod.append("attacks")
        plugins = spec_attacks_plugins(attack)
        startup_plugins(plugins)
        REPORT['attacks'] = resultJson
    else:
        warn("模块不存在!")

After such a change, and then performing the above scanning steps, the report template is more accurate, and even if the attacks.xss and attacks.blindsqli of the same module are executed, the latter will not cover the former, as follows:

Of course, this does not guarantee that there will be no problems at all. I will carefully consider the optimization in this area in the future!

Writing plugins

Everyone click the download address in Chapter 1. After downloading the tool, open it with PyCharm or vscode or your handy tool. The plugin scanner is in the scan directory.

Our plugin writing starts with the scan\lib\utils\settings.py global configuration file

The first step: first look at the basic path configuration, the directory structure of the entire project is here

The second step is to configure the dictionary path. The dictionary needed by the vulnerability detection module is placed under this path.

Step 3: Configure the plug-in path, and the new plug-ins are uniformly configured in the following format

Step 4: Configure the plug-in description information, which corresponds to the plug-in path in the third step

Step 5: Configure the vulnerability module description information, which corresponds to the plug-in description in Step 4

Step 6: Configure the vulnerability module path information, which corresponds to the plug-in description in Step 4

Step 7: Write the load module method of the plug-in

Unified naming rules Plugins are named _plugins() [all module execution methods]; spec_plugin name_plugins(key:str)

Step 8: Write a detection vulnerability module, taking the bshi (shell-breaking vulnerability detection) detection module under the attacks plugin as an example,

Import the necessary core libraries:

Create a vulnerability detection class, inherit the Request class, and the class name must be the same as the file name

The vulnerability detection method written is uniformly named check:

Finally, write the run method. It is not recommended to modify the name. If you really need to modify it, you need to modify the corresponding module import logic in the ninth step:

Step 9: Write a local API call under the localapi.py file. Why is there a local API call? Because I plan to write a remote API call method to be used in conjunction with the web service of the security tool (this step is not necessary, only after the extension If you add a new vulnerability detection module based on the original plugin, you can skip this step)

The format and basic path of the local API function, refer to the following figure:

To sum up, the extension logic is that named plug-ins facilitate unified management of modules, modules are written in a fixed format, and local APIs are written to run detection modules according to specified plug-ins. The detection module of the scanner can continue to be updated and extended.

Since I have learned Python syntax for half a day and deployed applications in a few days, I still need to continue to understand and learn. The languages ​​are all the same, and those who know Java can learn Python quickly. Learning Python well is conducive to the expansion of this open source tool. and development.

Reference: https://zhuanlan.zhihu.com/p/368859499