1. Background
I often see some powerful bigwigs on SRC and CNVD submit a lot of vulnerabilities. I have always been curious about how they can dig so many holes. At first I thought they dig holes when they are not working but sleeping. Later, I had the opportunity to meet some bigwigs. It is found that most of their vulnerabilities are actually discovered by tools. For example, the following is the white hat boss on CNVD
I want to be a boss how to do
I have always felt that I am a person with dreams, and I also hope that one day my ID will appear in the leaderboard, so I relied on my little development knowledge to study the security tools on the market and how to develop security tools. tool.
Security Tool Analysis
After my research, I found that there are actually only two types of security tools on the market, one is a tool for a certain vulnerability, such as SQLMap, and the other is a comprehensive scanning tool, such as AWVS;
As a person who just wants to dig vulnerabilities, I am more inclined to develop comprehensive scanners, but the development of comprehensive scanners is really difficult. It is necessary to clearly understand the principles of various vulnerabilities, and also need to use their code To achieve it, if I develop it from scratch alone, I can't do it at all.
But I don't intend to give up, I am going to gather the world's most powerful tools for my scanner; the ideal is there, but the reality is how I want to realize it, which really troubles me.
2. Things to do
The core purpose of the scanner I want to make is to be easy to use, and the other thing is that I can modify it as I like; I hope that as long as I give him a URL address, it can help me scan the vulnerabilities of the website and the host itself
After dismantling it carefully, I think the most needed functions are these
- Can automatically collect URL addresses, crawler collection and blasting collection
- Can extract host IP from URL
- Can quickly detect common popular POC
- Can automatically identify the fingerprint information of the website
- Quick port scan for IP
- Ability to identify services for port banners
- Can detect SQL injection vulnerabilities
- Can detect reflective XSS vulnerabilities
- Able to use the corresponding POC tool through fingerprint information
- Can quickly expand functions without affecting the overall logic
The first version is almost all these functions. Although there are not many functions, it will take a lot of development time to implement it completely from scratch.
3. Thinking Analysis
In order to achieve high efficiency while being autonomous and controllable, I decided to be a good sewing man. The simple understanding is that I need to integrate many tools into the tools I develop. The first question to consider here is that every The usage methods, input parameters, and output results of each tool are different, and the result of tool A may not be recognized by tool B.
To solve this problem, it is easy to say and difficult to say. In short, I succeeded in crossing the river by feeling the stones. The principle is to make a shell for each tool. To call tool A externally, you need to call the shell of tool A first, and then Only then will it be transmitted to tool A. When tool A returns the result, the shell of tool A will be obtained first, and then the result will be parsed out and output in a unified format.
Through this simple method, I am equivalent to turning other security tools into a function of mine, and I can call this function when I need it.
According to the requirements I mentioned earlier, I sorted out the following tools to try out:
serial number
| serial number | need | tool |
|---|---|---|
| 1. | Crawl to the URL with | RAD |
| 2. | There are URLs for blasting | DIRMAP |
| 3. | Extract host IP | Regular |
| 4. | Quick detection of popular POC | xray |
| 5. | Fingerprinting a website | dismap |
| 6. | Quick scan of IP ports | masscan |
| 7. | Ability to identify services for port banners | nmap |
| 8. | Can detect SQL injection vulnerabilities | sqlmap |
| 9. | Can detect reflective XSS vulnerabilities | xsser |
These tools are relatively common tools. My first step is to be familiar with their usage. Take the xray tool as an example.
The command to use xray is as follows
./xray_linux_amd64 webscan --url "http://192.168.1.100/" --json-output /tmp/11.json
When xray is executed, it will output the result to the specified location, but the data format is not what I expected, I need to read its format, and then convert it into the format I need.
Here I wrote a simple script in PHP that does the following things:
- Defines the parameter source location and the result output location
- Get the URL in the parameter and execute the xray tool
- Get the execution result of xray and parse it into a custom format
- write the final result to the output location
A code example is shown below
<?php
//获取输入的参数
$inputFile = "/data/share/input_".getenv("xflow_node_id").".json";
$outputFile = "/data/share/output_".getenv("xflow_node_id").".json";
//没有input,直接返回
if (!file_exists($inputFile)) {
var_dump($outputFile, json_encode(['code' => 0, 'msg' => "{$inputFile}文件不存在", 'data' => []], JSON_UNESCAPED_UNICODE));
return 0;
}
//读取上游数据
$inputData = json_decode(file_get_contents($inputFile), true);
$url = $inputData['url'];
$data = execTool($url);
//将结果写入到指定位置,供蜻蜓平台导入数据
file_put_contents($outputFile, json_encode($data, JSON_UNESCAPED_UNICODE));
//将工具执行
function execTool($url)
{
$hash = md5($url);
$resultPath = "/tmp/{$hash}/tool.json";
//清理之上一轮的结果
if (file_exists($resultPath)) unlink($resultPath);
//创建文件夹
if (!file_exists(dirname($resultPath))) {
mkdir(dirname($resultPath), 0777, true);
}
$result = [];
$toolPath = "/data/tools/xray";
if (!file_exists($toolPath)) die("xray 工具目录不存在:{$toolPath}");
$path = "cd $toolPath && ";
// 通过系统命令执行工具
$cmd = "{$path} ./xray_linux_amd64 webscan --url \"{$url}\" --json-output {$resultPath}";
echo $cmd;
exec($cmd, $result);
$toolResult = file_exists($resultPath) ? file_get_contents($resultPath) : '[]';
$toolResult = json_decode($toolResult, true);
print_r($toolResult);
return $toolResult;
}
Let’s take an example of sqlmap encapsulation. First, you need to know how to use sqlmap, as shown below
sqlmap -u "http://192.168.1.100/index.php?id=1" --batch --random-agent
After sqlmap is executed, I need to know where the execution result is, parse the result, and output it to the specified address in a standardized format.
Here I also wrote a script in PHP and did the following things:
- Defines the parameter source location and the result output location
- Get the URL in the parameter and execute the sqlmap tool
- Get the execution result of sqlmap and parse it into a custom format
- write the final result to the output location
<?php
//获取输入的参数
$inputFile = "/data/share/input_".getenv("xflow_node_id").".json";
$outputFile = "/data/share/output_".getenv("xflow_node_id").".json";
//没有input,直接返回
if (!file_exists($inputFile)) {
file_put_contents($outputFile, json_encode([]));
return 0;
}
//读取上游数据
$list = json_decode(file_get_contents($inputFile), true);
print_r($inputFile);
print_r($list);
$data = [];
//处理数据
foreach ($list as $val) {
$url = $val['url'];
$toolPath = "/data/tools/sqlmap/";
print_r("开始扫描URL:{$url}".PHP_EOL);
execTool($url, $toolPath);
//录入检测结果
$tempList = writeData($toolPath, $url);
print_r("扫描URL:{$url}完成".PHP_EOL);
print_r($tempList);
$data = array_merge($data, $tempList);
}
print_r($data);
//将结果写入到指定位置,供蜻蜓平台导入数据
file_put_contents($outputFile, json_encode($data, JSON_UNESCAPED_UNICODE));
function writeData($toolPath, $url)
{
$arr = parse_url($url);
$file_path = $toolPath . 'result/';
$host = $arr['host'];
$outdir = $file_path . "{$host}/";
$outfilename = "{$outdir}/log";
//sqlmap输出异常
if (!is_dir($outdir) or !file_exists($outfilename) or !filesize($outfilename)) {
print_r("sqlmap没有找到注入点: $url");
return [];
}
$ddd = file_get_contents($outfilename);
print_r($ddd);
exec("rm -rf $outdir");
return [["raw" => $ddd]];
}
function execTool($v, $toolPath)
{
$arr = parse_url($v);
$blackExt = ['.js', '.css', '.json', '.png', '.jpg', '.jpeg', '.gif', '.mp3', '.mp4'];
//没有可以注入的参数
if (!isset($arr['query']) or (strpos($arr['query'], '=') === false)) {
print_r(["URL地址不存在可以注入的参数".PHP_EOL, $v]);
return false;
}
$file_path = $toolPath . 'result/';
$cmd = "cd {$toolPath} && python3 ./sqlmap.py -u '{$v}' --batch --random-agent --output-dir={$file_path}";
exec($cmd);
return true;
}
Through the previous examples of xray and sqlmap tool encapsulation, you will find that the encapsulation process of each tool is almost the same, almost just the analysis of the output result of the program, so until now I have solved the problem of the ability of the scanner.
4. Hands-on practice
Now I just need to connect several functions, and here is a new problem to consider; the parameters required by sqlmap are indeed multiple specific URL addresses, that is to say, before calling sqlmap, I need to collect all the URLs Then call sqlmap, there is a data dependency problem here.
This problem is also easy to handle. We need to prepare three tables: target table, functional dependency table, and data storage table.
target table
| ID | URL | create_time |
|---|
menu
| ID | tool_name | pre_tool_name | create_time |
|---|
data sheet
| ID | tool_name | url | result | create_time |
|---|
We can first obtain a target to be scanned from the target table, and then read all the functions, for loop function table, only need to judge whether there is any dependency problem, or the dependency problem has been solved, then we can get the required dependency data , just execute the function directly.
The execution completion result can be seen on the result page, here is my execution result.
Pseudocode looks like this:
<?php
$id = getTarget();
$toolLst = getToolList();
foreach($toolList as $val){
//判断当前工具上级依赖为空或者上级工具已执行
if($val['pre_tool_name'] == '' or 上级工具已经执行){
//开始使用工具对URL扫描
scanUrl();
//保存结果
svaeResult();
} else(){
//上级工具还没执行完成,先跳过
continue;
}
}
This is the script I wrote, you can easily modify the application, you can copy and use it with one click
At present, I have integrated 46 common tools and put them open source in GitHub, address: https://github.com/StarCrossPortal/QingTing
Author: Tang Qingsong
Date: 2022-11-29