Passive Vulnerability Scanner - Agent Design and Implementation
- Introduction
Before rushing graduation has been planning to write an own scanner, the last complete set of various busy time, but finally did not finish my thoughts using GourdScan V2.0 design. This traditional personal feeling on the scanner five years ago, ten years ago there used to be swept to many common Web vulnerabilities. But the web is now under way to use this environment for scanning directly, in addition to issuing a lot of very significant traffic outside the test results is quite limited. So the idea of their own design scanner has not given up, this time just taking advantage of the departure of the research study, and finally the agent did not get down get.
- Program
First, let's talk about probably a scanner program. At first envisaged when the scanner is very simple from a cell phone or a variety of terminal devices to get traffic generated, a scan task to scan these flows. In fact, these production flow in our lives is often overlooked, but for businesses these interfaces is their virtual assets to a user, or digging in vulnerability scanning process, these flows are often reptile biggest pain point, which is based on a flow of scanning the biggest advantage.
After the most primitive like a good program, I began to think about how to design proxy module, the first time is to create the wheel themselves, the idea was simply to Socks5 proxy server, Socks5 server to resolve the http traffic analysis package export. When trying to write their own version of a Socks5 protocol simplest functions, not to mention the performance, I do not know the reason is not the operating system, found that different clients have the Socks5 protocol differences, traffic flow is small okay great service end often jump unstable. Ss behind the brainstorm to open, but traffic is encrypted transmission, security is high, the client is also more abundant, we chose ss some custom designs.
- Socks5 protocol
Speaking proxy, Socks5 protocol had to start with, in fact, Socks5 protocol is very simple:
sock5 proxy procedures are:
1. The need for a request message to the proxy server side.
2. Agent transponder
3 sends the agent required and the transmission destination port ip after receiving the response to the Agent
4 Agent connection with the destination
information will need to 5. Agent Agent sent to the destination side, the information sent by destination spread requires a proxy party. Acting is complete.
First of all server and client to establish the connection after the handshake, the handshake format can be seen in official documents:
+----+----------+----------+
|VER | NMETHODS | METHODS |
+----+----------+----------+
| 1 | 1 | 1 to 255 |
+----+----------+----------+- VER: version of socks (socks5 corresponds 0x05)
- NMETHODS: The number appearing in the method METHODS field.
- METHODS: The client list of supported authentication methods, each method accounting for 1 byte.
The server returned the following information:
+----+--------+
|VER | METHOD |
+----+--------+
| 1 | 1 |
+----+--------+- VER: socks version (in socks5 is 0x05 in);
- METHOD: the selected method server (if the value 0xFF indicates that no method is selected, the client needs to close the connection);
We talked about back here to see the document will know how to connect, you can see the document in detail, very short. Since the introduction, that a TCP proxy process to write about it:
1. Establish a connection to the 1080 tcp port server.
2. The server sends 050 100 (this is the hexadecimal code, hereinafter the same)
3. If it is possible to proxy 05 00
4 01 + 050 100 send destination address (4 bytes) + object port (2 bytes), destination address and port are hexadecimal code. Examples of the information transmitted is 202.103.190.27 -7201: 05 01 00 01 67 the CA 21 is the BE. IB 1C (the CA 67 = 202 = 103 = 190. IB the BE 1C21 = 27 = 7201)
5. The receiving port and its own address returned by the server , connection completion
6. directly after the operation and the same TCP connection destination.
RFC documents here: https://www.ietf.org/rfc/rfc1928.txt
- SS source code analysis
This part of the analysis is relatively long, long time to read the source code, source code is simple to understand, but the specific application stability more difficult. ss is a very good and stable tool, if you want to understand the source of the flow, can I read about this a few slices article:
Shadowsocks source code analysis - an agreement with the structure
Shadowsocks source code analysis --TCP agent
Shadowsocks source code analysis - timeout handling
Shadowsocks source code analysis --UDP agent
Two front main read on it, where further ado, sent directly to a connection structure of FIG ss:
Our main flow when ssserver crawl and target servers connected, this part of the traffic is decrypted ss. Here is the process for creating and remote server connections, dns resolution first, successful communication with a remote socket created.
def _handle_dns_resolved(self, result, error):
if error:
self.destroy()
return
if not (result and result[1]):
self.destroy()
return
ip = result[1]
self._stage = STAGE_CONNECTING
remote_addr = ip
if self._is_local:
remote_port = self._chosen_server[1]
else:
remote_port = self._remote_address[1]
remote_sock = self._create_remote_socket(remote_addr,
remote_port)
try:
remote_sock.connect((remote_addr, remote_port))
except (OSError, IOError) as e:
pass
self._loop.add(remote_sock,
eventloop.POLL_ERR | eventloop.POLL_OUT,
self._server)
self._update_stream(STREAM_UP, WAIT_STATUS_READWRITING)
self._update_stream(STREAM_DOWN, WAIT_STATUS_READING)remote_sock, this is the remote server connections and socket, put an immediate export out ok.
Https flow
When seemingly everything is all right, another question is, if it is Https traffic, then how to do? Here socket ssl derived data are also encrypted. I think for a long time and I still feel like Burp this, in the middle to make a ca server certificates you trust yourself more feasible. Here you find a good use of the python library mitmproxy, the library built to achieve a set of proxy and authentication, this library can be fused together and ss. To keep the code too coupling, first from a local server with mimtproxy a socks5, and at the same time when I made the request to the remote host in the socket ss change it, add a layer of socks5 protocol, ss force the flow of imported mitmproxy, such two-part coupling is relatively small, while management traffic is also more convenient. The following is a modified method _create_remote_socket.
def _create_remote_socket(self, ip, port):
addrs = socket.getaddrinfo(ip, port, 0, socket.SOCK_STREAM,
socket.SOL_TCP)
#print("-------------------------------------------目标地址----------------------------------------------")
#print(addrs)
#print("-------------------------------------------END----------------------------------------------")
print(self._client_address[0],self._config['server_port'])
if len(addrs) == 0:
raise Exception("getaddrinfo failed for %s:%d" % (ip, port))
af, socktype, proto, canonname, sa = addrs[0]
if self._forbidden_iplist:
if common.to_str(sa[0]) in self._forbidden_iplist:
raise Exception('IP %s is in forbidden list, reject' %
common.to_str(sa[0]))
remote_sock = socket.socket(af, socktype, proto)
remote_sock = socks.socksocket()
remote_sock.set_proxy(socks.SOCKS5, "localhost", 8080,username='tttest')
self._remote_sock = remote_sock
self._fd_to_handlers[remote_sock.fileno()] = self
remote_sock.setblocking(False)
remote_sock.setsockopt(socket.SOL_TCP, socket.TCP_NODELAY, 1)
return remote_sockAt this time, from the local port 8080 with a mimtproxy socks5 server.
$ curl -xsocks5://127.0.0.1:1086 https://www.baidu.com
<!DOCTYPE html>
<!--STATUS OK--><html> <head><meta http-equiv=content-type content=text/html;charset=utf-8><meta http-equiv=X-UA-Compatible content=IE=Edge><meta content=always name=referrer><link rel=stylesheet type=text/css href=https://ss1.bdstatic.com/5eN1bjq8AAUYm2zgoY3K/r/www/cache/bdorz/baidu.min.css><title>百度一下，你就知道</title></head> <body link=#0000cc> <div id=wrapper> <div id=head> <div class=head_wrapper> <div class=s_form> <div class=s_form_wrapper> <div id=lg> <img hidefocus=true src=//www.baidu.com/img/bd_logo1.png width=270 height=129> </div> <form id=form name=f action=//www.baidu.com/s class=fm> <input type=hidden name=bdorz_come value=1> <input type=hidden name=ie value=utf-8> <input type=hidden name=f value=8> <input type=hidden name=rsv_bp value=1> <input type=hidden name=rsv_idx value=1> <input type=hidden name=tn value=baidu><span class="bg s_ipt_wr"><input id=kw name=wd class=s_ipt value maxlength=255 autocomplete=off autofocus=autofocus></span><span class="bg s_btn_wr"><input type=submit id=su value=百度一下 class="bg s_btn" autofocus></span> </form> </div> </div> <div id=u1> <a href=http://news.baidu.com name=tj_trnews class=mnav>新闻</a> <a href=https://www.hao123.com name=tj_trhao123 class=mnav>hao123</a> <a href=http://map.baidu.com name=tj_trmap class=mnav>地图</a> <a href=http://v.baidu.com name=tj_trvideo class=mnav>视频</a> <a href=http://tieba.baidu.com name=tj_trtieba class=mnav>贴吧</a> <noscript> <a href=http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u=http%3A%2F%2Fwww.baidu.com%2f%3fbdorz_come%3d1 name=tj_login class=lb>登录</a> </noscript> <script>document.write('<a href="http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u='+ encodeURIComponent(window.location.href+ (window.location.search === "" ? "?" : "&")+ "bdorz_come=1")+ '" name="tj_login" class="lb">登录</a>');
</script> <a href=//www.baidu.com/more/ name=tj_briicon class=bri style="display: block;">更多产品</a> </div> </div> </div> <div id=ftCon> <div id=ftConw> <p id=lh> <a href=http://home.baidu.com>关于百度</a> <a href=http://ir.baidu.com>About Baidu</a> </p> <p id=cp>©2017 Baidu <a href=http://www.baidu.com/duty/>使用百度前必读</a> <a href=http://jianyi.baidu.com/ class=cp-feedback>意见反馈</a> 京ICP证030173号 <img src=//www.baidu.com/img/gs.gif> </p> </div> </div> </div> </body> </html> See mitmproxy can also receive traffic:
Back flow storage can be loaded directly mitmproxy script flow mongodb import them. Acting almost like this part of the framework design is complete, and now has reached the desired effect. But some time later iterations of the pit to be filled, which is currently only a single user, multiple users if it is, how to distinguish between different users or client traffic is a problem. It is envisaged that part of ss can be distinguished by the source ip through from Different ports to authenticate the user, or a different user ss distinguishing a mitmproxy different user characteristics, but mitmproxy not support socks5 user authentication, the source above socks5 certification this one left empty, future generations need to be filled.