The application of RPC technology and its framework Sekiro in the reverse engineering of crawler, encrypted data shuttle!

Pay attention to the WeChat public account: Brother K crawler, continue to share technical dry goods such as advanced crawler, JS/Android reverse!

what is RPC

~~RPC, English RangPaCong, Chinese let crawlers, aims to open the way for crawlers, kill everything in seconds, and make crawlers unimpeded!~~

Just kidding, RPC is actually a remote procedure call, the full name is Remote Procedure Call, which is a technical idea rather than a specification or protocol. In fact, the birth of RPC is inseparable from distributed development. RPC mainly solves two problems:

Solve the problem of mutual invocation between services in distributed systems;
RPC makes remote calls as convenient as local calls, so that the caller cannot perceive the logic of remote calls.

The existence of RPC makes it easier to build distributed systems. Compared with the HTTP protocol, RPC uses binary bytecode transmission, so it is more efficient and secure. In a typical RPC usage scenario, it includes service discovery, load, fault tolerance, network transmission, serialization and other components. The complete RPC architecture diagram is shown in the following figure:

JSRPC

RPC technology is very complex. For those who engage in crawling and reverse engineering, we do not need to fully understand it. We only need to know how to apply this technology in reverse engineering.

In reverse RPC, in simple terms, the local and the browser are regarded as the server and the client, and RPC communication is carried out between the two through the WebSocket protocol, the encryption function is exposed in the browser, and the browser is directly called locally. The corresponding encryption function in the controller can be obtained to obtain the encryption result. It is unnecessary to pay attention to the specific execution logic of the function, and it also saves operations such as deducting code and complementing the environment, which can save a lot of reverse debugging time. We take the login of a group's web page as an example to demonstrate the specific use of RPC in reverse. (Assuming you already have a certain reverse foundation and understand the WebSocket protocol, pure white can first read Brother K's previous articles)

Homepage (base64):aHR0cHM6Ly9wYXNzcG9ydC5tZWl0dWFuLmNvbS9hY2NvdW50L3VuaXRpdmVsb2dpbg==
Parameters: h5Fingerprint

First grab the package, the login interface has a super long parameter h5Fingerprint, as shown in the following figure:

Just do a search to find the encryption function:

utility.getH5fingerprint()The parameters passed in are window.location.origin + urlformatted, and the parameters are as follows:

url = "https://passport.脱敏处理.com/account/unitivelogin"
params = {
    "risk_partner": "0",
    "risk_platform": "1",
    "risk_app": "-1",
    "uuid": "96309b5f00ba4143b920.1644805104.1.0.0",
    "token_id": "DNCmLoBpSbBD6leXFdqIxA",
    "service": "www",
    "continue": "https://www.脱敏处理.com/account/settoken?continue=https%3A%2F%2Fwww.脱敏处理.com%2F"
}

Both uuid and token_id can be searched directly. They are not the focus of this research, so I won't go into details here. Next, we will use RPC technology to directly call the utility.getH5fingerprint()method First, write the server code locally so that it can always enter the waiting list. Encrypt the string, receive and print the encrypted string:

# ==================================
# --*-- coding: utf-8 --*--
# @Time    : 2022-02-14
# @Author  : 微信公众号：K哥爬虫
# @FileName: ws_server.py
# @Software: PyCharm
# ==================================


import sys
import asyncio
import websockets


async def receive_massage(websocket):
    while True:
        send_text = input("请输入要加密的字符串: ")
        if send_text == "exit":
            print("Exit, goodbye!")
            await websocket.send(send_text)
            await websocket.close()
            sys.exit()
        else:
            await websocket.send(send_text)
            response_text = await websocket.recv()
            print("\n加密结果：", response_text)


start_server = websockets.serve(receive_massage, '127.0.0.1', 5678)  # 自定义端口
asyncio.get_event_loop().run_until_complete(start_server)
asyncio.get_event_loop().run_forever()

Write the JS code on the browser client side, and directly utility.getH5fingerprint()get the :

/* ==================================
# @Time    : 2022-02-14
# @Author  : 微信公众号：K哥爬虫
# @FileName: ws_client.js
# @Software: PyCharm
# ================================== */


var ws = new WebSocket("ws://127.0.0.1:5678");  // 自定义端口

ws.onmessage = function (evt) {
    console.log("Received Message: " + evt.data);
    if (evt.data == "exit") {
        ws.close();
    } else {
        ws.send(utility.getH5fingerprint(evt.data))
    }
};

Then we need to inject the client code into the web page. There are many methods here, such as the packet capture software Fiddler to replace the response, the browser plug-in ReRes to replace JS, the browser developer tool Overrides to rewrite the function, etc. You can also use plug-ins, oil monkeys, etc. Wait for the way to inject Hook. Anyway, there are many methods. Friends who don't know much about these methods can go to see Brother K's previous articles, all of which are introduced.

Here we use the browser developer tool Overrides to rewrite the function, add the WebSocket client code to the encrypted JS file and save it with Ctrl+S. Here it is written as an IIFE self-execution method. The reason for this is to prevent global pollution. Variables, without self-execution, are of course possible.

Then run the local server code first, log in once on the web page, log in once on the web page, log in once on the web page, and say the important steps three times! Then you can pass in the string to be encrypted locally and get the utility.getH5fingerprint()encrypted result:

Ax

Through the previous example, you can find that it is too troublesome to write the server, and it is not easy to expand. Is there a ready-made wheel in this regard? The answer is yes, here are two projects:

JsRPC-hliang：https://github.com/jxhczhl/JsRpc
Sekiro ：https://github.com/virjar/sekiro

JsRPC-hliang is written in go language and is a project specially made for JS reverse engineering, while Sekiro is more powerful. Sekiro is written by Deng Weijia, commonly known as Mr. Slag, an Android Private API exposure based on long links and code injection. The framework can be used in APP reverse, APP data capture, Android group control and other scenarios. At the same time, Sekiro is also the only stable JSRPC framework in the current public solution. The usage of the two in JS reverse is actually similar. This article mainly introduces Sekiro in Applications in Web JS Reverse.

Referring to the Sekiro documentation, first compile the project locally:

Linux & Mac: Execute the script build_demo_server.shand get the output distribution tarball:sekiro-service-demo/target/sekiro-release-demo.zip
Windows: Direct download available: https://oss.virjar.com/sekiro/sekiro-demo

Then run it locally (requires a Java environment, configure it yourself):

Linux & Mac：bin/sekiro.sh
Windows：bin/sekiro.bat

Take Windows as an example, after startup it is as follows:

Next, you need to inject code into the browser. You need to inject sekiro_web_client.js (download address: https://sekiro.virjar.com/sekiro-doc/assets/sekiro_web_client.js) provided by the author into the browser environment. Then communicate with the Sekiro server through SekiroClient, you can directly call the browser's internal methods through RPC. The official SekiroClient code sample is as follows:

function guid() {
    function S4() {
        return (((1+Math.random())*0x10000)|0).toString(16).substring(1);
    }
    return (S4()+S4()+"-"+S4()+"-"+S4()+"-"+S4()+"-"+S4()+S4()+S4());
}

var client = new SekiroClient("wss://sekiro.virjar.com/business/register?group=ws-group&clientId="+guid());

client.registerAction("clientTime",function(request, resolve, reject){
    resolve(""+new Date());
})

In the wss link, if it is a free version, change business to business-demo, and explain the terms involved:

group : business type (interface group), one group for each business, multiple terminals (SekiroClient) can be registered under the group, and multiple actions can be mounted in the group at the same time;
clientId : refers to the device, multiple devices use multiple machines to provide API services, provide group control capabilities and load balancing capabilities;
SekiroClient : Service provider client, the main scenario is mobile phone/browser, etc. The final Sekiro call is forwarded to the SekiroClient. Each client needs to have a unique clientId;
registerAction : interface, there can be multiple interfaces under the same group, each with different functions;
resolve : the method to pass the content back to the client;
request : The request sent by the client. If there are multiple parameters in the request, the parameters can be extracted from it in the form of key-value pairs and then processed.

Having said so much, it may not be easy to understand. Let’s take the example of logging in to a certain group’s web page. We will write the communication code of sekiro_web_client.js and SekiroClient together, and then rewrite the communication code according to the requirements:

The ws link is changed to: ws://127.0.0.1:5620/business-demo/register?group=rpc-test&clientId=, and the custom groupis rpc-test;
register an event registerActionas getH5fingerprint;
resolveThe returned result is utility.getH5fingerprint(request["url"])to encrypt and return the url parameter passed by the client.

The complete code is as follows (note the writing of the SekiroClient communication code section at the end):

/* ==================================
# @Time    : 2022-02-14
# @Author  : 微信公众号：K哥爬虫
# @FileName: sekiro.js
# @Software: PyCharm
# ================================== */

(function () {
    'use strict';
    function SekiroClient(wsURL) {
        this.wsURL = wsURL;
        this.handlers = {};
        this.socket = {};
        // check
        if (!wsURL) {
            throw new Error('wsURL can not be empty!!')
        }
        this.webSocketFactory = this.resolveWebSocketFactory();
        this.connect()
    }

    SekiroClient.prototype.resolveWebSocketFactory = function () {
        if (typeof window === 'object') {
            var theWebSocket = window.WebSocket ? window.WebSocket : window.MozWebSocket;
            return function (wsURL) {

                function WindowWebSocketWrapper(wsURL) {
                    this.mSocket = new theWebSocket(wsURL);
                }

                WindowWebSocketWrapper.prototype.close = function () {
                    this.mSocket.close();
                };

                WindowWebSocketWrapper.prototype.onmessage = function (onMessageFunction) {
                    this.mSocket.onmessage = onMessageFunction;
                };

                WindowWebSocketWrapper.prototype.onopen = function (onOpenFunction) {
                    this.mSocket.onopen = onOpenFunction;
                };
                WindowWebSocketWrapper.prototype.onclose = function (onCloseFunction) {
                    this.mSocket.onclose = onCloseFunction;
                };

                WindowWebSocketWrapper.prototype.send = function (message) {
                    this.mSocket.send(message);
                };

                return new WindowWebSocketWrapper(wsURL);
            }
        }
        if (typeof weex === 'object') {
            // this is weex env : https://weex.apache.org/zh/docs/modules/websockets.html
            try {
                console.log("test webSocket for weex");
                var ws = weex.requireModule('webSocket');
                console.log("find webSocket for weex:" + ws);
                return function (wsURL) {
                    try {
                        ws.close();
                    } catch (e) {
                    }
                    ws.WebSocket(wsURL, '');
                    return ws;
                }
            } catch (e) {
                console.log(e);
                //ignore
            }
        }
        //TODO support ReactNative
        if (typeof WebSocket === 'object') {
            return function (wsURL) {
                return new theWebSocket(wsURL);
            }
        }
        // weex 和 PC环境的websocket API不完全一致，所以做了抽象兼容
        throw new Error("the js environment do not support websocket");
    };

    SekiroClient.prototype.connect = function () {
        console.log('sekiro: begin of connect to wsURL: ' + this.wsURL);
        var _this = this;
        // 不check close，让
        // if (this.socket && this.socket.readyState === 1) {
        //     this.socket.close();
        // }
        try {
            this.socket = this.webSocketFactory(this.wsURL);
        } catch (e) {
            console.log("sekiro: create connection failed,reconnect after 2s");
            setTimeout(function () {
                _this.connect()
            }, 2000)
        }

        this.socket.onmessage(function (event) {
            _this.handleSekiroRequest(event.data)
        });

        this.socket.onopen(function (event) {
            console.log('sekiro: open a sekiro client connection')
        });

        this.socket.onclose(function (event) {
            console.log('sekiro: disconnected ,reconnection after 2s');
            setTimeout(function () {
                _this.connect()
            }, 2000)
        });
    };

    SekiroClient.prototype.handleSekiroRequest = function (requestJson) {
        console.log("receive sekiro request: " + requestJson);
        var request = JSON.parse(requestJson);
        var seq = request['__sekiro_seq__'];

        if (!request['action']) {
            this.sendFailed(seq, 'need request param {action}');
            return
        }
        var action = request['action'];
        if (!this.handlers[action]) {
            this.sendFailed(seq, 'no action handler: ' + action + ' defined');
            return
        }

        var theHandler = this.handlers[action];
        var _this = this;
        try {
            theHandler(request, function (response) {
                try {
                    _this.sendSuccess(seq, response)
                } catch (e) {
                    _this.sendFailed(seq, "e:" + e);
                }
            }, function (errorMessage) {
                _this.sendFailed(seq, errorMessage)
            })
        } catch (e) {
            console.log("error: " + e);
            _this.sendFailed(seq, ":" + e);
        }
    };

    SekiroClient.prototype.sendSuccess = function (seq, response) {
        var responseJson;
        if (typeof response == 'string') {
            try {
                responseJson = JSON.parse(response);
            } catch (e) {
                responseJson = {};
                responseJson['data'] = response;
            }
        } else if (typeof response == 'object') {
            responseJson = response;
        } else {
            responseJson = {};
            responseJson['data'] = response;
        }


        if (Array.isArray(responseJson)) {
            responseJson = {
                data: responseJson,
                code: 0
            }
        }

        if (responseJson['code']) {
            responseJson['code'] = 0;
        } else if (responseJson['status']) {
            responseJson['status'] = 0;
        } else {
            responseJson['status'] = 0;
        }
        responseJson['__sekiro_seq__'] = seq;
        var responseText = JSON.stringify(responseJson);
        console.log("response :" + responseText);
        this.socket.send(responseText);
    };

    SekiroClient.prototype.sendFailed = function (seq, errorMessage) {
        if (typeof errorMessage != 'string') {
            errorMessage = JSON.stringify(errorMessage);
        }
        var responseJson = {};
        responseJson['message'] = errorMessage;
        responseJson['status'] = -1;
        responseJson['__sekiro_seq__'] = seq;
        var responseText = JSON.stringify(responseJson);
        console.log("sekiro: response :" + responseText);
        this.socket.send(responseText)
    };

    SekiroClient.prototype.registerAction = function (action, handler) {
        if (typeof action !== 'string') {
            throw new Error("an action must be string");
        }
        if (typeof handler !== 'function') {
            throw new Error("a handler must be function");
        }
        console.log("sekiro: register action: " + action);
        this.handlers[action] = handler;
        return this;
    };

    function guid() {
        function S4() {
            return (((1 + Math.random()) * 0x10000) | 0).toString(16).substring(1);
        }

        return (S4() + S4() + "-" + S4() + "-" + S4() + "-" + S4() + "-" + S4() + S4() + S4());
    }

    var client = new SekiroClient("ws://127.0.0.1:5620/business-demo/register?group=rpc-test&clientId=" + guid());

    client.registerAction("getH5fingerprint", function (request, resolve, reject) {
        resolve(utility.getH5fingerprint(request["url"]));
    })

})();

As with the previous method, use the browser developer tool Overrides to rewrite the function and inject the above code into the web page JS:

Then Sekiro provides us with some APIs:

View the group list: http://127.0.0.1:5620/business-demo/groupList
View queue status: http://127.0.0.1:5620/business-demo/clientQueue?group=test
Call forwarding: http://127.0.0.1:5620/business-demo/invoke?group=test&action=test¶m=testparm

For example, what if we want to call the utility.getH5fingerprint()encryption method now? It's very simple. After the code is injected into the browser, you must first log in manually, log in manually, log in manually, and say important things three times! Then refer to the above call forwarding API to rewrite:

Our custom grouping groupis rpc-test;
event actionis getH5fingerprint;
The name of the parameter to be encrypted is url, and its value is, for example:https://www.baidu.com/

Then our call link should be: http://127.0.0.1:5620/business-demo/invoke?group=rpc-test&action=getH5fingerprint&url=https://www.baidu.com/, open directly in the browser, the returned dictionary, and the encrypted result in the data:

Similarly, if you use Python locally, you can just do the requests directly:

We copied sekiro_web_client.js and injected it into the browser together with the communication code. Here we can also have a more elegant method, create a new script for the document directly, and insert sekiro_web_client.js in the form of a link, here we need to pay attention A few questions:

The first is the timing problem. You need to wait for the document elements to be loaded before establishing SekiroClient communication. Otherwise, calling SekiroClient will report an error. Here you can use the setTimeout method, which is used to call a function or calculate an expression after a specified number of milliseconds. , encapsulate the SekiroClient communication code into a function separately, for example function startSekiro(), wait for 1-2 seconds before executing the SekiroClient communication code;
Since the communication code of SekiroClient is encapsulated into a function, the direct call utility.getH5fingerprintat will prompt undefined, so we must first import it as a global variable, for example window.getH5fingerprint = utility.getH5fingerprint, it can be called window.getH5fingerprintdirectly .

The complete code looks like this:

/* ==================================
# @Time    : 2022-02-14
# @Author  : 微信公众号：K哥爬虫
# @FileName: sekiro.js
# @Software: PyCharm
# ================================== */

(function () {
    var newElement = document.createElement("script");
    newElement.setAttribute("type", "text/javascript");
    newElement.setAttribute("src", "https://sekiro.virjar.com/sekiro-doc/assets/sekiro_web_client.js");
    document.body.appendChild(newElement);

    window.getH5fingerprint = utility.getH5fingerprint

    function guid() {
        function S4() {
            return (((1 + Math.random()) * 0x10000) | 0).toString(16).substring(1);
        }
        return (S4() + S4() + "-" + S4() + "-" + S4() + "-" + S4() + "-" + S4() + S4() + S4());
    }

    function startSekiro() {
        var client = new SekiroClient("ws://127.0.0.1:5620/business-demo/register?group=rpc-test&clientId=" + guid());

        client.registerAction("getH5fingerprint", function (request, resolve, reject) {
            resolve(window.getH5fingerprint(request["url"]));
        })
    }

    setTimeout(startSekiro, 2000)
})();

Advantages and disadvantages

At present, if you don't reverse JS to implement encryption parameters, the most used automation tools, such as Selenium, Puppeteer, etc. Obviously, these automation tools are cumbersome to configure and extremely inefficient, and RPC technology does not need to load redundant resources, The stability and efficiency are obviously higher. RPC does not need to consider browser fingerprints and various environments. If the risk control is not strict, high concurrency can be easily achieved. On the contrary, because RPC is always mounted on the same browser Therefore, for sites with strict risk control, such as detecting the binding of UA, IP and encryption parameters, it is not feasible to call the PRC too frequently. Of course, you can also study the browser group control technology and manipulate multiple different Browsers can alleviate this problem to some extent. In short, RPC technology is still very good, except for JS reverse, it can be said to be a more versatile and efficient method at present, and to a certain extent, it can encrypt parameters!