Guide: Simulated click refers to the completion of some automated click operations through scripts and system instructions, without manual clicks. It is generally used in automated testing to help read screen text, etc. This article mainly introduces several mainstream simulation click technologies and their applications.
Text｜Li Yiyang,
NetEase Yidun Android Security Engineer
Background: Simulated clicking refers to the completion of some automatic clicking operations through scripts and system instructions, without manual clicking. It is generally used in automated testing to help visually impaired or elderly people use mobile phones more conveniently. However, with the update of technology, simulated clicks have also been applied in different scenarios, such as automatic grabbing red envelopes, automatic chat and other black and gray production scenarios. This article mainly introduces several mainstream simulation click technologies and their applications.
Software scene
For software with many Views, such as IM, e-commerce, short video and other software, sometimes there are risks of automated scripts such as snatching red envelopes, chat bots, and grabbing flash sales. Because View is more complex, such scripts are often implemented relying on accessibility services. Because of the support of the system layer, the accessibility service is very convenient for View operations. For simulated click tools that do not rely on accessibility, it is very difficult to read View. For example, if you want to know a user name, you need to screenshot OCR to match fonts. Therefore, for this part of content, accessibility services have a natural advantage.
Automatic red envelope grabbing
Automatic grabbing of red envelopes goes against the original intention of developers, destroys the environment of fair competition, easily causes dissatisfaction and complaints from users, and is not conducive to the long-term sustainable development of applications.
auto chat
Some dating apps have a charging mode for chatting, which attracts black and gray products to make profits through automatic chatting through scripts, such as the script below, which can automatically receive and reply to messages. The principle is to use simulated clicks, automatically enter text and click the send button, but with a well-selected question and answer library, it has a high degree of credibility.
Such an approach undermines the manufacturer's marketing strategy, and it is "sweeping wool" from the manufacturer.
game scene
For games, common scenarios are script batch opening of new accounts, group control operations, script hangup, and script quick operations. This type of simulated click tool often has strong image recognition and color recognition functions. Usually, it can be judged based on the color of a few pixels whether the game has entered a stage that requires operation, such as entering a copy.
Auto hangup script
In some SLG games, resource transactions between players are allowed. Originally, this can shorten the distance between players and drive people around them to play together. However, due to the batch hang-up script, players would originally buy resources with cash from black and gray manufacturers. It will lead to the depreciation of resources, destroy the game currency system, and affect the normal operation of the game.
Automatic daily tasks, batch new numbers
Most games often have a lot of rewards in the novice stage, such as card draw games, and new accounts that are opened in batches will be bought all the time. Compared with recharging and drawing cards in the game, it will be cheaper to buy a new account with novice rewards to gamble on luck. But for the game party, this part of the income is lost in vain, and it is earned by the black and gray producers. In the same way, daily tasks can be completed in batches. The original intention is to ensure that users are not lost and can go online every day. However, black and gray manufacturers can operate a large number of accounts in batches to complete daily tasks. The core idea is similar to batch new accounts. Through Rewards that add up to a lot, draw cards for people who only want a specific type, or "replace the liver", invalidate the original means of guaranteeing players to go online, causing huge losses to the game party.
Automatic running ring, running tasks
In legendary games, the script of automatic running circle is also very common. Similar to the previous behavior, it is generally used to open new accounts in batches, which will also destroy the fairness of the game and affect the currency system.
In addition, there are many hazards of simulated clicks to software and games, and I will not show them one by one for the sake of space. The implementation of these simulated clicks will be analyzed from a technical point of view below.
Status and principle
Judging from the scripting tools on the market, the more famous ones include "Key Wizard", "Touch Wizard", "Auto.js" and other software. From the perspective of their implementation principles, the simulated clicks on the Android side can be said to be diverse.
In the early days, software such as "Key Wizard" wrote data to the "/dev/input/event*" path to simulate clicks, but this would modify the file permissions and be easily detected.
So later, there were more and more simulated click methods of various tricks, which will be introduced in detail below.
Accessibility Services
AccessibilityService (accessibility service class): For the accessibility service, the most powerful content is not the click mode on the coordinates. In fact, it was not until the Android7.0+ version that it began to support the click, long press, Click mode for operations such as dragging.
In addition to directly reading the properties of the View control, the accessibility service can also operate on the View control, such as button click and long press, text box input, multi-selection box selection, and drop-down box selection.
Nowadays, the simulated click tools mainly for games have also begun to slowly add the mode of barrier-free services, such as "key wizard" and "one-key play", etc., because they do not rely on the nature of root, many players without root can also play Experience the script.
View control operation
The accessibility service inherits AccessibilityService in the implementation class, and rewrites the specific method, which will be triggered by the specified event. The triggered event (Event) of AccessibilityService is: android.view.accessibility.AccessibilityEvent class.
The specific implementation method is to rewrite the onAccessibilityEvent method, taking the open source accessibility tool Auto.js as an example.
As can be seen from the above figure, after the service is triggered by the specified event, there is a rootInActiveWindow variable, which is obtained by the method in AssessibilityService - getRootInActiveWindow(). This method returns the root node of the window that the user currently most wants to be operated on. . After calling this method, the AccessibilityService application will traverse all child nodes according to the obtained root node until it finds the child node we want.
The graph of the Activity is a tree structure. Taking the barrier-free script mentioned above as an example, it can be found that the root node of the top layer is a FrameLayout layout.
After traversing all the child nodes of this root node, you can get all the View information in the window you want to operate, such as text, ID, Class, parent-child relationship, and a series of content. In the AccessibilityNodeInfo node class, there is a performAction method, which can directly send the desired operation to the View.
For Party A's manufacturer, although it is possible to use only Touch monitoring across the board, this will make the barrier-free services originally set up for the disabled meaningless. Therefore, more detection methods need to be combined to judge the improper use caused by the simulated click tool without affecting the normal use of the disabled.
gesture operation
Unlike tools that simulate clicks, Accessibility does not split all click events into atomic events (down, up, move, etc.), but treats gestures as a whole to execute. The dispatchGesture method in AccessibilityService realizes the distribution of gestures. Taking an Auto.js script as an example, this JS code realizes the simulation of three fingers sliding down.
Auto.js tools layer converted to Java layer:
Constructor, converted to a gesture:
Complete the distribution:
At present, more and more game plug-ins are beginning to adopt this method to supplement the coverage of non-root players. Both the development threshold and the usage threshold are very simple. It can also be seen from the above content that this approach can also ensure compatibility, and it can be directly used in the plug-in version of the accessibility service without modifying the existing script. It can be seen that in the future, it will become more and more common for simulated click plug-ins to be compatible with accessibility services.
Non-Accessible Services
For the simulated click implemented by non-accessible services, it actually refers to the implementation of input in the shell command.
Through source code analysis, we can see that:
The implementation of this method depends on the injectInputEvent interface in the system API. However, for any tool that wants to simulate clicks from outside the application, the first problem faced by this method is the permission problem. In the Android click event distribution process, the native function injectInputEvent will check the PID and UID of the incoming click event. If the permission is not obtained, the click process distribution will fail.
Therefore, to use this function, the Manifest.permission.INJECT_EVENTS permission must be declared in the application, and the permission can only be obtained when the application is started with root or adb permissions. Using this method to achieve simulated clicks has better performance and more High invisibility is also one of the most widely used technologies in black and gray production.
detection
Early detection generally relies on the system API to obtain the list of accessible services that have been turned on, and judges whether it is a malicious application through the blacklist, so as to intercept it, or uses the collection of the list of installed applications to determine whether there are traces of clicking tools. The method is simple and rude. And in the risk control scene, the rate of accidental injury is also high. With the promotion of package modification technology, privacy compliance, and the Android system is gradually restricting the acquisition of application installation lists, behavior detection and identification of click processes will become more reliable solutions for identifying simulated clicks in the future.
Summarize
From the data of NetEase Yidun's "2021 Game Security White Paper", it can be seen that for game manufacturers, they are more troubled by simulated click-type plug-ins. Different from invasive memory cheats, simulated click cheats are less intrusive and more difficult to detect, but the threshold for use is lower than other types of cheats. As long as users enable the accessibility service or use adb, they can use the scripts carefully prepared by plug-in manufacturers and script developers for simulating clicks.
Simulated clicking plug-ins have a great impact on manufacturers. Whether it destroys the marketing strategy of the software or the ecosystem of the game, it will affect the normal user experience and affect the revenue and reputation of the game manufacturer. But for manufacturers, it is impossible to cut off all simulated clicks simply and rudely, especially the accessibility service of software.
In recent years, with the popularization of smart devices, a large number of people who are inconvenient to use smart devices, such as the elderly and the disabled, also have a certain demand for using smart devices. The market is also gradually improving accessibility services, so manufacturers also need to consider the positive impact of accessibility services and simulated clicks, and use AI and other dimensions to judge whether simulated clicks are being used in an improper way.