scrcpy-client
There is a scrcpy-client library in python, which can realize real-time screencasting and control of Android devices. It is the same as scrcpy to implement Android screencasting. Both push a scrcpy-server.jar file to the Android device through adb, and use adb instructions to execute scrcpy-server.jar to start screencasting and control the server. The computer side uses python Create a client to receive video stream data and send control stream data. The video stream data is Android real-time screen data, and the control stream data is the control actions we perform on the Android device on the computer side. In the scrcpy-client library, the author provides a screen projection control UI interface built using PySide6, which can complete the screen projection control of a single Android device. We can make our own screen projection control interface to complete the screen projection control of multiple Android devices.
Installation instructions: pip3 install scrcpy-client
Use directly
After installing the scrcpy-client library, we can directly use the UI interface provided by the author to mirror the Android device.
import scrcpy_ui
scrcpy_ui.main()Make sure that an Android device is connected to our computer via USB, and that the USB debugging function of the Android device is turned on and the computer is allowed to be debugged. At this time, we can get the screencasting UI interface of the Android device by executing the above code, as shown in the following figure:
In this interface, we can use the mouse to click and slide on the screen casting interface to control the screen of the Android device. You can switch devices through the device serial number drop-down box. After Flip is checked, you can get a mirrored screen. Click the HOME button below to return to the home screen, which is equivalent to pressing the home button of the device. After clicking the BACK button, it will return to the previous interface, which is equivalent to pressing the back key on the device. It also supports keyboard input. We can let Android generate key events through the computer keyboard.
Custom use
If you feel that the UI interface provided by the author cannot meet your needs, we can also customize the UI interface to implement more operation methods. The premise is that you must be able to use a UI framework such as PySide6. You must know how to use dynamic elements in the UI interface, how to implement mouse click and move events, mouse wheel scrolling events, and keyboard input events. If you don’t know how to use UI interface-related frameworks, you can learn it first. If you know UI-related frameworks, continue reading below.
Create a screencasting service
Use the Client class in scrcpy to establish a screen projection control service. The instantiation method in the Client class is as follows:
class Client:
def __init__(
self,
device: Optional[Union[AdbDevice, str, any]] = None,
max_width: int = 0,
bitrate: int = 8000000,
max_fps: int = 0,
flip: bool = False,
block_frame: bool = False,
stay_awake: bool = False,
lock_screen_orientation: int = LOCK_SCREEN_ORIENTATION_UNLOCKED,
connection_timeout: int = 3000,
encoder_name: Optional[str] = None,
):device: The device serial number of the Android device (can be viewed using the adb devices command).
max_width: The maximum width of the image frame. By default, the frame width in the Android broadcast information is used.
bitrate: bit rate, default 8000000 bits.
max_fps: The maximum number of frames, the number of frames is not limited by default.
flip: flip the image (mirror image), not mirrored by default.
block_frame: Returns a non-empty frame. It is not returned by default. Returning a non-empty frame may block the openCv2 rendering thread.
stay_awake: The screen of the Android device remains always on when connected to USB. The screen does not remain always on by default.
lock_screen_orientation: Lock the screen orientation (disable automatic screen rotation), not locked by default.
connection_timeout: Timeout time for connecting to the screen projection control service (socket service). If the connection is not successful within the set time, the initialization will fail. The default is 3000 milliseconds.
encoder_name: encoder name, optional OMX.google.h264.encoder, OMX.qcom.video.encoder.avc, c2.qti.avc.encoder, c2.android.avc.encoder, automatically selected by default.
We instantiate the Client class to get a screen mirroring control service object of an Android device. If the serial number of the Android device is 123456789, we can create a screen mirroring control service instance object through the following code:
import scrcpy
server = scrcpy.Client(device='123456789', bitrate=100000000)start method
start is an instance method of Client used to start the screen projection control service. start can receive two parameters. One is threaded, which defaults to False. When it is True, it means that the screen projection control service is enabled in the child thread; the other is daemon_threaded, which defaults to False. False, when True, it means to enable process daemon for the child thread. When either threaded or daemon_threaded is True, the screen projection control service will be enabled in the child thread. When both are False, the screen projection control service will be enabled in the main thread. When you want to enable multiple screen projection control services at the same time, you need to enable the screen projection control service in a sub-thread. You can also create a sub-thread yourself.
server.start()add_listener method
add_listener is an instance method of Client used to set the listener dictionary listeners. There are two elements in the listeners dictionary. The key of the first element is frame, which represents the image frame element, and the second element is init, which represents the initialization element. The value of both elements is an empty list, used to store functions. If you want to put the screen image of the Android device on an element of the UI interface, you need to write a method in the UI framework that can receive the image and display the image, and then add this method to the first element list of the listeners dictionary. . If you want to do some operations when establishing a screen projection control service, write an operation-related method in the UI framework, and put this method in the second element list of the listeners dictionary.
def on_frame(self, frame): # 在使用PySide6的UI框架中定义了一个用于显示图像的方法
app.processEvents()
if frame is not None:
ratio = self.max_width / max(self.client.resolution)
image = QImage(
frame,
frame.shape[1],
frame.shape[0],
frame.shape[1] * 3,
QImage.Format_BGR888,
)
pix = QPixmap(image) # 处理图像
pix.setDevicePixelRatio(1 / ratio) # 设置图像大小
self.ui.label.setPixmap(pix) # 在UI界面显示图像
self.resize(1, 1)When initializing the UI interface, add the method of displaying images to the first element of the listener dictionary (key='frame').
def __init__(self):
super().__init()
self.server = scrcpy.Client(device='123456789', bitrate=100000000)
self.server.add_listener(scrcpy.EVENT_FRAME, self.on_frame)These are just examples, the actual method needs to be written according to your needs. If you want to display the image of the Android device on a certain element of the UI interface, you must write a method to display the image, and then add this method to the first element of the listener dictionary (key='frame').
remove_listener method
The remove_listener method is an instance method of the Client and is used to remove a method in the listener dictionary. If we no longer want to display the image, we can remove the method that displays the image from the listeners.
def no_display(self):
self.server.remove_listener(scrcpy.EVENT_FRAME, self.on_frame)stop method
The stop method is an instance method of Client used to end the screen projection control service. When we close the screen projection UI interface, we need to end the screen projection control service and release memory resources in time.
def closeEvent(self, _):
self.server.stop()Control Method
We have projected the screen of the Android device to the computer, and now we need to operate the Android device through some methods of controlling the Android device. There is a control attribute in the instance attributes of Client, which is obtained by instantiating the ControlSender class. The ControlSender class is an operation class specially used to control Android devices.
self.control = ControlSender(self)So if we want to control Android, we need to control the control attribute of the screen projection object.
keycode method
@inject(const.TYPE_INJECT_KEYCODE)
def keycode(
self, keycode: int, action: int = const.ACTION_DOWN, repeat: int = 0
) -> bytes:The keycode method is an instance method of the ControlSender class used to send key events to Android devices. The keycode method can receive 3 parameters. The first parameter keycode represents the key value (you need to understand the adb key value); the second parameter action represents whether to press or lift, the default is to press; the third parameter repeat represents repeated operations. times, how many times you want to press it repeatedly.
def click_home(self):
self.server.control.keycode(scrcpy.KEYCODE_HOME, scrcpy.ACTION_DOWN)
self.server.control.keycode(scrcpy.KEYCODE_HOME, scrcpy.ACTION_UP)Click the home button, press it first and then lift it up to complete one button press.
text method
@inject(const.TYPE_INJECT_TEXT)
def text(self, text: str) -> bytes:The text method is an instance method of the ControlSender class and is used to enter text into Android, provided that an input box in the Android device is activated. The text method receives a parameter, which is the text content we want to enter on the Android device.
def input_text(self, text):
self.server.control.text(text)touch method
def touch(
self, x: int, y: int, action: int = const.ACTION_DOWN, touch_id: int = -1
) -> bytes:The touch method is an instance method of the ControlSender class used for multi-touch control of the Android device screen. The touch method can receive 4 parameters. The first two parameters are the x coordinate and y coordinate of the touch point; the third parameter action is press, move, and lift; the fourth parameter is the touch event id, and the default is -1 , you can set different IDs to execute multiple touch events at the same time to achieve multi-touch purposes.
def mouse_move(self, evt: QMouseEvent):
focused_widget = QApplication.focusWidget()
if focused_widget is not None:
focused_widget.clearFocus()
ratio = self.max_width / max(self.one_client.resolution)
self.server.control.touch(evt.position().x() / ratio, evt.position().y() / ratio, scrcpy.ACTION_MOVE)scroll method
@inject(const.TYPE_INJECT_SCROLL_EVENT)
def scroll(self, x: int, y: int, h: int, v: int) -> bytes:The scroll method is an instance method of the ControlSender class used for scrolling events on the Android device screen. The scroll method can receive 4 parameters, the first two are the coordinate position of the scroll point; the third parameter is the horizontal scroll distance; the fourth parameter is the vertical scroll distance.
def on_wheel(self):
"""鼠标滚轮滚动事件"""
def wheel(evt: QWheelEvent):
ratio = self.max_width / max(self.one_client.resolution)
position_x = evt.position().x() / ratio
position_y = evt.position().y() / ratio
angle_x = evt.angleDelta().x()
angle_y = evt.angleDelta().y()
if angle_y > 0:
angle_y = 1
else:
angle_y = -1
self.server.control.scroll(position_x, position_y, angle_x, angle_y)
return wheelAfter writing this method, we can use the mouse wheel to control the screen of the Android device to scroll up and down.
back_or_turn_screen_on method
@inject(const.TYPE_BACK_OR_SCREEN_ON)
def back_or_turn_screen_on(self, action: int = const.ACTION_DOWN) -> bytes:The back_or_turn_screen_on method is an instance method of the ControlSender class for pressing the return key, and will wake up the screen if the screen is turned off. Only one parameter action is received, which is press or lift.
def click_back(self):
self.server.control.back_or_turn_screen_on(scrcpy.ACTION_DOWN)
self.server.control.back_or_turn_screen_on(scrcpy.ACTION_UP)expand_notification_panel method
@inject(const.TYPE_EXPAND_NOTIFICATION_PANEL)
def expand_notification_panel(self) -> bytes:The expand_notification_panel method is an instance method of the ControlSender class and is used to open the drop-down notification bar of Android devices.
def open_notification(self):
self.server.control.expand_notification_panel()expand_settings_panel method
@inject(const.TYPE_EXPAND_SETTINGS_PANEL)
def expand_settings_panel(self) -> bytes:The expand_settings_panel method is an instance method of the ControlSender class used to open the drop-down menu bar of Android devices.
def open_settings(self):
self.server.control.expand_settings_panel()collapse_panels method
@inject(const.TYPE_COLLAPSE_PANELS)
def collapse_panels(self) -> bytes:The collapse_panels method is an instance method of the ControlSender class and is used to collapse the drop-down notification bar or menu bar of Android devices.
def close_panel(self):
self.server.control.collapse_panelsl()get_clipboard method
def get_clipboard(self) -> str:The get_clipboard method is an instance method of the ControlSender class used to obtain the contents of the Android device's pasteboard. We can get the text copied on the Android device through this method.
def get_android_clipboard(self):
return self.server.control.get_clipboard()set_clipboard method
@inject(const.TYPE_SET_CLIPBOARD)
def set_clipboard(self, text: str, paste: bool = False) -> bytes:The set_clipboard method is an instance method of the ControlSender class used to set the contents of the Android device's pasteboard. The set_clipboard method can receive two parameters. The first parameter text is the text content to be set to the pasteboard; the second parameter paste is the paste state. The default is False. When it is True, the text will be pasted into the input box immediately. (When the cursor of the Android device is in an input box).
def set_android_clipboard(self, text: str, paste=False):
self.server.control.set_clipboard(text, paste)set_screen_power_mode method
@inject(const.TYPE_SET_SCREEN_POWER_MODE)
def set_screen_power_mode(self, mode: int = scrcpy.POWER_MODE_NORMAL) -> bytes:The set_screen_power_mode method is an instance method of the ControlSender class used for the screen power mode of Android devices. The default state is normal, which means turning on the screen power of the Android device. At this time, the screen of the Android device is in a normal state. It can also be set to the off state (scrcpy.POWER_MODE_OFF). At this time, the screen of the Android device is off, but it is not in the screen-off state (the screen power is off and the screen is off are two different things), and the screen can still be seen in the screen projection interface. In this way, the power consumption of the Android device can be reduced when mirroring and controlling the Android device.
def set_screen_power_mode(self, mode=2):
self.server.control.set_screen_power_mode(mode)totate_device method
@inject(const.TYPE_ROTATE_DEVICE)
def rotate_device(self) -> bytes:The totate_device method is an instance method of the ControlSender class used to rotate the screen of the Android device.
def totate_screen(self):
self.server.control.totate_device()swipe method
def swipe(
self,
start_x: int,
start_y: int,
end_x: int,
end_y: int,
move_step_length: int = 5,
move_steps_delay: float = 0.005,
) -> None:The swipe method is an instance method of the ControlSender class used to slide the screen of an Android device. This method is an encapsulation of the touch method, which is equivalent to one-point touch. The swipe method can receive 6 parameters. The first 4 parameters are the starting and ending coordinates of the sliding; the 5th parameter is the step size (the distance of each sliding), which defaults to 5 coordinate units; the 6th parameter is the The pause time for sliding one step, the default is 0.005 seconds.
def swipe_event(self, start_x: int, start_y: int, end_x: int, end_y: int, step: int, delay: float):
self.server.control.swipe(start_x, start_y, end_x, end_y, step, delay)Conclusion
By using the above methods in the UI framework of python, we can realize screencasting control of Android devices. What this screencasting control application looks like is completely determined by your own needs and aesthetics. If you want to operate multiple Android devices at the same time, you can create multiple screen projection control services, and then put these services into a list or dictionary (preferably a dictionary) to switch control devices and control a device individually or at the same time. Purpose of operating multiple devices.