Jump and jump physically hang python+arduino+windows script

When I had nothing to do, I wrote it myself based on the idea of ​​jumping and jumping plug-ins provided on the Internet.

Divided into 3 parts:

1. Run the windows script. The first two instructions of the script are to take a screenshot of the mobile phone screen, then push it to the PC, and specify the storage path by yourself;

2. Run the .py file, use the python image tool matplotlib, process the mouse click event, measure the jump distance before and after, and send it to the serial port;

3. Run the .ion file and use arduino to control the servo to simulate the finger pressing on the phone screen to achieve jumping.

Basically jumping 1000 points is no problem.

The specific implementation is as follows:

1. Install the adb shell, modify the path settings of the computer or open the cmd to the adb path, connect the Android phone, enable usb debugging, use the command to take screenshots, and upload and store them to the computer.

1. adb shell /system/bin/screencap -p /sdcard/screenshot.png % Get the screenshot of the mobile phone and save it to the mobile phone %  
2. adb pull /sdcard/screenshot.png d:/screenshot.png % Upload the screenshot of the mobile phone to the pc, the path can be set by yourself%  

Other adb commands can be queried by themselves.

windows script (.bat):

1. :abc % provides the target   for the goto statement, that is, the following statement is executed in a loop%
2. adb shell /system/bin/screencap -p /sdcard/screenshot.png % Get the screenshot of the mobile phone and save it to the mobile phone %  
3. adb pull /sdcard/screenshot.png d:/screenshot.png %Upload the screenshot of the phone to pc%  
4. python jump.py% execute.py program%  
5. goto  abc % jump back to the first statement, loop execution%  
6. %If there is a program execution error, delete the comment and try again%  

2. Install python2, install the python matplotlib tool, you can use pip to install, the command is python pip -m install  matplotlib.

Python specific implementation (.py):

1. # -*- coding: utf-8 -*
   
2. import math  
3. import matplotlib  
4. import matplotlib.pyplot as plt  
5. import numpy as np  
6. from numpy import *  
7. import Image  
8. import serial  
9. import time  
10. from pylab import *  
11.   
12. img=Image.open( "d:/screenshot.png" , "r" )#The location where the image uploaded using adb is stored, "r" is read-only open  
13. img_gray = img.convert("L")    
14. plt.imshow(img)    
15. [a,b] = ginput(2)#Two mouse click events, two mouse clicks are the current point and the next jump target point, record the coordinates of the two mouse clicks  
16. plt.close()#Close the picture  
17. x=b[0]-a[0]#The difference between the abscissas of two mouse clicks   
18. The ratio of the image size displayed by x=x/173.4#matplotlib to the actual size of the screen, the converted value is the actual abscissa distance between the two points on the screen  
19. y=b[1]-a[1]#The difference between the ordinates of two mouse clicks   
20. y=y/173.4#The ratio of the image size displayed by matplotlib to the actual size of the screen, the converted value is the actual ordinate distance between the two points on the screen  
21. z=x*x+y*y  
22. z=math.sqrt(z)# Calculate the distance according to the Pythagorean theorem  
23. z=z*10#The calculated distance is in centimeters, and multiplied by ten is converted to millimeters  
24. print  int (z) # print out the integer distance  
25. ser=serial.Serial( "COM3" ,9600)# under windows, instantiate the serial port, "COM3", the baud rate is "9600"  
26. #ser=serial.Serial('/dev/ttyUSB0',9600)# Under ubuntu, instantiate the serial port, "/dev/ttyUSB0", the baud rate is "9600"  
27. time.sleep(2)#Wait for 2 seconds. This statement is necessary, without waiting for direct data transmission, there will be problems with serial port reception.  
28. for i in range(0,int(z)):  
29.     ser.write( '1' )#Send z times '1',  
30. time.sleep(7)#Wait for 7 seconds. The operation is to wait for the servo to press the screen and jump to complete the jump.  

The relevant statement function is written in the comments

The purpose of the last statement is to ensure that the jump press has enough time to execute. Since the windows script file is always executed in a loop, it is necessary to wait for the python program and the arduino program to be executed before executing the windows script.

If you run the program successfully, the image above will appear. Then click the mouse twice, click the current position and the next jump target position twice.

Then the shaping distance will be output, and the relevant parameters can be adjusted according to the size of the picture displayed on your mobile phone and computer.

My double jump distance is 26.

After the distance is measured, data will be sent to the serial port. After connecting to the arduino, you can check the serial port number to make sure that the sending serial port is the same as the one used by the arduino, otherwise an error will be reported.

 3. Use arduino to receive data from the serial port, calculate the pressing time according to the distance, and control the servo to press. The arduino UNO and 9g servo are used. The servo has three wires, the red one is connected to 5V, the brown one is grounded, and the orange one is connected to pin 4. Just burn the program to the development board, you don't need to use the serial port debugging assistant to check the current serial port status. If you open the serial port debugging assistant on the arduino ide to check the serial port status, an error will appear in the python program running, and the serial port can only be opened by one program. It is not supported to open and use two programs at the same time.

Code implementation: (.ino)

1. #include <Servo.h>  
2. servo-servo;  
3.   
4. void setup() {  
5.   // put your setup code here, to run once:  
6.   Serial.begin(9600);  
7.   pinMode(4,OUTPUT); //Use pin 4 as the servo signal  
8.   pinMode(2,OUTPUT);  
9.   servo.attach(4);  
10.   servo.write(120); //Servo initialization  
11.   delay(200);  
12. }  
13.    
14. void loop() {  
15.     float i=0;  
16.     int j=0;  
17.     String str1 =  "" ; // Receive data transmitted by serial port, use string to receive  
18.     while (!Serial.available()); //waiting for data  
19.     while  (Serial.available() > 0)   //If the serial port has data, receive it  
20.     {    
21.           str1 +=  char (Serial.read());   //Receive data and put it in the string array  
22.           delay(5); //delay5ms to prevent missing data. In fact, if there is no delay, characters will be lost.  
23.     }    
24.     j=str1.length();  //Determine the length of the string. The length of the string is the distance between two points.  
25.     i=j*7; //i is the delay time of the servo, that is, the time of pressing the screen, and the parameter multiplied by j is adjusted according to the actual situation.  
26.     for ( int  q=140;q>=120;q--) //Servo rotation range  
27.     {  
28.         servo.write(q);  
29.      }  
30.      delay(i); //Press the screen delay  
31.      for ( int  q=120;q<=140;q++) //The servo rotates  
32.      {  
33.         servo.write(q);  
34.      }   
35. }  

After receiving the data, adjust the parameters according to the actual situation. This parameter is the parameter of the linear relationship between the distance and the pressing time.

Final instructions:
1. All my programs run under windows, if you want to run under ubuntu, you can write a shell yourself, python files can be run directly, you can install an arduino ide on ubuntu, you can go to Download from the official website, https://www.arduino.cc/en/Main/Software;

2、python需要安装的其他工具，都可以使用python pip -m install ****安装；

3、三个程序有两个参数需要根据自己实际调整，第一个是.py中的第18、20行，根据手机屏幕实际大小和截图大小，进行比例换算，第二个是.ion中的第25行，根据实际对跳跃距离和按压时间的线性参数进行调整。

4、使用舵机按压手机屏幕时，舵机头可绑上蘸水棉签或者电容笔，同时打开手机手套模式，或者自己探索可以代替手指按压屏幕的物品；