In pyautogui, if you want to control the mouse, you first have to track the mouse position on the screen, which means you have to know what to do to take a coordinate system in pyautogui in, and that is our common x, y coordinate system.

In my computer is 1920x1080 resolution, for example
Here Insert Picture Description

Through the entire x, y coordinate system can be represented in each position of the entire screen
(0,0) represents the top left corner of the screen, (1919,1079) represents the most bottom right corner of the screen
To the right to increase the x coordinate, y coordinate increases downward
All coordinates are positive integers, no negative coordinates

——

3. good GUI safety measures

You must first know why to make safety measures.

Because Python can move and click the mouse to unimaginable speed, too fast it is likely that other programs can not keep up.
If the program itself is a problem, but the program will still control the mouse tamper everywhere, but its operation is not what you want.

Based on the above common occurrence, so we need to improve some of the security measures in advance to avoid a rollover.
Here Insert Picture Description

A safety measure: pause wait

pyautogui.PAUSE = 1 Each function is called after performing PyAutoGUI operation will wait 1 second; non PyAutoGUI instruction will not stall.

Security Measures II: the fail-safe

pyautogui.FAILSAFE = True Move the mouse to the upper left corner of the screen, which will lead to pyautogui produce pyautogui.FailSafeException abnormal, then processed through a try and except statements to stop the program.

Overall code is as follows:

import pyautogui
pyautogui.PAUSE = 1
pyautogui.FAILSAFE = True

Here Insert Picture Description

Second, move the mouse!

First, we know some PyAutoGui function method, how to make your own mouse move, click, drag and scroll together.
Here Insert Picture Description
-

1. Control mouse movement

[01] pyautogui.size () returns the number of wide screen and high-resolution

import pyautogui
print(pyautogui.size())
width,height = pyautogui.size()
print(width,height)

Result of the program
Here Insert Picture Description
[02] pyautogui.moveTo () /. MoveRel () to achieve the mouse moves

import pyautogui
# pyautogui.moveTo()  绝对位置移动（移动到指定坐标）
for i in range(10):
	pyautogui.moveTo(100,100,duration=0.25)  # duration指定鼠标移到目的位置所需的秒数
	pyautogui.moveTo(200,100,duration=0.25)
	pyautogui.moveTo(200,200,duration=0.25)
	pyautogui.moveTo(100,200,duration=0.25)
	
#pyautogui.moveRel()  相对位置移动（基于目前鼠标位置进行移动）
for i in range(10):
	pyautogui.moveRel(100,0,duration=0.25)
	pyautogui.moveRel(0,100,duration=0.25)
	pyautogui.moveRel(-100,0,duration=0.25)
	pyautogui.moveRel(0,-100,duration=0.25)

This program is up and running Jiang Zi drop! (Note the cursor intermediate)
Here Insert Picture Description

[03] pyautogui.position () returns the current mouse position

import pyautogui
print(pyautogui.position())

Here Insert Picture Description
——

2. Control mouse interaction

[01] pyautogui.click () mouse click

Click the mouse Sao operation there are many!

pyautogui.click()           # 当前位置点击书包
pyautogui.click(100,200,button='left')   # 点击指定坐标鼠标左键，参数有left、middle、right

pyautogui.mouseDown()     # 按下鼠标
pyautogui.mouseUp()       # 释放鼠标

pyautogui.doubleClick()    # 鼠标左键双击
pyautogui.rightClick()     # 鼠标右键单击
pyautogui.middleClick()    # 鼠标中键单击

[02] pyautogui.dragTo () /. DragRel () drag the mouse

Herein may refer to moveTo () / moveRel () Usage

import pyautogui, time
print('5 second til it starts')
time.sleep(5)
pyautogui.click()    # 点击画图工具中获取焦点
distance = 300
shrink = 20
while distance > 0:
    pyautogui.dragRel(distance, 0, duration=0.1)   # 向右移动
    distance = distance - shrink
    pyautogui.dragRel(0, distance, duration=0.1)   # 向下移动
    pyautogui.dragRel(-distance, 0, duration=0.1)  # 向左移动
    distance = distance - shrink
    pyautogui.dragRel(0, -distance, duration=0.1)  # 向上移动

Drawing tools can be opened up and running to try the duck! Is not this hand than their residual much better.
Here Insert Picture Description

[03] pyautogui.scroll () scroll mouse

import pyautogui
pyautogui.scroll(200)     # 正数向上滚动
pyautogui.scroll(-300)    # 负数向下滚动

Here Insert Picture Description

Third, the keyboard to move!

You know how to make your own mouse rotation after the jump, let's look at how to control the keyboard automatically using text entry, keyboard presses and a variety of shortcut keys.
Here Insert Picture Description
-

[01] pyautogui.typewrite () input string information

import pyautogui,time

time.sleep(5)

pyautogui.click((361,266))    # 鼠标在记事本上获得焦点
pyautogui.typewrite('Hello world!','0.25')   # 输入字符串，每个字符之间间隔0.25秒

Try running
Here Insert Picture Description
[02] pyautogui.keyDown () /. KeyUp () /. Press () press the keyboard

import pyautogui
pyautogui.keyDown('shift')       # 按下Shift
pyautogui.press('4')             # 按下并松开4 
pyautogui.keyUp('shift')         # 松开Shift

[03] pyautogui.hotkey () using shortcuts

Sequentially passed shortcut key name string, pyautogui.hotkey(key1,key2,key3,...)in the order given key is pressed and then released in reverse order

import pyautogui
def commentAfterDelay():
	pyautogui.click(100,100)
	pyautogui.typewrite('In IDLE, Alt-3 comments out a line.')
	time.sleep(2)
	pyautogui.hotkey('alt','3')

commentsAfterDelay()

Here Insert Picture Description

Fourth, the screen image recognition

Sometimes, we want to make GUI automation, positioning coordinates alone is not enough, since there may be a variety of reasons we need to locate the target element not on what we consider the coordinates.

Such as software, browser and other interface changes in the size, or suddenly pop out point and so on.

So we need to identify the entire face to face page,By RGB values to determine whether the element is consistent with what we expectSo from either find the target object.

This is a screen processing and image recognition mechanisms in pyautogui in.

——

1. processing screen

Here are three main elements method

pyautogui.screenshot() Get a snapshot of the current screen Image object
gepixel((x,y)) Return (x, y) coordinates in RGB tuple
pyautogui.pixelMatchesColor() Analyzing RGB coordinates match

import pyautogui

im = pyautogui.screenshot()   # 获得屏幕快照
im.getpixel((50,200))    # 返回该坐标的RGB
# RGB（130,135,144）

pyautogui.pixelMatchedColor(50,200,(130,135,144))  # 验证指定坐标是否与参数中的RGB值相匹配
# True

pyautogui.pixelMatchedColor(50,200,(131,135,144))  # 验证指定坐标是否与参数中的RGB值相匹配
# False  就算一小点的不一致，都会无法匹配

——

2. Image Recognition

If we do not know where the mouse clicks, you can provide an image of hope to PyAutoGui clicking through image recognition, let it find the coordinates of the current screen.

pyautogui.locateOnScreen() Returns the image on the screen of the upper left corner of the first discovered x, y coordinates, and the image width and height (tuple); None Return if none found; if a plurality of found, by List (), which returns the tuple list
pyauto.center() Returns the center coordinates of the image

import pyautogui
list(pyautogui.locateOnScreen('submit.png'))   # 返回屏幕上发现该图像的顶点坐标和宽高
#（643,745,70,29）
pyautogui.center((643,745,70,29))      # 返回图像的中心坐标
#（678,759）
pyautogui.click((678,759))  # 鼠标左键点击

Here Insert Picture Description

V. Replay summary

First, get a clear understanding of x, y coordinates, to know how the mouse on the desired position
Two security measures must not forget: pause and wait for fail-safe
The method of moving the mouse up: how to determine the current coordinates, absolute movement, relative movement, the various operations click, drag the mouse, the mouse sliding
The method of moving the keyboard up: how to enter text, how to simulate keyboard keys, or even the use of shortcut keys operation
More needs to think about how the GUI automation combined with practice, making work life more convenient

Here Insert Picture Description