Recently want to do a little scripting tools to facilitate the write operation documentation, its function is very simple, that is, after the script open, click mouse operation will be performed after clicking on the screenshot, and mark the position clicked on the map, somewhat similar to screen recording software version of the picture, so, if it is to use documented procedures for some systems, it opens the script meal operation, after completing the operation, every step is automatically recorded, and convenience. The last tool is made, but the middle of the exploration process is not smooth, so here recording his thoughts and problem-solving process.
The general idea: Click the left mouse button, click can get coordinates, and screenshots after the click, and then after picture processing library mouse click position to add some kind of mark, the idea is very simple, began to consider the implementation.
First, this recording tool, my first thought was, python + selenium because selenium has a handy screenshot function, and before touching will be more, it may be convenient to use gas, then began. The first problem encountered is how to obtain the position coordinates of the mouse click click, I realized that selenium is the code to control the behavior of the browser, but can not monitor the behavior of the browser, this is not it was originally designed, the into difficulties. Then think Although selenium can not monitor events in the browser, but js code can be, so to find a solution, that is to write a js to monitor the operation of mouse clicks, and then the return value custom attributes a div is stored in the page up, and then continue to look for the property values in the div selenium until updated coordinates, then subsequent processing. After the meal debugging, code was finally able to return to normal coordinate values, but also dealt with some special cases, such as point to open a new tab, close the tab and a series of operations, with a try except captured exception handling, and finally able to return relatively robust click coordinates.
With the coordinates, then study image processing, with a PIL library, the idea is to open shots, shot coordinate position drawn on a transparent circle, re-save, the idea is quite simple, but when it came to painting transparent circle a little difficult - obviously using a translucent color when painting the picture to save, it is no longer transparent circle, and study a little, and finally
Image.composite () method to solve, was finally able to put up more perfect circle with a transparent map .
Follow-up in the debugging process, we discovered an obscure function of selenium, driver.execute_async_script (), because the general execution js script, then use driver.execute_script () is executed, and later checked driver.execute_async_script () usage, I found it to be executed asynchronously, that is, to receive js execution of the return value, and if the return value is received to continue down, otherwise an error after waiting 30s, so I can get pass values directly coordinate with js to the webdriver selenium, so I removed the intermediate mass value of div, streamlined code, but also left a previous version in order to compare performance.
Then began formal testing gadgets you, but also new problems began ---- operation from the beginning of our background. Interface login normal, but after the board into the system, navigation section and click interface can return to normal coordinates, but the main area of the screen and click on the coordinate values can not return, and later found, because there iframe page, but webdriver not switched to iframe in, so get click coordinates less than an iframe, but if I switch to an iframe, they can not return to the coordinates of the navigation section, this issue really is too much trouble, I thought for a moment whether we can do this thing with two threads , that is a main area of coordinate monitoring, iframe in another process monitor coordinates? But also a meal modify, and process synchronization problems, and finally ended in failure, because an open browser corresponds only to drive a webdriver, even if there are two threads, one of which is switched to the iframe, then another because the same reference webdriver, it will also to monitor iframe, so this method does not work.
Still later, maybe I should think of another idea, then the more complex pages with selenium, the code will be more complicated, so we converted an idea, abandoned selenium, monitoring mouse clicks directly at the system level, behind the picture processing can be reused, along this line of thought to find python related packages found pyHook monitor mouse can be achieved, but looked under the installation a little complicated, I thought there must be alternatives, and finally found pynput, you can achieve the perfect mouse control, and then after Screenshot tool to find this more, read other people's blog after the discovery pyautogui is relatively good, so this equipment, as well as the use of a small episode, found pyautogui screenshot function always being given, there is no relevant answers online, and later observed source code is found, an error is introduced when dependencies, leading feature is not available, why not go into error (finding may be due to the introduction of the package generated when the last path diagonals are reversed), but they put the code that stick out reference manual once normal. Also found in this process, the original pyautogui screenshot feature is completely dependent on another obscure package pyscreeze, pyautogui function is very powerful, it can be said that system-level selenium, can achieve a variety of clicks, typing operation, but now I am it is used only fast screenshot function, so I will not use pyautogui, and go directly pyscreeze shots, the result is very perfect and streamlined, the last meal stitching debugging, finally completed this screenshot tool demo, code posted record it with the version python3.5.2.
import time
from PIL import Image, ImageDraw
import os
from pynput import mouse
from pynput.mouse import Button
import pyscreeze
# import pyautogui
# from pyscreeze import (center, grab, locate, locateAll, locateAllOnScreen,
# locateCenterOnScreen, locateOnScreen, pixel, pixelMatchesColor,
# screenshot)
def picture_draw(path, locate):
oriImg = pyscreeze.screenshot()
# Img.save(path)
# oriImg = Image.open(path)
maskImg = Image.new('RGBA', oriImg.size, (0, 0, 0, 0))
draw = ImageDraw.Draw(maskImg)
draw.ellipse(locate, fill=(255, 255, 0, 100))
final = Image.composite(maskImg, oriImg, maskImg)
final.save(path)
def on_move(x, y):
pass
# print('Pointer moved to {o}'.format((x,y)))
i = 1
def on_click(x, y, button, pressed):
global i
# button_name = ''
# print(button)
if button == Button.left:
button_name = 'Left Button'
elif button == Button.middle:
button_name = 'Middle Button'
elif button == Button.right:
button_name = 'Right Button'
else:
button_name = 'Unknown'
if pressed:
if button == Button.left:
button_name = 'Left Button'
picture_path = os.path.abspath('.') + '\\picture%s.png' % str(i)
picture_draw(picture_path, (int(x) - 25, int(y) - 25, int(x) + 25, int(y) + 25))
i += 1
print('{0} Pressed at {1} at {2}'.format(button_name, x, y))
else:
# print('{0} Released at {1} at {2}'.format(button_name, x, y))
pass
if not pressed:
return False
def on_scroll(x, y, dx, dy):
# print('scrolled {0} at {1}'.format(
# 'down' if dy < 0 else 'up',
# (x, y)))
pass
while True:
with mouse.Listener(no_move=on_move, on_click=on_click, on_scroll=on_scroll, suppress=False) as listener:
listener.join()
There are a lot of redundancy and imperfect code, we'll keep it, after possible to use, especially the powerful pyautogui. Thank the great God who blog, the main body of this code still retain your style.
To sum up, this gadget is himself when reading thought, wanted to achieve it, the implementation process has a lot of experience, the following three points: 1, only skilled with good js to make selenium functions to get the most of it, that will do almost anything selenium, so if you want to write automation code, the use of what js, not only to practice js, and make the code more robust. 2, after that python wonderful world, as long as you think of the function, there will be a corresponding library support, and more than one, we want to pick the most simple and practical most of the library (pynput), instead of writing on the blog up (pyHook). 3, the last great a feeling to do this function, began to take a detour, with selenium to solve, but because of a profound experience to the "Failure is the mother of success," I do not take a detour, but ruled out a wrong option, so I'm closer from the truth, so after experiencing a problem, the first solutions which do not have too much to consider more perfect, or take a detour to worry about a waste of time, in fact, how much waste will not be wasted, even though it wants to do, not just written procedures oh.