Article directory
Official website : http://sikulix.com/
Github : https://github.com/sikuli/sikuli
Tutorial :
-
https://sikulix-2014.readthedocs.io/en/latest/genindex.html
What is SikuliX?
SikuliX (Sikuli) automates everything you see on the screen of a desktop computer running Windows, Mac or some Linux/Unix. It uses the image recognition function provided by OpenCV to identify GUI components. This is handy in situations where you don't have easy access to the inside of the GUI or the source code of the application or web page you want to manipulate.
SikuliX supports scripting language
- Python language level 2.7 (supported by Jython)
- Support for running RobotFramework text scripts
- Ruby language levels 1.9 and 2.0 (supported by JRuby)
- JavaScript (supported by Java script engine)
...and you can use it in Java programming and programming/scripting in any Java-aware programming/scripting language (Jython, JRuby, Scala, Clojure, etc.).
In addition to positioning images on the screen, SikuliX can also operate mouse and keyboard to interact with identified GUI elements. It works well in multi-monitor environments and even on remote systems with some limitations.
SikuliX comes with Basic Text Recognition ( OCR ) which can be used to search for text in images. This feature is supported by Tesseract.
SikuliX is a Java application that runs on Windows, macOS and most Linux/Unix systems.
SikuliX supports scripting and programming . This article only introduces scripting. Programming will be discussed later with selenium .
The latest stable version is 2.0.5 (current time 2023.06.19)
SikuliX usage scenarios
You want to automate some repetitive tasks
- Daily use of the application or web page
- play games
- Management of IT systems and networks
...and you don't have enough tools at hand.
You want to test an application or web page you are developing.
You want to create how-to documentation or training material that runs live on a given application or web page.
You already have the appropriate tools and workflows, but you want to add SikuliX specific features to enhance your methods and improve efficiency and results.
Install SikuliX
1. Install JDK64 bit (version 8 or later)
2. Download SikuliIDE , different operating systems choose different packages to download.
-
Windows: Download the ready to use sikulixide-2.0.5-win.jar (SikuliX IDE for Windows)
-
macOS: Download the ready to use sikulixide-2.0.5-mac.jar (SikuliX IDE for macOS)
-
Linux: Download the ready to use sikulixide-2.0.5-lux.jar (SikuliX IDE for Linux)
common method
The Screen class is the base class for all methods provided by Sikuli. The Screen class contains predefined methods for all common operations on screen elements like single click, double click, providing input to a textbox, hovering, etc. Below is a list of commonly used methods provided by the Screen class.
Since class Screen extends the class Region, all methods of class Region are available to Screen objects.
Note that using the entire screen for seek operations has a performance impact. So, if possible, limit lookup operations to smaller region objects (eg
reg.find()) to speed things up.
look up
| method | describe |
|---|---|
| find | This method is used to find a specific element on the screen. It accepts image name as parameter. |
| findAll | Searches the entire screen for all matching visual patterns and returns a list of the locations of these similar patterns. This function allows us to get all selected items on the screen. |
| wait | Wait for the specified pattern to be loaded. |
| wait | Wait a few seconds. |
| waitVanish | Keep searching for the specified pattern in the area until the image disappears |
| exists | |
| sleep | Let the computer rest for a while, and don't rush to execute the following statement. in seconds |
mouse
| Order | describe |
|---|---|
| click | left click on element |
| rightClick | right click on element |
| hover | The mouse hovers over the specified element |
| dragDrop | Drag, dragDrop("1606202686079.png", "1606202694506.png") #Drag the first element to the second element position |
| paste | Text input (recommended, simply copy your content and paste it) |
keyboard
| button | Order |
|---|---|
| Shift | type(Key.SHIFT) |
| Ctrl | type(Key.CTRL) |
| Alt key | type(Key.ALT) |
| Tab (tab) | type(Key.TAB) |
| caps Lock | type(Key.CAPSLOCK) |
| enter | type(Key.ENTER) |
| delete | type(Key.DELETE) |
| backspace | type(Key.BACKSPACE) |
| insert | type(Key.INSERT) |
| Home | type(Key.HOME) |
| End | type(Key.END) |
| ESC | type(Key.ESC) |
| space | type(Key.SPACE) |
| Windows | type(Key.WIN) |
| change page | type(Key.PAGE_UP) |
| page down | type(Key.PAGE_DOWN) |
| print screen | type(Key.PRINTSCREEN) |
| Numeric keypad lock | type(Key.NUM_LOCK) |
| scroll lock | type(Key.SCROLL_LOCK) |
| F1 | type(Key.F1) |
| F2 | type(Key.F2) |
| F3 | type(Key.F3) |
| F4 | type(Key.F4) |
| F5 | type(Key.F5) |
| F6 | type(Key.F6) |
| F7 | type(Key.F7) |
| F8 | type(Key.F8) |
| F9 | type(Key.F9) |
| F10 | type(Key.F10) |
| F11 | type(Key.F11) |
| F12 | type(Key.F12) |
| Cursor keys: up | type(Key.UP) |
| Cursor keys: down | type(Key.DOWN) |
| Cursor keys: left | type(Key.LEFT) |
| Cursor keys: right | type(Key.RIGHT) |
| move the cursor up twice | type(Key.UP * 2) |
| move the cursor down twice | type(Key.DOWN * 2) |
| move cursor left twice | type(Key.LEFT * 2) |
| move cursor right twice | type(Key.RIGTH * 2) |
| press a specific key | KeyDown(Key.*) needs to use Region() for locale setting. |
| release specific key | KeyUp(Key.*) needs to use Region() to set the region. |
operator
| operation | Order |
|---|---|
| addition | + |
| subtraction | – |
| multiplication | * |
| remove | / |
| remainder calculation | ％ |
| the integer part of the divisor | // |
| A to the bth power | A ** B |
| Negative of A | -A |
| logical or | or |
| logic and | and |
| deny | not |
| a is greater than b | a > b |
| a is less than b | a < b |
| a is less than or equal to b | a <= b |
| a is greater than or equal to b | a >= b |
| a is equal to b | a == b |
| a is not equal to b | a != b |
hot key
| common operation | hot key | Order |
|---|---|---|
| copy | Ctrl + C | type(“c” , Key.CTRL) |
| cut out | Ctrl + X | type(“x” , Key.CTRL) |
| 贴上 | Ctrl + V | type(“v” , Key.CTRL) |
| 搜索 | Ctrl + F | type(“f” , Key.CTRL) |
| 打印 | Ctrl + P | type(“p” , Key.CTRL) |
| 撤消 | Ctrl + Z | type(“z” , Key.CTRL) |
| 重做 | Ctrl + Y Ctrl + Shift + Z | type(“y” , Key.CTRL); type(“z” , Key.CTRL + Key.SHIFT) |
| 关闭标签 | Ctrl + F4 | type(Key.F4 , Key.CTRL) |
| 还原标签 | Ctrl + Shift + T | type(“t” , Key.CTRL + Key.SHIFT) |
| 浏览器缓存刷新(强制刷新) | Ctrl + F5 | type(Key.F5 , Key.CTRL) |
| 关闭窗口 | Alt + F4 | type(Key.F4 , Key.ALT) |
| 最小化所有窗口 | Windows + M | type(“m” , Key.WIN) |
| 显示桌面 | Windows + D | type(“d” , Key.WIN) |
| 最小化窗口(当窗口为正常大小时)恢复为原始大小(当窗口为最大化时) | Windows + 光标键：向下,如果最大化,请执行上述两次。 | type(Key.DOWN , Key.WIN) |
| 窗口最大化 | Windows +光标键：向上 | type(Key.UP , Key.WIN) |
| 切换活动窗口 | Windows + Tab; Windows + Shift + Tab | type(Key.TAB , Key.WIN); type(Key.TAB , Key.WIN + Key.SHIFT) |
| 启动任务管理器 | Ctrl + Shift + ESC | type(Key.ESC , Key.CTRL + Key.SHIFT) |
| 启动资源管理器 | Windows + e | type(“e” , Key.WIN) |
其他
打开应用
#打开应用程序，注意应用程序的路径名里的斜杠“\”要改为双斜杠“\\”。
openApp(应用程序路径名)
#closeApp
#switchApp
If语句
if 条件 and 条件: #（注意有冒号，‘#’号表示注释）
语句
循环语句
for x inrange(10): #循环10次，x不用预先定义
循环体
弹出输入框
# 显示一个输入框，以供输入。要输入中文应该在前面加u,不然会乱码
input(u"字符串")
弹出提示框
popup("Obama has updated his message")
操作文件
#打开 写文件
f=open('c:/wy.txt','a')
t="hello\n"
f.write(t)
f.close()
#打开 读文件
f=open('c:/wy.txt','r')
t=f.read()
popup(t)
#open(路径+文件名,读写模式)
#读写模式:r只读,r+读写,w新建(会覆盖原有文件),a追加,b二进制文件.常用模式
示例脚本
示例一 自动抢票
流程：
1.打开浏览器输入抢票的网址
2.循环点击抢票按钮
这里用登入按钮代替抢票按钮哈。
脚本截图：
脚本代码：
n = 10
find("1687314581241.png")
for i in range(n):
click(Pattern("1687314614204.png").targetOffset(116,-4))
sleep(1)
print "done"
示例二 自动打开计算器
流程:
1.打开Windows搜索
2.输入calc
3.输入1+1
4.按=进行计算
脚本截图：
脚本代码：
click("1687331888480.png")
paste("calc")
type(Key.ENTER)
type(Key.ENTER)
wait(2)
paste("1+1")
wait(1)
click("1687332030574.png")
print("done")
示例三 自动访问CSDN博客搜索博主并关注
流程：
1.在桌面找到Chome浏览器
2.双击Chome浏览器
3.sleep 2秒
4.等待Chrome浏览器打开
5.敲击回退键
6.敲击https://www.csdn.net/
7.敲击ENTER键
8.敲击ENTER键
9.sleep 4秒等待浏览器加载内容
10.等待csdn加载完成
11.敲击lakernote
12.敲击ENTER键
13.敲击ENTER键
14.等待搜索完成
15.鼠标点击用户
16.鼠标点击关注
17.关闭浏览器
脚本截图：
脚本代码：
find("1687269549891.png")
doubleClick(Pattern("1687269549891.png").targetOffset(3,-21))# 点击指定图案的指定位置
sleep(2)
wait(Pattern("1687269907253.png").similar(0.60)) # 指定图案命中相似度为0.6，更容易命中
type(Key.BACKSPACE)
type("https://www.csdn.net/")
type(Key.ENTER)
type(Key.ENTER)
sleep(4)
wait("1687270102974.png")
type("1687270102974.png","lakernote")
type(Key.ENTER)
type(Key.ENTER)
sleep(2)
click("1687270231840.png")
sleep(2)
click("1687270273555.png")
sleep(3)
click("1687270623962.png")
其他
find("png") # 在整个电脑屏幕的范围内去找图标
r = Region("png")
r.find("xx") # 在指定的范围内找图标
if exists("查看的内容图片"):
click("查看的内容图片")
print"断言的内容"
else:
print("断言的内容")
for x in findAll(png):
click(x)
while exists(png):
sleep(5)
popup("vgod just logged off")
while not find(png):
sleep(1)
else:
print "not found", png
keyDown((Key.CTRL) # 一直按着
keyUp(Key.CTRL) # 松开
# 方位
Nearby(number)、above()、below()、left()、right()、getCenter()
SikuliX原理
https://sikulix-2014.readthedocs.io/en/latest/devs/system-design.html
SikuliX脚本
Sikuli Script（Sikuli脚本）是一个基于Jython和Java的库，可以使用图像模式来自动化GUI交互，通过图像模式来指定键盘/鼠标事件。Sikuli Script的核心是一个Java库，由两部分组成：
- java.awt.Robot（用于向适当位置发送键盘和鼠标事件的类）
- 以及基于OpenCV的C++引擎（用于在屏幕上搜索给定的图像模式）
C++引擎通过JNI与Java连接，并且需要为每个平台进行编译。在Java库的基础上，为最终用户提供了一个简单明了的命令集作为一层薄薄的Jython封装。因此，很容易为在JVM上运行的其他语言（如JRuby、Scala、Javascript等）添加更多薄薄的封装层。
Sikuli 源文件夹或压缩文件（.sikuli、.skl）的结构
Sikuli脚本（.sikuli）是一个目录，其中包含一个表示自动化工作流程或测试用例的Python源文件（.py），以及源文件使用的所有图像文件（.png）。Sikuli脚本中使用的所有图像只是一个指向.sikuli捆绑包中.png文件的路径。因此，Python源文件也可以使用任何文本编辑器进行编辑。
在使用Sikuli IDE保存脚本时，可以选择在.sikuli目录中创建一个额外的HTML文件，以便用户可以轻松地在Web上共享脚本的可视副本。
Sikuli压缩脚本（.skl）只是.sikuli文件夹中所有文件的压缩文件。它适用于通过电子邮件或Web上传进行分发，也可以从命令行运行并在Sikuli IDE中重新打开。（以前称为“Sikuli可执行文件”已被弃用，因为这样命名是误导性的：人们通常会认为它类似于一个自包含和自运行的软件包，类似于Windows的EXE文件，但实际上不是这样的）
缺点
- 依赖屏幕截图，使其在不同的操作系统上，不同的浏览器中，甚至是不同的显示分辨率下，需要独立维护一套图形源文件，对于其跨平台的能力造成障碍。
- 由于对于截图的检索依赖于实时的桌面显示，若出现程序逻辑之外的意外界面遮挡或焦点切换（如，弹出窗口等），则会对程序执行造成影响。
- IDE 处于开发初级阶段，稳定性和易用性存在一定问题。其本身仅支持初级的代码编辑功能，对于规模较大的代码开发和调试工作仍存在不便之处。其在 Windows 平台和 Linux 平台上的运行稳定性也稍逊于在 MacOSX 上的表现。