App screenshot + recognize text in the screenshot

In the process of automated testing, we often encounter scenarios that require text recognition, such as identifying verification codes, recognizing text in screenshots, reading values in screenshots, etc. How can we deal with these situations?

This machine must have a PaddleOCR environment, PaddleOCR can refer to my other article

PaddleOCR , image detection and recognition

Open our AirtestIDE, and 选项--设置--自定义python.exe路径 set the python environment we just installed the corresponding library in:

Full Screen Capture + Recognition

On the airtest client, enter the following script after connecting to the real machine

# -*- encoding=utf8 -*-
__author__ = "AirtestProject"

from airtest.core.api import *
from airtest.aircv import *
from PIL import Image
from paddleocr import PaddleOCR, draw_ocr
from PIL import Image
 

auto_setup(__file__)
screen = G.DEVICE.snapshot()

# # 局部截图
# screen = aircv.crop_image(screen,(301,502,372,540))

# 保存局部截图到指定文件夹中
pil_image = cv2_2_pil(screen)
pil_image.save("E:/airtestLog/1.png", quality=99, optimize=True)

# 读取截图并识别截图中的文字

ocr = PaddleOCR(use_angle_cls=True, lang="ch") # need to run only once to download and load model into memory
img_path = r'E:/airtestLog/1.png'
result = ocr.ocr(img_path, cls=True)
for line in result:
    print(line)
 
image = Image.open(img_path).convert('RGB')
boxes = [line[0] for line in line]
# print(boxes)
txts = [line[1][0] for line in line]
# print(txts)
scores = [line[1][1] for line in line]
# print(scores)
im_show = draw_ocr(image, boxes, txts, scores, font_path='/path/to/PaddleOCR/doc/simfang.ttf')
im_show = Image.fromarray(im_show)
im_show.save('result.jpg')

Result display

Partial screenshot + recognition

Now I want to read partial text, identifying verification codes is a common scenario

# 局部截图
screen = aircv.crop_image(screen,(301,502,372,540))

Result display

You can also use Tesseract, another open source ocr software, for partial screenshots. For reference, 2 lines of code help you get the text recognition of the automated test. You need to do verification code recognition and read the screenshot text. Check it out~ https://mp.weixin.qq .com/s/mrx2fndE9t_477yViZrpRA

If Tesseract reports an error Python tesseract is not installed or it's not in your path, change the beginning of the source code tesseract_cmd to the local installation directory