foreword

Use the target detection algorithm to realize the acquisition of the target position of the game interface
PyKeyboard, ctypes realize mouse and keyboard control
implement specific operations

1. Target detection algorithm

Take a screenshot of the game interface, label the specific target with Labelimg, and train a target detection algorithm model

2. Operation game process

1. Get the screen interface image

def grab_screen(region=None):

    hwin = win32gui.GetDesktopWindow()

    if region:
        left, top, x2, y2 = region
        width = x2 - left + 1
        height = y2 - top + 1
    else:
        width = win32api.GetSystemMetrics(win32con.SM_CXVIRTUALSCREEN)
        height = win32api.GetSystemMetrics(win32con.SM_CYVIRTUALSCREEN)
        left = win32api.GetSystemMetrics(win32con.SM_XVIRTUALSCREEN)
        top = win32api.GetSystemMetrics(win32con.SM_YVIRTUALSCREEN)


    hwindc = win32gui.GetWindowDC(hwin)
    srcdc = win32ui.CreateDCFromHandle(hwindc)
    memdc = srcdc.CreateCompatibleDC()
    bmp = win32ui.CreateBitmap()
    bmp.CreateCompatibleBitmap(srcdc, width, height)
    memdc.SelectObject(bmp)
    memdc.BitBlt((0, 0), (width, height), srcdc, (left, top), win32con.SRCCOPY)

    signedIntsArray = bmp.GetBitmapBits(True)
    img = np.fromstring(signedIntsArray, dtype='uint8')
    img.shape = (height, width, 4)

    srcdc.DeleteDC()
    memdc.DeleteDC()
    win32gui.ReleaseDC(hwin, hwindc)
    win32gui.DeleteObject(bmp.GetHandle())

    return cv2.cvtColor(img, cv2.COLOR_BGRA2BGR)

2. Get the game window interface image

def get_screan():
    # hwnd = win32gui.FindWindow(None, 'C:\Windows\system32\cmd.exe')
    app = QApplication(sys.argv)
    hwnd = win32gui.FindWindow(None, 'window') # window 窗口名称
    # win32gui.SetForegroundWindow(hwnd)
    rect = win32gui.GetWindowRect(hwnd)
    # print('aaaaaaa', hwnd)
    screen = QApplication.primaryScreen()
    img = screen.grabWindow(hwnd).toImage()
    # img.save("screenshot.jpg")
    arr = convertQImageToMat(img)
    return cv2.cvtColor(arr, cv2.COLOR_BGRA2BGR), rect

3. Mouse and keyboard operation

Tried many methods, this method can successfully operate the mouse and keyboard on the game interface

def mouse_click(x, y, n=1):
    # x, y鼠标在窗口中的坐标
    # win32api.SetCursorPos([x, y])
    ctypes.windll.user32.SetCursorPos(int(x), int(y))
    for i in range(n):
        # win32api.mouse_event(win32con.MOUSEEVENTF_MOVE, x, y)
        win32api.mouse_event(win32con.MOUSEEVENTF_LEFTDOWN, 0, 0, 0, 0)
        win32api.mouse_event(win32con.MOUSEEVENTF_LEFTUP, 0, 0, 0, 0)

3. Verification code identification

Number and letter verification code recognition, the accuracy is very good, but a 32-bit python is required to call the E language library

class Ver_code_2:
    def __init__(self, path):
        # 载入识别库
        self.dll = windll.LoadLibrary(path + r"\OCRS.dll")
        # 载入字库与建立字库索引
        with open(os.path.join(path, r"zimushuziku.cnn"), "rb") as file:
            # 载入字库
            self.word_bank = file.read()
            # 建立字库索引
            self.word_index = self.dll.INIT(path, self.word_bank, len(self.word_bank), -1, 1)

    def ocr(self, image):
        Str = create_string_buffer(100)  # 创建文本缓冲区
        self.dll.OCR(self.word_index, image, len(image), Str)  # 利用DLL中的识别函数进行识别
        # print(Str)
        return Str.raw.decode("utf-8").rstrip('\x00')  # 对识别的返回值进行编码后返回，这里的\x00是删除缓冲区的空白符

Summarize

Recognition depends on the accuracy of the target detection algorithm, and the logic processing in the operation process needs to be rigorous

AI Game Playing Based on Deep Learning Object Detection