Live translation screen plugin

The function of the program plug-in is: click the button, a screenshot will be taken, and the output image will be translated. (Currently only supports translation in English)

To achieve this function, we can use the Python programming language, combined with some libraries. Here is a simple implementation:

1. Use pywin32the library to create a sticky window and add buttons.
2. Use pyautoguithe library to take screenshots.
3. Use opencv-pythonlibraries for image manipulation.
4. Use pytesseractthe library to recognize English text in screenshots.
5. Use googletransthe library to translate English to Chinese.
6. Use PILthe library to add translated Chinese to the screenshot and display it on the screen.

First, make sure the required libraries are installed:

pip install pywin32 pyautogui opencv-python pytesseract googletrans==4.0.0-rc1 pillow

 You can then implement the plugin functionality with the following code:

import cv2
import pytesseract
from googletrans import Translator
from PIL import Image
import pyautogui
import win32gui
import win32con
import win32ui
from ctypes import windll
import sys
import os

def create_top_button():
    wnd = win32gui.CreateWindowEx(
        win32con.WS_EX_TOPMOST,
        win32gui.RegisterClass(win32gui.WNDCLASS()),
        'Translate Button',
        win32con.WS_OVERLAPPEDWINDOW | win32con.WS_VISIBLE,
        100,
        100,
        150,
        50,
        None,
        None,
        win32gui.GetModuleHandle(None),
        None
    )

    btn_translate = win32gui.CreateWindow(
        'BUTTON',
        'Translate',
        win32con.WS_CHILD | win32con.WS_VISIBLE,
        30,
        10,
        80,
        25,
        wnd,
        1000,
        win32gui.GetModuleHandle(None),
        None
    )

    def on_click(hwnd, msg, wparam, lparam):
        if wparam == 1000:
            capture_and_translate()

    win32gui.SetWindowLong(wnd, win32con.GWL_WNDPROC, on_click)
    win32gui.ShowWindow(wnd, win32con.SW_SHOW)
    win32gui.UpdateWindow(wnd)
    win32gui.PumpMessages()

def capture_and_translate():
    screenshot = pyautogui.screenshot()
    screenshot.save('screenshot.png')

    img = cv2.imread('screenshot.png')
    gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    text = pytesseract.image_to_string(gray, lang='eng')

    translator = Translator(service_urls=['translate.google.com'])
    translated_text = translator.translate(text, dest='zh-CN').text

    img_pil = Image.fromarray(cv2.cvtColor(img, cv2.COLOR_BGR2RGB))
    draw = ImageDraw.Draw(img_pil)
    draw.text((10, 10), translated_text, font=ImageFont.truetype('arial.ttf', 20), fill=(0, 0, 255, 0))
    img_pil.show()

    os.remove('screenshot.png')

if __name__ == '__main__':
    pytesseract.pytesseract.tesseract_cmd = r'C:\Program Files\Tesseract-OCR\tesseract.exe'  # 请根据您的tesseract安装路径进行修改
    create_top_button()

NOTE: Make sure to tesseract_cmdset the variable to the correct path where Tesseract-OCR is installed on your system. After running this code, a sticky "Translate" button will be created on the screen. When you click this button, the code will do the following:

1. Use pyautoguithe library to take screenshots and save them as screenshot.png.
2. Use opencv-pythonthe library to convert screenshots to grayscale images for optimized text recognition.
3. Use pytesseractthe library to recognize English text in screenshots.
4. Use googletransthe library to translate English text to Chinese.
5. Use PILthe library to add translated Chinese text to the screenshot and display it.

When the plugin is no longer needed, the program can be terminated by closing the "Translate Button" window. This code only provides a basic implementation, which still needs to be optimized and perfected.