With the development of IIoT, many industrial instruments also have GUIs for remote management. Unlike the earlier use of serial ports for command interaction, these GUIs can directly present data remotely.
As a small company that wants to carry out secondary development of instruments and software, it will encounter the need to convert GUI manual operation to automation. In the case where automatic operation cannot be performed through traditional interfaces such as serial ports, GUI automation can only be considered. In the field of field engineers, the common GUI automation program control technology is mainly divided into two types: native interface and web interface.
- Native form (such as instruments with X-Window Form interface, Qt, etc.). Use the mouse to simulate clicking, Hook and other technologies, and use your own APP to simulate the keyboard, read the control and fill in the data inside the OS of the instrument.
- Web Automation Interaction (Web Page Instrumentation). Use a self-built browser to open the operation interface, and use JS to manipulate page elements to achieve interaction (this article).
Of course, there is a very special class of corner cases, which is the relay operation of physical switches. Relays are used to replace buttons, and the feedback control of buttons and display is achieved by analyzing the LED seven-segment code. This situation requires the use of a soldering iron, which is no longer a problem that can be solved by programming.
Since modern web pages can be operated through JS, in theory, as long as there is a browser that supports the JS control interface, GUI automation can be completed. However, for general browser secondary development, there are still many dependencies that need to be configured. Qt WebEngine is the successor of Qt Webkit. This module allows GUI programming with very simple code.
1. Analyze the composition of the page
Use firefox or chrome browser to open the management page, hit F12 to open the debugging page, and you can find the controls in the viewer.
There are several ways to use this control in the JS Runtime of the browser kernel.
- Direct access using ID. For example, the control in the above picture has its own ID called lgPwd, so you can directly use the ID reference
>> lgPwd
<input id="lgPwd" name="password" type="password" maxlength="32">
>> lgPwd.value="123456";
"123456"
- Use the CSS selector to access
Right-click the selected control part on the right side of the viewer in the above figure, copy the CSS selector,
and use the operation $() to access it in JS.
>> $('#lgPwd')
Object {
0: input#lgPwd, length: 1, context: HTMLDocument http://192.168.1.1/, selector: "#lgPwd" }
>> $('#lgPwd')[0]
<input id="lgPwd" name="password" type="password" maxlength="32">
>> $('#lgPwd')[0].value
"123456"
- Use xPath to locate
Right-click the selected control part on the right side of the viewer in the above figure, copy the XPath selector
and then use the $x picking syntax to enumerate the controls, and get the desired thing in the returned array.
>> $x('//*[@id="lgPwd"]')
Array [ input#lgPwd ]
>> $x('//*[@id="lgPwd"]')[0]
<input id="lgPwd" name="password" type="password" maxlength="32">
>> $x('//*[@id="lgPwd"]')[0].value='234';
"234"
>> $x('//*[@id="lgPwd"]')[0].value
"234"
Once the control that needs to be controlled and the control that needs to be read can be accessed on the console, it can be transferred to the following steps. Why not use JS scripts to run directly in Chrome, or interact with Chrome console through remote operations in Python? Obviously we want our program to appear relatively complete, giving users the look and feel of a "native" application. The above example uses the login interface of the router, but the login interface of the actual instrument is more complicated than this page.
2. Configure Qt WebEngine
As we said just now, we hope to bring the management page of the instrument to its own GUI and run it as a sub-page. This way, the instrument "looks" like it is running in our program. To complete such an operation, you can use Qt WebEngine. Some readers may have doubts, is it possible to use WebKit? In theory it is possible. It's just that WebKit has been removed from Qt's official support. After experiments, Qt Webkit generally supports some new JS features, but WebEngine is very smooth, even real-time graphics refresh is not a problem.
To use QWebEngine under Windows, only MS Visual C++ compiler can be used. This is a big regret of Qt WebEngine. Due to chromium compiler compatibility issues, MINGW is not supported. Fortunately, both VC and Qt have open source versions, and the online installation speed is not bad. We use the online installation of Qt, don't forget to check the "Qt WebEngine" option (not installed by default)
At the same time, make sure that the corresponding VC2019 or other versions of the compiler are installed.
3 Program development
Compared with the old Webkit, QWebEngine operates mostly asynchronously. But its API setting is relatively easy to use. If you need to quickly implement complex functions, you can directly integrate all the framework of the simple_browser example into your own program, and then perform certain tailoring. If you just want to operate a fixed page, you can directly integrate it.
First, import webenginewidgets in the module
QT += core gui
greaterThan(QT_MAJOR_VERSION, 4): QT += widgets webenginewidgets
CONFIG += c++17
The easiest way is to drag in a QWebEngineView. If there is no designer extension, drag in Widget and promote it to this class.
In this way, we can get the browser instance in the code:
ui->web->load(QUrl("http://192.168.1.122"));
When you need to fill in the control and get the return value, you only need to run:
ui->web->page()->runJavaScript(
"lgPwd.value=\"123456\";",
[](const QVariant & result)->void{
qDebug()<<result;
});
output
QVariant(QString, "123456")
It is very important to note that when quotation marks are used in double quotation marks in Qt, they need to be escaped with \. Use single quotes, if the interpreter may not recognize it, try to replace it with double quotes. This is especially the case when using xPath and CSS selectors.
ui->web->page()->runJavaScript(
"$x(\"//*[@id=\"lgPwd\"]\")[0].value=\"234\";",
[](const QVariant & result)->void{
qDebug()<<result;
});
In practical applications, complete feedback control can be achieved by obtaining the status of the webpage and adjusting the values of the interface controls according to the external environment. A typical scenario is to directly control the opening and closing of the telescopic rod on the workshop ceiling according to the output of the environmental detection instrument. Of course, more complex scenarios require debugging.
4. Debugging
QWebEngine can easily enable the in-process debugger,
void WidgetWCtrl::on_pushButton_debug_clicked()
{
QWebEngineView * v = new QWebEngineView(this);
ui->web->page()->setDevToolsPage(v->page());
v->show();
ui->tabWidget->addTab(v,"DEV");
}
At this point, we immediately introduced web development tools into our own program:
5. Release
To publish the program, you need to copy the EXE to a separate folder, and run windeployqt on the cmd command line:
C:\> C:\Qt\6.4.2\msvc2019_64\bin\windeployqt.exe --compiler-runtime C:\projects\WebCtrlBIN\WebCtrl.exe
C:\projects\WebCtrlBIN\WebCtrl.exe 64 bit, release executable
Adding Qt6Svg for qsvgicon.dll
...
C:\>
At this point, there are all the files needed at runtime in the folder:
It should be noted that, in addition to our own EXE, there is an additional QtWebEngineProcess.exe, which is the background process of QWebEngine.
Unlike QtWebKit, this framework still needs the support of an independent executable file behind it.