Chapter 5: Writing Benchmarks
How to write benchmark tests.
In the last chapter, we will demonstrate how to use Qt testing to write benchmark tests.
Benchmarks
In order to create benchmarks, we use the QBENCHMARK macro to extend the test function. The benchmark function usually consists of the setup code and the QBENCHMARK macro, which contains the code to be measured. This test function benchmarks QString::localeAwareCompare().
// benchmarking.cpp
#include <QtWidgets>
#include <qtest.h>
class TestBenchmark : public QObject
{
Q_OBJECT
private slots:
void simple();
void multiple_data();
void multiple();
void series_data();
void series();
};
void TestBenchmark::simple()
{
QString str1 = QLatin1String("This is a test string");
QString str2 = QLatin1String("This is a test string");
QCOMPARE(str1.localeAwareCompare(str2), 0);
QBENCHMARK {
str1.localeAwareCompare(str2);
}
}
void TestBenchmark::multiple_data()
{
QTest::addColumn<bool>("useLocaleCompare");
QTest::newRow("locale aware compare") << true;
QTest::newRow("standard compare") << false;
}
void TestBenchmark::multiple()
{
QFETCH(bool, useLocaleCompare);
QString str1 = QLatin1String("This is a test string");
QString str2 = QLatin1String("This is a test string");
int result;
if(useLocaleCompare)
{
QBENCHMARK {
result = str1.localeAwareCompare(str2);
}
} else {
QBENCHMARK {
result = (str1 == str2);
}
}
Q_UNUSED(result);
}
void TestBenchmark::series_data()
{
QTest::addColumn<bool>("useLocaleCompare");
QTest::addColumn<int>("stringSize");
for(int i = 1; i < 10000; i += 2000)
{
QByteArray size = QByteArray::number(i);
QTest::newRow(("locale aware compare--" + size).constData()) << true << i;
QTest::newRow(("standard compare--" + size).constData()) << false << i;
}
}
void TestBenchmark::series()
{
QFETCH(bool, useLocaleCompare);
QFETCH(int, stringSize);
QString str1 = QString().fill('A', stringSize);
QString str2 = QString().fill('A', stringSize);
int result;
if(useLocaleCompare) {
QBENCHMARK {
result = str1.localeAwareCompare(str2);
}
} else {
QBENCHMARK {
result = (str1 == str2);
}
}
Q_UNUSED(result);
}
QTEST_MAIN(TestBenchmark)
#include "benchmarking.moc"
answer
void TestBenchmark::simple()
{
QString str1 = QLatin1String("This is a test string");
QString str2 = QLatin1String("This is a test string");
QCOMPARE(str1.localeAwareCompare(str2), 0);
QBENCHMARK {
str1.localeAwareCompare(str2);
}
}
The setting can be completed at the beginning of the function, and the clock does not run at this time. The code in the QBENCHMARK macro will be measured. In order to get accurate measurement results, it may be repeated several times.
There are several backends available:
- Walltime
- CPU tick counter
- Event Counter
- Valgrind Callgrind
- Linux Perf
Data function
Data functions are very useful for creating benchmarks that compare multiple data inputs, such as supporting locale comparisons and standard comparisons.
void TestBenchmark::multiple_data()
{
QTest::addColumn<bool>("useLocaleCompare");
QTest::newRow("locale aware compare") << true;
QTest::newRow("standard compare") << false;
}
The test function then uses this data to determine what to benchmark.
void TestBenchmark::multiple()
{
QFETCH(bool, useLocaleCompare);
QString str1 = QLatin1String("This is a test string");
QString str2 = QLatin1String("This is a test string");
int result;
if(useLocaleCompare)
{
QBENCHMARK {
result = str1.localeAwareCompare(str2);
}
} else {
QBENCHMARK {
result = (str1 == str2);
}
}
Q_UNUSED(result);
}
The "if (useLocaleCompare)" switch is placed outside the QBENCHMARK macro to avoid measuring its overhead. Each benchmark function can have an active QBENCHMARK macro.
Test simulation data
********* Start testing of TestBenchmark *********
Config: Using QtTest library 5.14.0, Qt 5.14.0 (i386-little_endian-ilp32 shared (dynamic) debug build; by MSVC 2017)
PASS : TestBenchmark::initTestCase()
PASS : TestBenchmark::simple()
RESULT : TestBenchmark::simple():
0.0021 msecs per iteration (total: 70, iterations: 32768)
PASS : TestBenchmark::multiple(locale aware compare)
RESULT : TestBenchmark::multiple():"locale aware compare":
0.0014 msecs per iteration (total: 94, iterations: 65536)
PASS : TestBenchmark::multiple(standard compare)
RESULT : TestBenchmark::multiple():"standard compare":
0.00010 msecs per iteration (total: 54, iterations: 524288)
PASS : TestBenchmark::series(locale aware compare--1)
RESULT : TestBenchmark::series():"locale aware compare--1":
0.0011 msecs per iteration (total: 76, iterations: 65536)
PASS : TestBenchmark::series(standard compare--1)
RESULT : TestBenchmark::series():"standard compare--1":
0.000088 msecs per iteration (total: 93, iterations: 1048576)
PASS : TestBenchmark::series(locale aware compare--2001)
RESULT : TestBenchmark::series():"locale aware compare--2001":
0.024 msecs per iteration (total: 51, iterations: 2048)
PASS : TestBenchmark::series(standard compare--2001)
RESULT : TestBenchmark::series():"standard compare--2001":
0.00094 msecs per iteration (total: 62, iterations: 65536)
PASS : TestBenchmark::series(locale aware compare--4001)
RESULT : TestBenchmark::series():"locale aware compare--4001":
0.0493 msecs per iteration (total: 101, iterations: 2048)
PASS : TestBenchmark::series(standard compare--4001)
RESULT : TestBenchmark::series():"standard compare--4001":
0.0018 msecs per iteration (total: 60, iterations: 32768)
PASS : TestBenchmark::series(locale aware compare--6001)
RESULT : TestBenchmark::series():"locale aware compare--6001":
0.073 msecs per iteration (total: 75, iterations: 1024)
PASS : TestBenchmark::series(standard compare--6001)
RESULT : TestBenchmark::series():"standard compare--6001":
0.0027 msecs per iteration (total: 90, iterations: 32768)
PASS : TestBenchmark::series(locale aware compare--8001)
RESULT : TestBenchmark::series():"locale aware compare--8001":
0.096 msecs per iteration (total: 99, iterations: 1024)
PASS : TestBenchmark::series(standard compare--8001)
RESULT : TestBenchmark::series():"standard compare--8001":
0.0035 msecs per iteration (total: 58, iterations: 16384)
PASS : TestBenchmark::cleanupTestCase()
Totals: 15 passed, 0 failed, 0 skipped, 0 blacklisted, 3976ms
********* Finished testing of TestBenchmark *********
External tools
Tools for processing and visualizing test data are part of the qtestlib-tools project. These include tools for comparing performance data obtained from test runs, and utilities for generating web-based performance data graphs.
For more information about these tools and a simple graphical example, please refer to the qtestlib tool declaration.