Sqlite3 write performance optimization - millions of writes per second

Recently, Sqlite3 was used in the project to save the result data, about 1 million pieces of data, it took 5 minutes to insert into the sqlite database, which took up too much time in the actual data processing process, which is unacceptable, so how? How to optimize the performance of sqlite's writing data?

Optimization method

By checking the information and other bloggers' blogs, I made sure that there are a few points to try:

Turn off write synchronization, PRAGMA synchronous = OFF, there are three synchronous modes in sqlite3, namely FULL, NORMAL and OFF. When the system terminates unexpectedly, the security is gradually weakened. In FULL mode, it is guaranteed that the data will not be damaged and the security Highest and slowest write speed. OFF mode will be more than 50 times faster than FULL mode.
Using transactions, if there is a lot of data that needs to be inserted into the database, insert them one by one, resulting in frequent commits and disk IO. Using the transaction mechanism, data can be inserted in batches, which can greatly improve the writing speed. The actual test situation is that after the transaction is opened, the writing speed can also be increased by nearly 50 times.
Execution preparation is equivalent to compiling SQL statements in advance, eliminating the need for syntax checking and other operations every time SQL statements are executed, which can greatly optimize the execution efficiency of SQL statements. The principle is a bit like LuaJit transforms Lua into static machine code , to increase the running speed. In the actual measurement situation, the use of execution preparation can improve the write speed by 40 times.
Memory mode, sqlite3 supports memory mode, create the database directly into memory, open the address and pass in ":memory:". Compared with normal mode, memory mode can save IO time. The acceleration idea of using memory mode is: First create the database into memory, and after the data is written completely, call the "VACUUM INTO 'out.db3';" statement to write it to the disk. When execution preparation is enabled, this method will be slightly faster. a little bit.

Efficiency comparison

Using the method mentioned above, the test speed comparison is as follows:

Optimization	no optimization	turn off write synchronization	open transaction	Prepare for execution	memory mode
Inserts per second	13	1321 Articles	50,000	2.13 million	2.15 million

test code

Error checking macro definition:

#define CHECKZERO(a) if((a)!=0) throw("error.");

no optimization

sqlite3* db = nullptr;
CHECKZERO(sqlite3_open(path, &db));
CHECKZERO(sqlite3_exec(db, "CREATE TABLE Test(ID INTEGER,var0 INTEGER,var1 REAL,var2 TEXT);", 0, 0, 0));
const int maxcount = 100;
for (int i = 0; i < maxcount; i++) {
    CHECKZERO(sqlite3_exec(db, "INSERT INTO Test (ID,var0,var1,var2) VALUES (0,1,2.0,\\"hello sqlite3.\\");", 0, 0, 0));
}
CHECKZERO(sqlite3_close(db));

turn off write synchronization

sqlite3* db = nullptr;
CHECKZERO(sqlite3_open(path, &db));
CHECKZERO(sqlite3_exec(db, "PRAGMA synchronous = OFF", 0, 0, 0));
CHECKZERO(sqlite3_exec(db, "CREATE TABLE Test(ID INTEGER,var0 INTEGER,var1 REAL,var2 TEXT);", 0, 0, 0));
const int maxcount = 10000;
for (int i = 0; i < maxcount; i++) {
    CHECKZERO(sqlite3_exec(db, "INSERT INTO Test (ID,var0,var1,var2) VALUES (0,1,2.0,\\"hello sqlite3.\\");", 0, 0, 0));
}
CHECKZERO(sqlite3_close(db));

open transaction

sqlite3* db = nullptr;
CHECKZERO(sqlite3_open(path, &db));
CHECKZERO(sqlite3_exec(db, "PRAGMA synchronous = OFF", 0, 0, 0));
CHECKZERO(sqlite3_exec(db, "CREATE TABLE Test(ID INTEGER,var0 INTEGER,var1 REAL,var2 TEXT);", 0, 0, 0));
CHECKZERO(sqlite3_exec(db, "BEGIN", 0, 0, 0));
const int maxcount = 1000000;
for (int i = 0; i < maxcount; i++) {
    CHECKZERO(sqlite3_exec(db, "INSERT INTO Test (ID,var0,var1,var2) VALUES (0,1,2.0,\\"hello sqlite3.\\");", 0, 0, 0));
    if (i % 10000 == 9999) {
        CHECKZERO(sqlite3_exec(db, "COMMIT", 0, 0, 0));
        CHECKZERO(sqlite3_exec(db, "BEGIN", 0, 0, 0));
    }
}
CHECKZERO(sqlite3_exec(db, "COMMIT", 0, 0, 0));
CHECKZERO(sqlite3_close(db));

Prepare for execution

sqlite3* db = nullptr;
CHECKZERO(sqlite3_open(path, &db));
CHECKZERO(sqlite3_exec(db, "PRAGMA synchronous = OFF", 0, 0, 0));
CHECKZERO(sqlite3_exec(db, "CREATE TABLE Test(ID INTEGER,var0 INTEGER,var1 REAL,var2 TEXT);", 0, 0, 0));
// 执行准备
sqlite3_stmt *pPrepare = nullptr;
auto sql = "INSERT INTO Test (ID,var0,var1,var2) VALUES (?,?,?,?);";
CHECKZERO(sqlite3_prepare_v2(db, sql, strlen(sql), &pPrepare, 0));
CHECKZERO(sqlite3_exec(db, "BEGIN", 0, 0, 0));
const int maxcount = 10000000;
for (int i = 0; i < maxcount; i++) {
    CHECKZERO(sqlite3_reset(pPrepare));
    CHECKZERO(sqlite3_bind_int(pPrepare, 1, 0));
    CHECKZERO(sqlite3_bind_int(pPrepare, 2, 1));
    CHECKZERO(sqlite3_bind_double(pPrepare, 3, 2.0));
    const char* str = "hello sqlite3.";
    CHECKZERO(sqlite3_bind_text(pPrepare, 4, str, strlen(str), 0));
    int err = sqlite3_step(pPrepare);
    assert(SQLITE_DONE == err);
    if (i % 10000 == 9999) {
        CHECKZERO(sqlite3_exec(db, "COMMIT", 0, 0, 0));
        CHECKZERO(sqlite3_exec(db, "BEGIN", 0, 0, 0));
    }
}
CHECKZERO(sqlite3_exec(db, "COMMIT", 0, 0, 0));
CHECKZERO(sqlite3_finalize(pPrepare)); // 释放
CHECKZERO(sqlite3_close(db));

memory mode

sqlite3* db = nullptr;
CHECKZERO(sqlite3_open(":memory:", &db));
CHECKZERO(sqlite3_exec(db, "PRAGMA synchronous = OFF", 0, 0, 0));
CHECKZERO(sqlite3_exec(db, "CREATE TABLE Test(ID INTEGER,var0 INTEGER,var1 REAL,var2 TEXT);", 0, 0, 0));
// 执行准备
sqlite3_stmt *pPrepare = nullptr;
auto sql = "INSERT INTO Test (ID,var0,var1,var2) VALUES (?,?,?,?);";
CHECKZERO(sqlite3_prepare_v2(db, sql, strlen(sql), &pPrepare, 0));
CHECKZERO(sqlite3_exec(db, "BEGIN", 0, 0, 0));
const int maxcount = 10000000;
for (int i = 0; i < maxcount; i++) {
    CHECKZERO(sqlite3_reset(pPrepare));
    CHECKZERO(sqlite3_bind_int(pPrepare, 1, 0));
    CHECKZERO(sqlite3_bind_int(pPrepare, 2, 1));
    CHECKZERO(sqlite3_bind_double(pPrepare, 3, 2.0));
    const char* str = "hello sqlite3.";
    CHECKZERO(sqlite3_bind_text(pPrepare, 4, str, strlen(str), 0));
    int err = sqlite3_step(pPrepare);
    assert(SQLITE_DONE == err);
    if (i % 10000 == 9999) {
        CHECKZERO(sqlite3_exec(db, "COMMIT", 0, 0, 0));
        CHECKZERO(sqlite3_exec(db, "BEGIN", 0, 0, 0));
    }
}
CHECKZERO(sqlite3_exec(db, "COMMIT", 0, 0, 0));
CHECKZERO(sqlite3_finalize(pPrepare)); // 释放
// 导出
CHECKZERO(sqlite3_exec(db, "VACUUM INTO 'out.db3';", 0, 0, 0));
CHECKZERO(sqlite3_close(db));

Summarize

As such a powerful and lightweight database engine, sqlite3 will not be very slow to insert. If you find efficiency problems during your use, you must have not found a suitable usage. In the final test results, the writing speed of sqlite3 A staggering 2 million per second.

The complete test engineering code can be downloaded here: sqlite3 performance optimization source code data insertion open transaction execution preparation performance improvement million data writes per second - other document resources - CSDN library