SQLite3 in Python uses executemany to insert data in batches

Thousands of words are not as deep as a demo. As far as I know, both MySQL and sqlite in Python can use executemany to insert large amounts of data in batches, and the efficiency is basically an order of magnitude improvement over ordinary insertion.

Benefits of using executemany

efficient

I have tested it myself. For the same more than 10,000 pieces of data, it takes 54.5 seconds for normal insertion and 0.22 seconds for executemany. I won’t go into more details about efficiency.

However, SQL is slightly different. Does SQLite use it as a placeholder instead of %s, %d and the like! Examples of the correct method are as follows:

sql = 'insert into filelist (pkgKey, dirname, filenames, filetypes) values (?, ?, ?, ?);'

import sqlite3

class DbOperate(object):
    def __new__(cls, *args, **kwargs):
        if not hasattr(cls, "_instance"):
            cls._instance = super(DbOperate, cls).__new__(cls)
        return cls._instance

    def __init__(self, db_name):
        self.db_name = db_name
        self.connect = sqlite3.connect(self.db_name)
        self.cursor = self.connect.cursor()

    def __enter__(self):
        return self

    def __exit__(self, exc_type, exc_val, exc_tb):
        self.connect.close()

    def execute_sql(self, sql):
        try:
            self.cursor.execute(sql)
            self.connect.commit()
        except Exception as e:
            self.connect.rollback()

    def executemany_sql(self, sql, data_list):
        # example:
        # sql = 'insert into filelist (pkgKey, dirname, filenames, filetypes) values (?, ?, ?, ?);'
        # data_list = [(1, '/etc/sysconfig', 'openshift_option', 'f'), (1, '/usr/share/doc', 'adb-utils-1.6', 'd')]
        try:
            self.cursor.executemany(sql, data_list)
            self.connect.commit()
        except Exception as e:
            self.connect.rollback()
            raise Exception("executemany failed")

sqlite_path = "***.sqlite"
with DbOperate(sqlite_path) as db:
    t1 = time.clock()
    sql = 'insert into filelist (pkgKey, dirname, filenames, filetypes) values (?, ?, ?, ?);'
    data_list = [(1, '/etc/sysconfig', 'openshift_option', 'f'), (1, '/usr/share/doc', 'adb-utils-1.6', 'd')]
    db.executemany_sql(sql, data_list)
    t2 = time.clock()
    print('insert data into filelist cost %s seconds' % (t2 - t1))
    print('success insert data into filelist with %s' % sqlite_path)

Not susceptible to special characters

I did it like the following when inserting a single one before:

with DbOperate(sqlite_path) as db:
    t1 = time.clock()
    for item in packages:
        insert_data = f'insert into packages values {item};'
        insert_data = insert_data.replace('None', 'null')
        print(insert_data)
        db.execute_sql(insert_data)
    t2 = time.clock()

As a result, because there are always many special characters in some fields, such as single quotes, escape characters, etc., directly splicing SQL statements will cause the SQL statement to not meet the SQLite grammar due to various reasons, so that the data will always be reported when inserting. Wrong, it made me very annoyed. There are no such errors when using executemany. I am actually a little confused. Perhaps it is because executemany passes in the parameters correspondingly:

Traceback (most recent call last):
  File "C:/Users/lukaiyi/insight-tool/test.py", line 144, in <module>
    insert_data_sqlite(sqlite_list, filelist, packages)
  File "C:/Users/lukaiyi/insight-tool/test.py", line 42, in insert_data_sqlite
    db.execute_sql(insert_data)
  File "C:\Users\lukaiyi\insight-tool\src\tool\utils.py", line 92, in execute_sql
    self.cursor.execute(sql)
sqlite3.OperationalError: near "s": syntax error