Sentiment]
After writing this log, the program has been argued for some time, there has been little experience under Share:
A) climb a stable number (dong) According to (xi) the best storage down, especially in a foreign country the kind of database, the download time is too costly, and save up reprocessing, will save a lot of time;
B) For python using multiple threads to write data to the database sqlite practice is absolutely a waste of emotion, behavior is nonsense, a word called "overkill", a single thread is enough.
C) the number of write data to the database commit the best reduction, after 10 000 can insert commit, of course, depending on the transaction itself and other factors, I'm here to double the speed.
D) data on the climb back, stored as text for retail use is optimal, fast processing speed, well-defined format, the regular functions victory sqlite.
[] Problem situation
Recently busy hand, I wanted to write a simple batch download page of python tools. Of course reptile lightweight framework scrapy previously used, not bad. Because of this demand is very simple, direct human flesh did. Results of various problems arise.
Where the idea is that multi-threaded database to store these pages, but also facilitate data after processing. However, when running multi-threaded, the script often die. Opened 50 threads, fundamental errors occur prompt is "Recursive use of cursors not allowed".
【problem analysis】
Database running multi-threaded story, usually did not submit the transaction can not guarantee atomicity problems. Problem Solution can see directly below the [problem-solving] section.
Check out the first pass:
Question 1: find a select query did not submit, commit a bit.
Analysis 1: here feel there is no problem, because it does not change the database query did not submit the same.
Run again:
Error seems less certain. At this time, sleep time I insert data is 1sec. Tried several times, collapse or halfway out.
Check the second time:
Question 2: insert shortened sleep time, the problem is more serious. Sister soon collapse.
Analysis: sql statement should increase the probability of conflict insert frequently caused. Always felt there is a problem here, and he sent the following information, it really is multi-threaded sqlite3 when transmission of the disease. Thus, the position of each submission have increased thread-locking, enhanced synchronization effect.
Run again:
Error "Recursive use of cursors not allowed" does not occur again.
【solution】
Lock submission.
lock = threading.Lock()
def Func(host,cursor,db):
try:
lock.acquire(True)
res = cursor.execute('''...''',(host,))
# do something
finally:
lock.release()
【other】
sqlite3 using python's argument check_same_thread multithreading is needed:
self.conn = sqlite3.connect(dbname, check_same_thread=False)
Disclaimer: This article is CSDN blogger original article "counsellor", and follow CC 4.0 BY-SA copyright agreement, reproduced, please attach the original source link and this statement.
Original link: https: //blog.csdn.net/counsellor/article/details/43715007