pymongo.errors.CursorNotFound: Cursor not found
Background story : Data acquisition start database of all db [ 'test'] find ( ), then the results for loop, but when do_something function takes too long, long time operation on the cursor, the cursor initiator mongodb. server timeout.
Analysis of reasons: you're in a db.collection.find (), which returns not all of the data, but in fact is a "cursor". Its default behavior is: the first query to the database 101 documents, or 1 MB of documents, depending on which condition is satisfied first; after each exhaustion after the cursor in the document, inquiry 4 MB of documentation. Further, the default behavior find () is to return a no operation time-out after 10 minutes of the cursor. If I did a batch of documents processed within ten minutes, again after treatment is over, then remove with a cursor id a batch to the server, this time the cursor id of course already expired, which would explain why I get the cursor id invalid mistake.
Idea Summary: default mongo server maintenance windows connect ten minutes; the default single server to retrieve data from the data is greater than 101 or less than 1M to 16M, so by default, if not processed data within 10 minutes, then toss the abnormality.
solution:
1. Modify the number of pieces of the acquired data amount per batch, i.e., batch size:
collection.find(condition).batch_size(10)
Batch number of minutes required to estimate the amount of data that can be processed
2. Set no_cursor_timeout = True, never timeout, the cursor will not take the initiative to close the connection, you need to manually shut down
cursor=db.images.find({}{'_id':0},no_cursor_timeout=True)
I propose: the feeling of Option II is not very appropriate, assuming that the process of execution do_something, the error logically directly out, then the cursor is still open at this time of state?