When using multiple processes and coroutines to crawl data concurrently, it was found that the CPU of the mongodb server soared to 100%, which seriously affected the speed of concurrent crawling. So I checked and said it was adding an index. The code for operating mongodb in the program is as follows:
# 存在则更新，不存在则插入
await self._async_collection.update_one({'url_hash':self.url_hash},{"$set":data},True)So create an index:
db.novels.createIndex({"url_hash":1})The CPU immediately dropped. At the same time, the crawl speed has obviously increased, as shown in the figure below.
It seems that the power of mongodb index is quite big.