Multi-threaded run the project. There are N number of worker threads get jobs from the DB, and the result is written back to the DB.
When the project is running for some time, it found that the database connection is depleted, but fortunately good memory, and then has been to raise the last number of connections on more than 8,000. When depleted the number of connections, postgresql like this error occurs:
FATAL: Remaining Connection slots are Reserved for non-superuser Connections Replication
is probably the knowledge so two points:
1. If a Web project, when the end of the request, Django I will go to close out the connection. Yes, there is no connection pool.
2. Because we are a non-Web project, so the end of the request event does not exist, so they did not close the connection. But Actually, this should not cause problems, since not been closed by chanting, but do not know where the problem is, there will be leakage connection, the connection data will always grow.
The final solution is to find time to take the initiative to close the database connection, specific to our project, that is, each worker thread after the completion of a task, put it relevant connection turned off, because we are using ThreadPoolExecutor, so Django is easy to do this a little.
Code is as follows:
from django.db Import Connections
DEF on_done (Future):
connections.close_all ()
DEF main ():
with the ThreadPoolExecutor () AS Executor:
the while True:
Future executor.submit = (do, get_a_job ())
future.add_done_callback(on_done)
More technical information may concern: gzitcast