what's the DBUtils
DBUtils Python package is a set of database connection pool, the high-frequency high concurrent access to the database to provide better performance, can automatically manage the creation and release of the connection object. And to allow non-thread-safe database interface to be thread-safe packaging.
DBUtils provides two external interfaces:
- PersistentDB: provide thread specific database connection, and automatically manage connections.
- PooledDB: providing inter-thread can share database connections and automatically manage the connection.
Experimental results show that PersistentDB speed is the highest , but in some special circumstances, the database connection process may be unusually slow, but at this time PooledDB can provide relatively shorter average connection time management.
In addition, the database also been used in actual driving dependence, such as PersistentDB SQLite database connection pooling can only be used. Download: http://www.webwareforpython.org/downloads/DBUtils/
DBUtils use
Connection pool object initialization only once, generally as a module to ensure the level code. PersistentDB connection example:
import DBUtils.PersistentDB persist=DBUtils.PersistentDB.PersistentDB(dbpai=MySQLdb,maxusage=1000,**kwargs)
Dbpai herein refers to parameters used by the underlying database module, compatible with the DB-API. maxusage, compared with a maximum number of connections to use the official reference examples. ** kwargs the latter was transferred to the actual parameters MySQLdb.
Obtaining a connection: conn = persist.connection () used in actual programming is directly connected conn.Close close () returns the connection to the connection pool.
PooledDB use the same PersistentDB, but the parameters are different.
- dbapi: database interface
- mincached: the number of air connections open at startup
- maxcached: connection pool maximum available number of connections
- maxshared: shared connection pool maximum number of connections
- maxconnections: maximum allowable number of connections
- blocking: When the maximum number is blocked
- maxusage: Maximum single connection multiplexed frequency
- setsession: ready for delivery to the session database, such as [ "set name UTF-8"].
db=pooled.connection() cur=db.cursor() cur.execute(sql) res=cur.fetchone() cur.close() # or del cur db.close() # or del db
python without MySQL connection method connection pool
import MySQLdb conn= MySQLdb.connect(host='localhost',user='root',passwd='pwd',db='myDB',port=3306) #import pymysql #conn = pymysql.connect(host='localhost', port='3306', db='game', user='root', password='123456', charset='utf8') cur=conn.cursor() SQL="select * from table1" r=cur.execute(SQL) r=cur.fetchall() cur.close() conn.close()
The method of connecting the connection pool
Import MySQLdb from DBUtils.PooledDB Import PooledDB the pool = PooledDB (MySQLdb,. 5, Host = ' localhost ' , = User ' the root ' , the passwd = ' pwd ' , DB = ' myDB ' , Port = 3306) # . 5 is connected to the pool the minimum number of connection conn = pool.connection () # after each requires a database connection is to use the connection () function to get connected just fine CUR = conn.cursor () SQL = " the SELECT * from table1 " r = cur.execute ( the SQL) R & lt = cur.fetchall () cur.close () conn.close()
DBUtils Download: https://pypi.python.org/pypi/DBUtils/
import sys import threading import MySQLdb import DBUtils.PooledDB connargs = { "host":"localhost", "user":"user1", "passwd":"123456", "db":"test" } def test(conn): try: cursor = conn.cursor() count = cursor.execute("select * from users") rows = cursor.fetchall() for r in rows: pass finally: conn.close() def testloop(): print ("testloop") for i in range(1000): conn = MySQLdb.connect(**connargs) test(conn) def testpool(): print ("testpool") pooled = DBUtils.PooledDB.PooledDB(MySQLdb, **connargs) for i in range(1000): conn = pooled.connection() test(conn) def main(): t = testloop if len(sys.argv) == 1 else testpool for i in range(10): threading.Thread(target = t).start() if __name__ == "__main__": main()
Take a look at the test results of 10 threads.
$ time ./main.py testloop testloop testloop testloop testloop testloop testloop testloop testloop testloop real 0m4.471s user 0m0.570s sys 0m4.670s $ time ./main.py -l testpool testpool testpool testpool testpool testpool testpool testpool testpool testpool real 0m2.637s user 0m0.320s sys 0m2.750s
Although the test is not very rigorous way, but the results still can feel the performance DBUtils brings. Of course, we can always reuse an off Connection in testloop (), but this is not suitable for the case when the actual development.
DBUtils provides several parameters for us to better align resources.
DBUtils.PooledDB.PooledDB(self, creator, mincached=0, maxcached=0, maxshared=0, maxconnections=0, blocking=False, maxusage=None, setsession=None, failures=None, *args, **kwargs) Docstring: Set up the DB-API 2 connection pool. creator: either an arbitrary function returning new DB-API 2 connection objects or a DB-API 2 compliant database module mincached: initial number of idle connections in the pool (0 means no connections are made at startup) maxcached: maximum number of idle connections in the pool (0 or None means unlimited pool size) maxshared: maximum number of shared connections (0 or None means all connections are dedicated) When this maximum number is reached, connections are shared if they have been requested as shareable. maxconnections: maximum number of connections generally allowed (0 or None means an arbitrary number of connections) blocking: determines behavior when exceeding the maximum (if this is set to true, block and wait until the number of connections decreases, otherwise an error will be reported) maxusage: maximum number of reuses of a single connection (0 or None means unlimited reuse) When this maximum usage number of the connection is reached, the connection is automatically reset (closed and reopened). setsession: optional list of SQL commands that may serve to prepare the session, e.g. ["set datestyle to ...", "set time zone ..."] failures: an optional exception class or a tuple of exception classes for which the connection failover mechanism shall be applied, if the default (OperationalError, InternalError) is not adequate args, kwargs: the parameters that shall be passed to the creator function or the connection constructor of the DB-API 2 module
DBUtils only available to the connection pool management, the actual database operations by the target database module is still in line with standard DB-API 2 is completed.
Using an object-oriented DBUtils chestnuts
# Coding = UTF-. 8 "" " Use DBUtils connection pool database connections, the database operation OperationalError: (2006, 'Away the MySQL Server has Gone') " "" Import JSON Import pymysql Import datetime from DBUtils.PooledDB Import PooledDB Import pymysql class mysqlclient (Object): __pool = None; DEF the __init__ (Self, mincached = 10, maxcached = 20 is, maxshared = 10, MaxConnections = 200 is, blocking = True, maxusage = 100, setsession = None, RESET = True, Host = '127.0.0.1 ' , Port = 3306, DB = ' Test ' , User = ' the root ' , the passwd = ' 123456 ' , charset = ' utf8mb4 ' ): "" " : param mincached: connection pool initial number of idle connections : param maxcached: the maximum number of connections idle connections in the pool : param maxshared: maximum number of shared connections : param maxconnections: to create the maximum number of connection pool : param blocking: the number of connections exceeds the maximum performance time, waiting for the decline in the number of connections True, as direct false error process : param maxusage: the maximum number of repeat single connection using : param setsession: optional list of SQL commands that may serve to prepare the session at The, EG [ "the dateStyle to the SET ...", "... the SET Time Zone"] : param the RESET: How Connections Should BE the RESET the when returned to at The the pool (False or None to ROLLBACK transcations Started with the begin (), True to Always Issue A ROLLBACK for Safety's Sake) : param Host: the database ip address : param port: database port : param db: library name : param user: username : param passwd: password : param charset: the character encoding PooledDB (pymysql,"" " IF not Self. __pool : Self. __class__ . __pool = mincached, maxcached, maxshared, MaxConnections, blocking, maxusage, setsession, reset, host=host, port=port, db=db, user=user, passwd=passwd, charset=charset, cursorclass=pymysql.cursors.DictCursor ) self._conn = None self._cursor = None self.__get_conn() def __get_conn(self): self._conn = self.__pool.connection(); self._cursor = self._conn.cursor(); def close(self): try: self._cursor.close() self._conn.close() except Exception as e: print e def __execute(self, sql, param=()): count = self._cursor.execute(sql, param) print count return count @staticmethod def __dict_datetime_obj_to_str(result_dict): "" " to the dictionary of datatime object into a string, so no conversion error json " "" IF result_dict: result_replace = {K:. V __str__ () for K, V in result_dict.items () IF the isinstance (V, A datetime.datetime)} result_dict.update (result_replace) return result_dict DEF select_one (Self, SQL, param = ()): "" " query individual results " "" COUNT = Self. __execute (SQL, param) result = self._cursor.fetchone () """: the Result of the type: dict "" " result Self =. __Dict_datetime_obj_to_str (the Result) return COUNT, the Result DEF select_many (Self, SQL, param = ()): "" " query multiple results : param sql: qsl statement : param param: sql parameters : return: the number and the results of the query result set "" " COUNT = Self. __execute (SQL, param) the result = self._cursor.fetchall () " "" : the result of the type: List "" " [. Self __dict_datetime_obj_to_str (row_dict) for row_dict in the result] return count, result def execute(self, sql, param=()): count = self.__execute(sql, param) return count def begin(self): """开启事务""" self._conn.autocommit(0) def end(self, option='commit'): """结束事务""" if option == 'commit': self._conn.autocommit() else: self._conn.rollback() if __name__ == "__main__": mc = MysqlClient() sql1 = 'SELECT * FROM shiji WHERE id = 1' result1 = mc.select_one(sql1) print json.dumps(result1[1], ensure_ascii=False) sql2 = 'SELECT * FROM shiji WHERE id IN (%s,%s,%s)' param = (2, 3, 4) print json.dumps(mc.select_many(sql2, param)[1], ensure_ascii=False)