Python operation MySQL and actual combat

Python operation MySQL and actual combat

1. Affairs

Transactions are supported in the innodb engine, but not in myisam.

CREATE TABLE `users` (
  `id` int(11) NOT NULL AUTO_INCREMENT PRIMARY KEY,
  `name` varchar(32) DEFAULT NULL,
  `amount` int(11) DEFAULT NULL
) ENGINE=InnoDB DEFAULT CHARSET=utf8;

For example: Li Jie transfers 100 to Wu Peiqi, which will involve 2 steps.

• Li Jie account minus 100
• Add 100 to Wu Peiqi's account

These two steps must be completed at the same time to be considered complete, and if the first step is completed and the second step fails, it is still rolled back to the initial state.

Transactions are here to solve this situation. Vernacular: If you want to succeed, you will succeed; if you want to fail, you will fail.

Transactions have four characteristics (ACID):

• Atomicity

  原子性是指事务包含的所有操作不可分割，要么全部成功，要么全部失败回滚。
  

• Consistency

  执行的前后数据的完整性保持一致。
  

• Isolation

  一个事务执行的过程中,不应该受到其他事务的干扰。
  

• Durability

  事务一旦结束,数据就持久到数据库
  

1.1 MySQL client

mysql> select * from users;
+----+---------+---------+
| id | name    | amount  |
+----+---------+---------+
|  1 | wupeiqi |    5    |
|  2 |  alex   |    6    |
+----+---------+---------+
3 rows in set (0.00 sec)

mysql> begin;  -- 开启事务 start transaction;
Query OK, 0 rows affected (0.00 sec)

mysql> update users set amount=amount-2 where id=1;   -- 执行操作
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

mysql> update users set amount=amount+2 where id=2;   -- 执行操作
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

mysql> commit;  -- 提交事务  rollback;
Query OK, 0 rows affected (0.00 sec)

mysql> select * from users;
+----+---------+---------+
| id | name    | amount  |
+----+---------+---------+
|  1 | wupeiqi |    3    |
|  2 |  ale x  |    8    |
+----+---------+---------+
3 rows in set (0.00 sec)

mysql> select * from users;
+----+---------+---------+
| id | name    | amount  |
+----+---------+---------+
|  1 | wupeiqi |    3    |
|  2 |  ale x  |    8    |
+----+---------+---------+
3 rows in set (0.00 sec)

mysql> begin; -- 开启事务
Query OK, 0 rows affected (0.00 sec)

mysql> update users set amount=amount-2 where id=1; -- 执行操作（此时数据库中的值已修改）
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

mysql> rollback; -- 事务回滚（回到原来的状态）
Query OK, 0 rows affected (0.00 sec)

mysql> select * from users;
+----+---------+---------+
| id | name    | amount  |
+----+---------+---------+
|  1 | wupeiqi |    3    |
|  2 |  ale x  |    8    |
+----+---------+---------+
3 rows in set (0.00 sec)

1.2 Python code

import pymysql

conn = pymysql.connect(host='127.0.0.1', port=3306, user='root', passwd='root123', charset="utf8", db='userdb')
cursor = conn.cursor()

# 开启事务
conn.begin()

try:
    cursor.execute("update users set amount=1 where id=1")
    int('asdf')
    cursor.execute("update tran set amount=2 where id=2")
except Exception as e:
    # 回滚
    print("回滚")
    conn.rollback()
else:
    # 提交
    print("提交")
    conn.commit()

cursor.close()
conn.close()

2. lock

When using MySQL, I don’t know if you have any doubts: There are many updates, inserts, and deletions at the same time. How does MySQL ensure that the data is not wrong?

MySQL has its own lock function, which can help us realize the situation of processing data at the same time encountered in the development process. For the locks in the database, in terms of the scope of the locks:

• Table-level lock, that is, when A operates the table, other people cannot operate on the entire table, and can only continue after A completes the operation.
• Row-level lock, that is, when A operates the table, other people cannot operate the specified row data, but other rows can operate, and wait for A to complete the operation before proceeding.

MYISAM支持表锁，不支持行锁；InnoDB引擎支持行锁和表锁。即：在MYISAM下如果要加锁，无论怎么加都会是表锁。    在InnoDB引擎支持下如果是基于索引查询的数据则是行级锁，否则就是表锁。

Therefore, in general, we will choose to use the innodb engine, and we will also use the index (hit index) when searching.

The next operation is based on the innodb engine:

CREATE TABLE `L1` (  `id` int(11) NOT NULL AUTO_INCREMENT,  `name` varchar(255) DEFAULT NULL,  `count` int(11) DEFAULT NULL,  PRIMARY KEY (`id`)) ENGINE=InnoDB  DEFAULT CHARSET=utf8;

In the InnoDB engine, the actions of update, insert, and delete will first apply for a lock (exclusive lock), and then perform related operations after the application is obtained, and finally release the lock.

所以，当多个人同时像数据库执行：insert、update、delete等操作时，内部加锁后会排队逐一执行。

Select does not apply for locks by default.

select * from xxx;

If you want select to apply for locks, you need to cooperate with transactions + special syntax to achieve.

• for update, exclusive lock, after locking, others cannot read and write.

  begin; 	select * from L1 where name="武沛齐" for update;    -- name列不是索引（表锁）commit;
  

  begin; -- 或者 start transaction;	select * from L1 where id=1 for update;			  -- id列是索引（行锁）commit;
  

• lock in share mode, Shared lock, after locking, others can read but not write.

  begin; 	select * from L1 where name="武沛齐" lock in share mode;    -- 假设name列不是索引（表锁）commit;
  

  begin; -- 或者 start transaction;	select * from L1 where id=1 lock in share mode;           -- id列是索引（行锁）commit;
  

2.1 Exclusive lock

Exclusive lock ( for update), after locking, other transactions cannot read and write.

Application scenario: There are 100 items in total, and each time you buy one item, you need to decrease the number of items by 1.

A: 访问页面查看商品剩余 100B: 访问页面查看商品剩余 100此时 A、B 同时下单，那么他们同时执行SQL：	update goods set count=count-1 where id=3由于Innodb引擎内部会加锁，所以他们两个即使同一时刻执行，内部也会排序逐步执行。但是，当商品剩余 1个时，就需要注意了。A: 访问页面查看商品剩余 1B: 访问页面查看商品剩余 1此时 A、B 同时下单，那么他们同时执行SQL：	update goods set count=count-1 where id=3这样剩余数量就会出现 -1，很显然这是不正确的，所以应该怎么办呢？这种情况下，可以利用 排它锁，在更新之前先查询剩余数量，只有数量 >0 才可以购买，所以，下单时应该执行：	begin; -- start transaction;	select count from goods where id=3 for update;	-- 获取个数进行判断	if 个数>0:		update goods set count=count-1 where id=3;	else:		-- 已售罄	commit;

Based on Python code example:

import pymysqlimport threadingdef task():    conn = pymysql.connect(host='127.0.0.1', port=3306, user='root', passwd='root123', charset="utf8", db='userdb')    cursor = conn.cursor(pymysql.cursors.DictCursor)    # cursor = conn.cursor()	    # 开启事务    conn.begin()    cursor.execute("select id,age from tran where id=2 for update")    # fetchall      ( {"id":1,"age":10},{"id":2,"age":10}, )   ((1,10),(2,10))    # {"id":1,"age":10}   (1,10)    result = cursor.fetchone()    current_age = result['age']        if current_age > 0:        cursor.execute("update tran set age=age-1 where id=2")    else:        print("已售罄")    conn.commit()    cursor.close()    conn.close()def run():    for i in range(5):        t = threading.Thread(target=task)        t.start()if __name__ == '__main__':    run()

2.2 Shared lock

Shared lock ( lock in share mode), can read, but not write.

After locking, other subsequent things can be read, but writing (update, delete, insert) is not allowed, because writing will also be locked by default.

Locking Read Examples

Suppose that you want to insert a new row into a table child, and make sure that the child row has a parent row in table parent. Your application code can ensure referential integrity throughout this sequence of operations.

First, use a consistent read to query the table PARENT and verify that the parent row exists. Can you safely insert the child row to table CHILD? No, because some other session could delete the parent row in the moment between your SELECT and your INSERT, without you being aware of it.

To avoid this potential issue, perform the SELECT using LOCK IN SHARE MODE:

SELECT * FROM parent WHERE NAME = 'Jones' LOCK IN SHARE MODE;

After the LOCK IN SHARE MODE query returns the parent 'Jones', you can safely add the child record to the CHILD table and commit the transaction. Any transaction that tries to acquire an exclusive lock in the applicable row in the PARENT table waits until you are finished, that is, until the data in all tables is in a consistent state.

3. Database connection pool

When operating a database, a database connection pool needs to be used.

pip3.9 install pymysqlpip3.9 install dbutils

import threadingimport pymysqlfrom dbutils.pooled_db import PooledDBMYSQL_DB_POOL = PooledDB(    creator=pymysql,  # 使用链接数据库的模块    maxconnections=5,  # 连接池允许的最大连接数，0和None表示不限制连接数    mincached=2,  # 初始化时，链接池中至少创建的空闲的链接，0表示不创建    maxcached=3,  # 链接池中最多闲置的链接，0和None不限制    blocking=True,  # 连接池中如果没有可用连接后，是否阻塞等待。True，等待；False，不等待然后报错    setsession=[],  # 开始会话前执行的命令列表。如：["set datestyle to ...", "set time zone ..."]    ping=0,    # ping MySQL服务端，检查是否服务可用。    # 如：0 = None = never, 1 = default = whenever it is requested,     # 2 = when a cursor is created, 4 = when a query is executed, 7 = always    host='127.0.0.1',    port=3306,    user='root',    password='root123',    database='userdb',    charset='utf8')def task():    # 去连接池获取一个连接    conn = MYSQL_DB_POOL.connection()    cursor = conn.cursor(pymysql.cursors.DictCursor)        cursor.execute('select sleep(2)')    result = cursor.fetchall()    print(result)    cursor.close()    # 将连接交换给连接池    conn.close()def run():    for i in range(10):        t = threading.Thread(target=task)        t.start()if __name__ == '__main__':    run()

4. SQL tools

Develop a common SQL operation class based on the database connection pool to facilitate future operation of the database.

4.1 Singletons and methods

# db.pyimport pymysqlfrom dbutils.pooled_db import PooledDBclass DBHelper(object):    def __init__(self):        # TODO 此处配置，可以去配置文件中读取。        self.pool = PooledDB(            creator=pymysql,  # 使用链接数据库的模块            maxconnections=5,  # 连接池允许的最大连接数，0和None表示不限制连接数            mincached=2,  # 初始化时，链接池中至少创建的空闲的链接，0表示不创建            maxcached=3,  # 链接池中最多闲置的链接，0和None不限制            blocking=True,  # 连接池中如果没有可用连接后，是否阻塞等待。True，等待；False，不等待然后报错            setsession=[],  # 开始会话前执行的命令列表。如：["set datestyle to ...", "set time zone ..."]            ping=0,            # ping MySQL服务端，检查是否服务可用。# 如：0 = None = never, 1 = default = whenever it is requested, 2 = when a cursor is created, 4 = when a query is executed, 7 = always            host='127.0.0.1',            port=3306,            user='root',            password='root123',            database='userdb',            charset='utf8'        )    def get_conn_cursor(self):        conn = self.pool.connection()        cursor = conn.cursor(pymysql.cursors.DictCursor)        return conn, cursor    def close_conn_cursor(self, *args):        for item in args:            item.close()    def exec(self, sql, **kwargs):        conn, cursor = self.get_conn_cursor()        cursor.execute(sql, kwargs)        conn.commit()        self.close_conn_cursor(conn, cursor)    def fetch_one(self, sql, **kwargs):        conn, cursor = self.get_conn_cursor()        cursor.execute(sql, kwargs)        result = cursor.fetchone()        self.close_conn_cursor(conn, cursor)        return result    def fetch_all(self, sql, **kwargs):        conn, cursor = self.get_conn_cursor()        cursor.execute(sql, kwargs)        result = cursor.fetchall()        self.close_conn_cursor(conn, cursor)        return resultdb = DBHelper()

from db import dbdb.exec("insert into d1(name) values(%(name)s)", name="武沛齐666")ret = db.fetch_one("select * from d1")print(ret)ret = db.fetch_one("select * from d1 where id=%(nid)s", nid=3)print(ret)ret = db.fetch_all("select * from d1")print(ret)ret = db.fetch_all("select * from d1 where id>%(nid)s", nid=2)print(ret)

4.2 Context Management

If you want him to also support with context management.

with 获取连接：	执行SQL（执行完毕后，自动将连接交还给连接池）

# db_context.pyimport threadingimport pymysqlfrom dbutils.pooled_db import PooledDBPOOL = PooledDB(    creator=pymysql,  # 使用链接数据库的模块    maxconnections=5,  # 连接池允许的最大连接数，0和None表示不限制连接数    mincached=2,  # 初始化时，链接池中至少创建的空闲的链接，0表示不创建    maxcached=3,  # 链接池中最多闲置的链接，0和None不限制    blocking=True,  # 连接池中如果没有可用连接后，是否阻塞等待。True，等待；False，不等待然后报错    setsession=[],  # 开始会话前执行的命令列表。如：["set datestyle to ...", "set time zone ..."]    ping=0,    host='127.0.0.1',    port=3306,    user='root',    password='root123',    database='userdb',    charset='utf8')class Connect(object):    def __init__(self):        self.conn = conn = POOL.connection()        self.cursor = conn.cursor(pymysql.cursors.DictCursor)    def __enter__(self):        return self    def __exit__(self, exc_type, exc_val, exc_tb):        self.cursor.close()        self.conn.close()    def exec(self, sql, **kwargs):        self.cursor.execute(sql, kwargs)        self.conn.commit()    def fetch_one(self, sql, **kwargs):        self.cursor.execute(sql, kwargs)        result = self.cursor.fetchone()        return result    def fetch_all(self, sql, **kwargs):        self.cursor.execute(sql, kwargs)        result = self.cursor.fetchall()        return result

from db_context import Connectwith Connect() as obj:    # print(obj.conn)    # print(obj.cursor)    ret = obj.fetch_one("select * from d1")    print(ret)    ret = obj.fetch_one("select * from d1 where id=%(id)s", id=3)    print(ret)

5. Other

navicat is a desktop application that allows us to manage MySQL databases more conveniently.

• mac system: https://www.macdo.cn/17030.html
• Win system:
  • Link: https://pan.baidu.com/s/13cjbrBquz9vjVqKgWoCQ1w Password: qstp
  • Link: https://pan.baidu.com/s/1JULIIwQA5s0qN98KP8UXHA Password: p18f