"Note" MySQL combat 45 say - practice articles (b)

Sql Bad Case

Function operating condition field
- Function of the index field to do the operation, could undermine the ordering of the index value, the optimizer decided to give away tree search
- Chestnut: month () function, where id + 1 = 10000, etc.
Implicit type conversion
- In MySQL, strings and numbers compare, then the string is converted into a digital
- Chestnuts: select "10"> 9 (1 return on behalf of a digital comparator to do
Implicit character encoding conversion
- utf8mb4 is a superset of utf8
- chestnut
  - select * from trade_detail where tradeid=$L2.tradeid.value; （原SQL
  - select * from trade_detail where CONVERT(traideid USING utf8mb4)=$L2.tradeid.value;
    - The CONVERT () function: the input character string converted into the character set utf8mb4
    - Cause connection process requires the index field is driven operating table plus function (results in table being driven to do the whole table scan
- Breaking the Road
  - Universal Character Set (if on a large amount of data and do not allow DDL operations temporarily
  - 修改SQL：…. where d.tradeid=CONVERT(l.tradeid USING utf8) and l.id=2;

Slow Situation

Queries do not return for a long time
- MDL lock, etc.
  - Now there is a request on a thread is t table or hold MDL write lock, the select statement block
- Seized plus table lock thread id
And other FLUSH
- Waiting for table flush schematic state
  - MySQL table to do inside to flush usage
```
 # 只关闭表 t
 flush tables t with read lock;
 # 关闭 MySQL 里所有打开的表
 flush tables with read lock
```
- Row lock, etc.
  - 加锁读方式：select * from t where id=1 lock in share mode（for update
  - View lock wait information: select * from t sys.innodb_lock_waits where locked_table =xxx
Query slow
- Number of scanning lines and more
  - 栗子：select * from t where c=50000 limit 1;
    - There is no index on the field c, this statement can only go id primary key order scanning, and therefore need to scan 50,000 lines
    - The amount of data execution time increases linearly up
- Read consistency
  - chestnut
    - select * from t where id = 1; Number (scan line 1, while 800 ms CEO
    - select * from t where id = 1 lock in share mode; Number (scan line 1, performing 0.2 ms long
  - data state id = 1
    - session B finished updating million times, generated 1 million to roll back log (undo log)
    - After reading the consistency you need to perform in order from the beginning 1000001 undo log, execute a million times, it will return the results

Magic Reading

Special Note
- Magic Reading (common query is a snapshot reading, do not see anything else inserted data to appear in the "current reading" under
  - The current reading of the rule is to be able to read the latest value of all the records that have been submitted
- Phantom read only specifically refers to "newly inserted line" (dialectical perspective on
The lock is designed to ensure data consistency
- Not only an internal data state database consistency at the moment, further comprising data logging and logical consistency in
To solve the problem of phantom read, InnoDB had to introduce a new lock, the lock is a gap (Gap Lock)
- The gap is the gap between the two values locks, lock
- Gap and row locks together said next-key lock, each next-key lock is the front opening and closing zone
Clearance and lock the introduction of next-key lock has brought some "troubled"
- Lock deadlock caused by the gap (clearance lock lock compatible with the gap, the gap with the lock insert intent lock conflicts
- Introducing clearance lock may result in the same statement to lock a larger scale, this is actually affects the degree of concurrency

Lock rules

I summarize the locking rules which, contains two "principles", two "optimization" and a "bug"
- Principle 1: The basic unit of locks is the next-key lock. I hope you remember, next-key lock is front opening and closing zone
- Principle 2: Find Objects process will have access to the lock.
- Optimization 1: equivalence query on the index, the only index to lock when, next-key lock degenerate into row lock
- Optimization 2: equivalence query on the index, and the last traversing the right conditions is not satisfied equivalent value, next-key lock latch degenerate gap
- A bug: the scope of the inquiry until the first unique index value does not satisfy the conditions of access to
Lock on the coverage index
- 栗子：select id from t where c = 5 lock in share mode;
- Line lock in share mode only lock cover index, for update will point to the primary key index to meet the conditions of the lock with a line
Primary key index range lock
- This time range lock session A is the primary key index, the line lock id = 10 and next-key lock (10,15]
- Find id = 10 rows is judged as equivalent to the query, and the right to scan the time id = 15, using a range queries Analyzing
The only index range lock bug
- InnoDB will scan forward to the first condition is not satisfied acts only, that is, id = 20
  - Since this is a scan range, and therefore the index (15, 20 on the id]
limit lock statement
- 栗子：delete from t where c = 10 / delete from t where c = 10 limit 2
  - The former lock range: (5,15) Rear lock range: (5,10)
- When you delete data as much as possible plus limit, not only can control the number of deleted data, allowing safer operation, can also reduce the scope locked
When the next-key lock locking, to be divided into two gaps and row locks performed
Reading submit a lower isolation level optimization: the process of execution of sentence plus a row lock, in a statement after the execution is complete, we will build on the row lock "line does not satisfy the condition of" direct release, you do not need to wait until the transaction commits

show time

Background: MySQL pressure peak of the business production environment is too large, can not respond to normal, short-term needs, temporarily lifting some performance
- This is the reason why this chapter is called "show time" of ( IT's your show time
- In these processing means, the abrupt manner both connected and disconnected, there are several methods to bypass pits statement by overwriting
- Both temporary high-risk program, there are proactive, relatively safe plan
- Disconnect the connection exception is often the case, your code must be properly connected and retry mechanism of weight
Short connections storm
- If you are using a short connection, during peak periods, when the situation might appear sudden increase in the number of connections
  - When the machine load is relatively high, the processing time becomes long existing request, each time the connection is also maintained for longer
- MySQL connection establishment, the cost is very high
  - In addition to the normal network connection three-way handshake, the need to judge and get permission to do this login data read and write access connections
- max_connections parameter for controlling a number of connections simultaneously present example MySQL limit
  - The system will refuse the connection request next, and error prompt "Too many connections"
  - The purpose of this parameter is designed max_connections want to protect MySQL (no brain do not turn up value
- Breaking the Road
  - To dispose of those threads occupied connected but not working
    - Max_connections calculation, not running to see who, as long as it is attached to take up a position count
    - For those who do not need to maintain a connection, we can kill connection + id initiative kicks off
      - Foreign affairs priority connection idle for too long off, if not enough, then consider disconnecting idle for too long within the matters connected
      - After the end of active service is disconnected, the client will receive a request at the initiative of "lost MySQL connection" error
  - Reducing the consumption of the connection process, so skip the permissions database validation phase
    - The method of validation is to skip the permissions: restart the database, and to start using the parameters -skip-grant-tables
      - Skip all the permissions validation phase, including the connection process and the implementation process, including statements
    - This program is not recommended, especially at your library scene outside the network can access
    - After enabling -skip-grant-tables parameter in MySQL version 8.0, the default parameters open the --skip-networking
      - He said this time can only be connected to the local database client
Slow query performance issues
- Cause performance problems slow query, generally have the following three possibilities
  - The index is not a good design
    - To solve (Online DDL, gh-ost by creating an index of emergency
  - SQL statement is not written
    - MySQL 5.7 provides query_rewrite function, to change one input statement is rewritten into another mode
  - MySQL wrong index
    - force index + query_rewrite
- In fact, the most frequent first two, through preventive measures ahead of time far better than emergency fire fighting
  - Before on-line regression testing (by slow log, Rows_examined etc.
QPS spurt problem
- Business sudden peak or application bug, leading to the sudden increase in QPS a statement, MySQL pressure is too large impact service
- The ideal situation is to get the business off this feature, the service will naturally recover
- If the end of the process from the database, then there is a corresponding method can be used for different scenes
  - One is a new bug due to service (white list can be directly removed from the end of the database
  - This new feature is the use of a separate database user (with the user to delete the administrator account, and then disconnect the existing connection
  - If the above can not be processed by the statement to limit (query rewrite function, the most stressful SQL rewrite select 1
    - High risk, it may cause injury, and lead to failure along the back of the business logic (the lowest priority.
- In fact, Options 1 and 2 must rely on standardized operation and maintenance system: virtualization, whitelist mechanism, separate business account
  - Thus, more preparation , often means a more stable system

Log Integrity

Prospects Summary: As long as redo log and binlog ensure persisted to disk, you can ensure that MySQL abnormal restart, the data can be restored
WAL mechanism is mainly due to two aspects
- redo log and binlog are sequential write, write sequential disk faster than random write speed
- Group delivery mechanism, can significantly reduce the consumption of disk IOPS
MySQL performance bottlenecks appeared (on IO), consider the following three methods
- Group submitted (parameters binlog_group_commit_sync_delay, binlog_group_commit_sync_no_delay_count
  - Binlog reduce the number of disk write, may increase the response time of the statement, but there is no risk of losing data
- The sync_binlog a value greater than 1 (100 to 1000 is common), but the host will lose binlog log down
- The innodb_flush_log_at_trx_commit set to 2, but the host will lose data when power failure
  - 0 setting is not recommended, because MySQL reboots will lose the data is written to the page cache and speed have been quickly
binlog-writing mechanism
- During the transaction, first log written binlog cache, when the transaction commits and then binlog cache file writes binlog
  - binlog a transaction can not be disassembled, so no matter how much this matters, but also to ensure that the write-once
- Each thread is a parameter for controlling the size of a single binlog_cache_size binlog cache memory associated with the thread
  - Exceeds the size limit of this parameter, it is necessary to scratch disk
- binlog write disk state
  - Each thread has its own binlog cache, but share the same file a binlog
  - write: The page cache is written to the log file system is faster (speed
  - fsync: data persistence (accounting for disk-to-disk IOPS operations
  - sync_binlog control the timing of write and fsync
    - sync_binlog = 0 when indicate that each transaction is committed only write, not fsync
      - Taking into account the loss of controllability of the log volume, it is generally not recommended that this parameter is set to 0
    - sync_binlog = 1, when the transaction will be executed each time you submit indicate fsync
    - sync_binlog = N (N> 1) when transactions are submitted to represent each write, but the cumulative N transactions after fsync
      - If the host binlog log abnormal restart, lost the last N transactions
Write mechanism of the redo log
- Affairs in the implementation process, generated redo log is first written to the redo log buffer does not require direct persisted to disk
  - If abnormal restart, this part of the log on the lost, but because the transaction did not commit, will not be lost loss
- MySQL redo log storage status
  - There is redo log buffer in memory is in the process of MySQL, is the red part of the figure (fast physically
  - Written to disk (write), but no persistent (fsync), is in the page cache file system's (fast physically
  - Persisted to disk, corresponding to the hard disk, which is the green part of the figure (slow, accounting for the same disk IOPS
  - innodb_flush_log_at_trx_commit parameter control redo log write strategy
    - 0: just stay in the redo log redo log buffer in the transaction commits
    - 1: will redo log directly persisted to disk when the transaction commits
    - 2: Every time a transaction is committed are just redo log writes page cache
  - You may be a "not submitted" affairs of the redo log is written to disk scenes
    - Background thread once per second poll (the redo log buffer is written to the log page cache and persistent fsync
    - redo log buffer space occupied will soon reach half innodb_log_buffer_size background thread will take the initiative to write disk
      - Just write, not fsync
    - When concurrent transaction commit, incidentally, this transaction redo log buffer persisted to disk
  - "Dual 1" Configuration: sync_binlog innodb_flush_log_at_trx_commit are set to 1 and
    - Complete a transaction before submitting necessary to wait for the disk to brush twice, once redo log (prepare phase), once binlog
- Group to the (group commit) mechanism
  - Logical log sequence number (log sequence number, LSN):
    - LSN is monotonically increasing, to a point corresponding to the number of writing the redo log
    - Each write length length redo log, the value will be plus length LSN
  - LSN also writes InnoDB data page, the page to ensure that data will not be repeated multiple times redo log
  - chestnut
    - trx1 was the first to arrive, it will be chosen as the leader of this group
    - So when trx1 to start writing the disc, the group which has been three transactions, LSN into a 160
    - trx1 to write the disc when the belt is LSN = 160 (redo log 160 or less average persistence
    - trx2 and trx3 direct return
    - A group which submitted the better, the more team members, save disk IOPS of
  - MySQL optimize the use of the group submitted: the time delay can be set to submit the binlog
    - After MySQL in order to allow the group to submit a better effect, the redo log time to do fsync drag binlog write
      - binlog write: binlog written binlog files on the disk from the binlog cache
    - Binlog effect of Group contributions are usually not as good as redo log group to submit effective (redo log fsync executed soon
    - Parameters (or both for the relationship to meet to enhance the effect binlog group submitted a then call fsync
      - binlog_group_commit_sync_delay parameters: the number of milliseconds delay after the call fsync
      - Just call fsync after the cumulative number of times: binlog_group_commit_sync_no_delay_count parameters
Log-related issues
- Why binlog cache is maintained by each thread its own, while the redo log buffer is globally shared?
  - The main reason is binlog is not "interrupted", binlog a transaction must be continuous write (and other matters submitted written End
  - redo log does not have this requirement, the resulting log can be written to the intermediate redo log Buffer, may also be persistent pickup
- Not to the time of submission stage crash occurred during the execution of a transaction, then, redo log is missing, this will not result in the primary and inconsistent it?
  - Not; binlog still binlog cache, no issue prepared by the library (see the cause of the crash from a business point of view has not submitted
- Database crash-safe guarantee was
  - If the client receives a successful transaction message, the transaction must persist up
  - If the client receives a transaction fails (such as primary key conflicts, rollback, etc.) of a message, the transaction must fail
  - If the client receives the message "abnormality", the application needs to continue with the subsequent re-connection by querying the current state of the logic
    - In this case only to ensure that the database (between the data and the logs, between the primary and the standby database library) can be a consistent internal

D.Chuan

Published 98 original articles · won 197 Like · views 60000 +

Private letter concerns

"Note" MySQL combat 45 say - practice articles (b)

Sql Bad Case

Slow Situation

Magic Reading

Lock rules

show time

Log Integrity

Guess you like