Experience: What is the impact the database query speed, what impact the performance of MySQL (rpm)

First, what impact the database query speed

Four factors that affect the database query speed of 1.1

    

1.2 Risk Analysis

QPS: Queries Per Second means "query per second rate", is a server capable of the appropriate number of queries per second, is a measure of a particular query server within the specified time how much of the processing flow.

TPS: is TransactionsPerSecond the abbreviation, which is the number of transactions / second. It is the unit of measurement software test results. The client requests the transmission start timing, end timing after receiving the server response, in order to calculate the time of use and the number of transactions completed.

    Tips: best not to back up the database in the main library, to cancel such plans before major events.

1. Inefficiency sql: ultra-high QPSand TPS.
2. Large number of concurrent: data connections are filled ( max_connectionby default 100, the general set a large number of connections).
   Concurrency: the number of requests for the same database server processing time
3. Ultra-high CPUusage: CPUresource depletion downtime.

4. Disk IO: Disk IOPerformance sudden drop, consume a lot of disk performance of scheduled tasks. Solution: Faster disk device, adjust the scheduled tasks, good disk maintenance.

1.3 NIC traffic: how to avoid the situation can not connect to the database

1. Decrease (copied from the master server from the server logs) from the number of servers
2. Classifying the cache (cache miss avoid a large number of front end)
3. Avoid using select * queries
4. Separate business network and server network

1.4 brings to the table a large problem ( 重要)

1.4.1 features a large table

1. The number of rows huge, over ten million single-table
2. Table huge data files over 10aG

1.4.2 Hazard large table

1. Slow queries: in a short time is difficult to filter out the data needed to
    lower the query word discrimination -> To filter out a large amount of data in the table which part of the data will produce large amounts of disk io -> reduce disk efficiency

2. DDLImpact:

    Indexing takes a long time:

• MySQL -v<5.5 Indexing will lock table
• MySQL -v>=5.5 Indexing will result in a master-slave delay ( mysqlindexing, first in the group on execution, and then executed on the library)

    Long time is required to modify the structure of the table lock table: the primary cause long ( 'delay of 480 seconds ") from the delay

1.4.3 How to deal with a large table on the database

Sub-library sub-table to a large table into multiple small tables

    difficulty:

1. Select part table primary key
2. Query and statistical data across partitions of the points table

1.5 large transaction problems caused by ( 重要)

1.5.1 What is a transaction

    

1.5.2 affairs ACIDproperties

1, atomicity ( atomicity ): all succeed, fail rolled back. Bank deposit.

2, consistency (consistent): the total amount of bank transfers remain unchanged.

3, isolation (isolation):

    Isolation rating:

• Uncommitted read ( READ UNCOMMITED) dirty read, seen between the two transactions each other;
• Read Committed ( READ COMMITEDbasic concepts) in line with the isolation, a transaction carried out, other things have been submitted to the transaction are visible, that is, other transactions can obtain the data submitted.
• Repeatable Read ( REPEATABLE READ)  InnoDB的默认隔离等级. When the transactions, all other transactions not visible, that perform read many times, the result is the same!
• Serializable ( SERIALIZABLE) on each line of data are read lock, will cause a lot of lock timeout and lock requisition, strict data consistency and concurrency is not available.

    Transaction isolation level view of the system: show variables like '%iso%';
    open a new transaction: begin;
    commit a transaction: commit;
    modify the isolation level things: set session tx_isolation='read-committed';

4, persistence ( DURABILITY ): From the perspective of the persistent database, the disk would not damage

    

    redo logMechanisms to ensure transaction consistency and durability updated

1.5.3 large transaction

Run for a long time, more data transaction operation;

    Risk: locking too much data, roll back a long time, a long execution time.

1. Locking too much data, causing heavy congestion and lock out;
2. The time required to roll back a long time, and the data will still be in locked;
3. If the execution time, it will cause a delay from the master, because only when the primary server to perform all written to the log when finished, will start from the server to synchronize, causing delays.

    Solutions:

1. Avoid a too much data processing to be processed in batches;
2. Removal of unnecessary SELECToperations, to ensure that a transaction is only necessary to write.

Second, what impact the performance of MySQL ( 非常重要)

2.1 several aspects that affect performance

1. Server hardware.
2. The server system (system parameter optimization).
3. Storage engine. 
   MyISAM: Not support transactions, table-level locking.
   InnoDB: Support services, support row-level locking, transactions ACID.
4. Database configuration parameters.
5. 数据库结构设计和SQL语句。（重点优化）

2.2 MySQL architecture

    Divided into three layers: Client -> Services Layer -> Storage Engine

    

1. MySQLShi 插件式的存储引擎, of which storage engine for very many. As long as the interface to achieve compliance with mysql storage engine, can develop their own storage engine!
2. All across storage engine functions are implemented in the service layer.
3. MySQL storage engine is for the table, not for the library. That can use different storage engines in a database. But this is not recommended.

2.3 InnoDB storage engine

    MySQL5.5And after versions of the default storage engine: InnoDB.

Table 2.3.1 InnoDB data storage space.

show variables like 'innodb_file_per_table

  如果innodb_file_per_table 为 ON 将建立独立的表空间，文件为tablename.ibd；

  如果innodb_file_per_table 为 OFF 将数据存储到系统的共享表空间，文件为ibdataX（X为从1开始的整数）；

  .frm ：是服务器层面产生的文件，类似服务器层的数据字典，记录表结构。

2.3.2 (MySQL5.5默认)系统表空间与(MySQL5.6及以后默认)独立表空间

    1.1 系统表空间无法简单的收缩文件大小，造成空间浪费，并会产生大量的磁盘碎片。

    1.2 独立表空间可以通过optimeze table 收缩系统文件，不需要重启服务器也不会影响对表的正常访问。

    2.1 如果对多个表进行刷新时，实际上是顺序进行的，会产生IO瓶颈。

    2.2 独立表空间可以同时向多个文件刷新数据。

强烈建立对Innodb 使用独立表空间，优化什么的更方便，可控。

2.3.3 系统表空间的表转移到独立表空间中的方法

    1、使用mysqldump 导出所有数据库数据（存储过程、触发器、计划任务一起都要导出 ）可以在从服务器上操作。

    2、停止MYsql 服务器，修改参数（my.cnf加入innodb_file_per_table），并删除Inoodb相关文件（可以重建Data目录）。

    3、重启MYSQL，并重建Innodb系统表空间。

    4、 重新导入数据。

    或者 Alter table 同样可以的转移，但是无法回收系统表空间中占用的空间。

2.4 InnoDB存储引擎的特性

2.4.1 特性一：事务性存储引擎及两个特殊日志类型：Redo Log 和 Undo Log

1. Innodb 是一种事务性存储引擎。
2. 完全支持事务的ACID特性。
3. 支持事务所需要的两个特殊日志类型：Redo Log 和Undo Log

    Redo Log：实现事务的持久性(已提交的事务)。
    Undo Log：未提交的事务，独立于表空间，需要随机访问，可以存储在高性能io设备上。

Undo日志记录某数据被修改前的值，可以用来在事务失败时进行 rollback； Redo日志记录某数据块被修改后的值，可以用来恢复未写入 data file的已成功事务更新的数据。

2.4.2 特性二：支持行级锁

1. InnoDB支持行级锁。
2. 行级锁可以最大程度地支持并发。
3. 行级锁是由存储引擎层实现的。

2.5 什么是锁

2.5.1 锁

    

2.5.2 锁类型

    

2.5.3 锁的粒度

MySQL的事务支持不是绑定在MySQL服务器本身，而是与存储引擎相关

    

    将table_name加表级锁命令：lock table table_name write; 写锁会阻塞其它用户对该表的‘读写’操作，直到写锁被释放：unlock tables；

1. 锁的开销越大，粒度越小，并发度越高。
2. 表级锁通常是在服务器层实现的。
3. 行级锁是存储引擎层实现的。innodb的锁机制，服务器层是不知道的

2.5.4 阻塞和死锁

    （1）阻塞是由于资源不足引起的排队等待现象。
    （2）死锁是由于两个对象在拥有一份资源的情况下申请另一份资源，而另一份资源恰好又是这两对象正持有的，导致两对象无法完成操作，且所持资源无法释放。

2.6 如何选择正确的存储引擎

    参考条件：

1. 事务
2. 备份(Innobd免费在线备份)
3. 崩溃恢复
4. 存储引擎的特有特性

    总结:Innodb大法好。
    注意:尽量别使用混合存储引擎，比如回滚会出问题在线热备问题。

2.7 配置参数

2.7.1 内存配置相关参数

确定可以使用的内存上限。

内存的使用上限不能超过物理内存，否则容易造成内存溢出；（对于32位操作系统，MySQL只能试用3G以下的内存。）

MySQL is determined for each connection 单独 memory usage.

sort_buffer_size #定义了每个线程排序缓存区的大小，MySQL在有查询、需要做排序操作时才会为每个缓冲区分配内存（直接分配该参数的全部内存）；
join_buffer_size #定义了每个线程所使用的连接缓冲区的大小，如果一个查询关联了多张表，MySQL会为每张表分配一个连接缓冲，导致一个查询产生了多个连接缓冲；
read_buffer_size #定义了当对一张MyISAM进行全表扫描时所分配读缓冲池大小，MySQL有查询需要时会为其分配内存，其必须是4k的倍数；
read_rnd_buffer_size #索引缓冲区大小，MySQL有查询需要时会为其分配内存，只会分配需要的大小。 

    注意：These four parameters are assigned to a thread, if there are 100 connection, then the need × 100.

MySQL database instance:

　①MySQL is 单进程多线程(and oracle is a multi-process), that MySQLinstance is a service process performance on a system that processes;

　②MySQL example is composed of threads and memory, for instance the real operation of the database file;

Generally an example operation of one or more databases; clustered where multiple instances of operating one or more databases.

    How to allocate memory for the buffer pool:
    Innodb_buffer_pool_size defines the size of the buffer pool Innodb used, its performance is very important and must be large enough, but too large, so that more time is needed Innodb closed when the dirty pages from the buffer pool to refresh disk;

总内存-（每个线程所需要的内存*连接数）-系统保留内存

    key_buffer_size, Defines the size of the buffer pool MyISAM used, since data is dependent on the operating system stored in the cache, the more memory space to be reserved for the operating system;

select sum(index_length) from information_schema.talbes where engine='myisam'

    Note : Even if the use of all development table Innodb table, but also for MyISAM reserved memory, because the system uses a MySQL table is still MyISAM tables.

    max_connections Control allows the maximum number of connections, generally larger 2000.
    Do not use foreign key constraints to ensure data integrity.

2.8 Performance Optimization order

    From top to bottom: