Thumbs up! 7000 word study notes, MySQL from entry to to give up

Introduction to MySQL database

MySQL has been ranked second in the past two years, and may surpass Oracle's plan to be promoted to first place at any time, because the performance of MySQL has been optimized, and the security mechanism has gradually matured, and more importantly, it is open source and free.

MySQL is a relational database management system. Relational databases store data in different tables instead of putting all data in a large warehouse, which increases speed and flexibility.

The SQL language used by MySQL is the most commonly used standardized language for accessing databases. MySQL software adopts a dual authorization policy, divided into community version and commercial version. Due to its small size, fast speed, low total cost of ownership, especially the characteristics of open source, the development of small and medium-sized websites generally chooses MySQL as the website database.

If you can't install MySQL, please move: MySQL service installation

MySQL InnoDB storage engine

The storage engine InnoDB is the default storage engine of the current MySQL version, and it is also the storage engine recommended by MySQL. It is a storage engine that combines high reliability and high performance.
In MySQL 5.7 version, unless the default storage engine is explicitly specified in the configuration file or the engine= statement is used to specify other storage engines when creating the table, the default is InnoDB.

Advantages of InnoDB storage engine:

DML statements support transaction functions and ensure ACID characteristics
The use of row-level locks ensures high concurrency properties
InnoDB optimizes query performance for tables with a primary key based on the primary key, also known as a clustered index, and stores all data in a clustered index to reduce the IO consumption of primary key queries
To ensure data consistency, InnoDB also supports foreign key attributes to ensure that there will be no inconsistent data between tables with foreign key constraints
When the server hardware or software failure causes MySQL to restart, InnoDB will automatically identify the data that has been submitted before the failure, and roll back all the data that was not submitted at the time of the failure, so as to maximize the protection of data from being lost (crash recovery)

1. Transaction
2. MVCC (multi-version concurrency control)
3. Row-level Lock
4. Support foreign keys
5. ACSR (Auto Crrash safe Recovery) automatic fail-safe recovery
6. Support hot backup

MySQL replication cluster principle and actual combat

There are two methods for MySQL replication:

Traditional way: Based on the bin-log of the master library, the log events and event locations are copied to the slave library, and the slave library is applied to achieve the purpose of master-slave synchronization.

Gtid method: global transaction identifiers replicate data based on transactions, so it does not rely on the location of the log file, and at the same time, it can better ensure the consistency of the master and slave database data.

The actual process of master-slave synchronization of MySQL database

The "pits" you don't know about the MySQL master-slave synchronization architecture (part 1)

The "pits" you don't know about in the MySQL master-slave synchronization architecture (below)

Multiple data backup methods:

Physical backup refers to the way to complete the backup by copying database files. This backup method is suitable for databases that are large, important and need to be quickly restored.
Logical backup refers to the way to complete the backup by backing up the logical structure of the database (create database/table statement) and data content (insert statement or text file). This backup method is suitable for situations where the database is not very large, or you need to make certain modifications to the exported files, or if you want to re-establish the database on a different type of server
Normally, the speed of physical backup is faster than logical backup. In addition, the granularity range of backup and recovery of physical backup is the entire database or a single file. Whether a single table has the ability to recover depends on the storage engine. For example, under the MyISAM storage engine, each table corresponds to an independent file and can be restored separately; but for the InnoDB storage engine table, each representation may correspond to an independent file. It is also possible that the table uses a shared data file
Physical backup is usually performed when the database is closed, but if it is performed when the database is running, the database must not be modified during the backup.
The speed of logical backup is slower than physical backup because logical backup needs to access the database and convert the content into the format required by logical backup; usually the output backup file size is also larger than physical backup; in addition, logical backup does not include database configuration File and log file content; the granularity of backup and recovery can be all databases, a single database, or a single table; logical backup needs to be executed while the database is running; its execution tool can be mysqldump or select... into outfile two ways

Send you a copy of the production database backup plan : Gaoqige enterprise-level MySQL database backup plan

MySQL database physical backup method : Xtrabackup realizes data backup and recovery

There are many types of MySQL replication:

Asynchronous replication: A master library, one or more slave libraries, and the data is asynchronously synchronized to the slave library.
Synchronous replication: a unique replication method in MySQL Cluster.
Semi-synchronous replication: On the basis of asynchronous replication, it is ensured that at least one slave database has received the transaction and recorded it in the log before committing any transaction on the main database.
Delayed replication: On the basis of asynchronous replication, artificially set the data synchronization delay time of the master library and the slave library, that is, ensure that the data delay is at least this parameter.

MySQL master-slave replication delay solution : a solution for high-availability database master-slave replication delay

MySQL high-availability architecture design and actual combat

Let’s first understand the introduction of MySQL high-availability architecture: Talking about MySQL cluster high-availability architecture
MySQL high-availability solution: MySQL synchronization replication and high-availability solution summary The
official also provides a high-availability solution : official tools｜MySQL Router high-availability principle and actual combat

MHA

MHA (Master High Availability) is currently a relatively mature solution for MySQL high availability. The software consists of two parts: MHA Manager (management node) and MHA Node (data node).
MHA Manager: It can be deployed on a separate machine to manage multiple master-slave clusters, or it can be deployed on a slave node.
MHA Node: Run on every MySQL server.
MHA Manager will periodically detect the master node in the cluster. When the master fails, it can automatically promote the slave with the latest data to the new master, and then redirect all other slaves to the new master. The entire failover process is completely transparent to the application.

MHA high-availability program combat : MySQL cluster high-availability architecture MHA

MGR

Mysql Group Replication (MGR) is a brand new highly available and highly expanded MySQL cluster service released from version 5.7.17.
High consistency, group replication technology based on native replication and paxos protocol, provides consistent data security guarantee through plug-ins;
High fault tolerance, most services can continue to work normally, automatic detection of resource requisition conflicts between different nodes, prioritized processing in order, built-in anti-brain mechanism;
High scalability, automatically add and remove nodes, and update group information;
High flexibility, single-master mode and multi-master mode. Single master mode automatically selects the master, and all update operations are performed on the master; in multi master mode, all servers are updated at the same time.

MySQL performance optimization

The most complete MySQL high-performance optimization actual combat summary in history!
The principle of MySQL index: What is the MySQL index? How to optimize?

As the name implies, B-tree indexes use the B-tree data structure to store data. Different storage engines use B-Tree indexes in different ways. For example, MyISAM uses prefix compression technology to make the index space smaller, while InnoDB stores data in the original data format. , And the MyISAM index records the physical location of the corresponding data in the index, while InnoDB records the corresponding primary key value in the index. B-Tree usually means that all values are stored in order, and the distance from each leaf page to the root is the same.
The B-Tree index drives the storage engine to no longer obtain data through a full table scan, but to search from the root node of the index. Both the root node and the intermediate node store pointers to lower-level nodes. By comparing the value of the node page and the search The value can find the appropriate pointer to enter the lower-level child nodes until the lowest-level leaf node. The final result is either to find the corresponding value or not find the corresponding value. The depth of the entire B-tree tree is directly related to the size of the table.
Full key-value matching: match all columns in the index, such as finding a person whose name is zhang san, who was born on January 1, 1982
Match the leftmost prefix: match with the leftmost column in the index, for example, find all people whose last name is zhang
Match column prefix: match the beginning of the leftmost column of the index, such as finding all people whose names start with z
Matching range value: match the range area value of the index column, such as finding people whose last name is between li and wang
Exact match on the left column and range match on the right column: For example, find all people whose last name is Zhang and whose first name starts with K
Only access index query: query results can be obtained through the index, also called a covering index, such as finding the names of all people whose surname is zhang

Introduction to MySQL table partitioning : a thorough understanding of MySQL partitioning

You can allow more data to be stored in one table, breaking through disk restrictions or file system restrictions.
It is easy to remove the expired or historical data from the table partition in the table partition, as long as the corresponding partition is removed.
For some query and modification statements, the data range can be automatically reduced to one or more table partitions to optimize the efficiency of statement execution. And you can execute the statement by displaying the specified table partition, such as select * from temp partition(p1,p2) where store_id <5;
Table partitioning is to divide the data of a table into different logical blocks according to a certain rule, and store them separately. This rule is called a partition function, and there can be different partitioning rules.
In MySQL 5.7 version, you can use the show plugins statement to check whether MySQL supports table partitioning.
The MySQL8.0 version removes the display of partition in show plugins, but the table partition function in the community version is enabled by default.
But when the table contains a primary key or unique key, each field used as a partition function must be all or part of the unique key and primary key in the table, otherwise the partition table cannot be created.

MySQL sub-database sub-table

If there is no division, no division. Tables within 10 million are not recommended to be fragmented. Through appropriate indexing, read-write separation, etc., performance problems can be solved well.
The number of shards should be as small as possible, and the shards should be evenly distributed across multiple DataHosts, because the more a query SQL spans across shards, the worse the overall performance, although it is better than the result of all data in one shard, only when necessary Expand capacity and increase the number of shards.
Fragmentation rules need to be carefully selected. The selection of fragmentation rules needs to consider data growth mode, data access mode, fragmentation relevance issues, and fragmentation expansion issues. The most recent fragmentation strategy is range fragmentation and enumeration fragmentation. Slices, consistent Hash shards, these types of shards are conducive to expansion.
Try not to span multiple shards in SQL in a transaction. Distributed transactions have always been a difficult problem to handle.
Optimize the query conditions as much as possible, try to avoid the Select * method, a large number of data result sets will consume a lot of bandwidth and CPU resources, query try to avoid returning a large number of result sets, and try to build indexes for frequently used query statements.

Overview of database sub-database sub-table : When is the database sub-database sub-table? How to divide?

Mysql sub-database sub-table solution : MySQL sub-database sub-table solution, the summary is very good!

Mysql sub-database sub-table thinking : rescue DBA-database sub-database sub-table thinking and case analysis

MySQL database read and write separation and high availability

The storage and access of massive data has become a bottleneck problem in the system design. The increasing business data will undoubtedly cause a considerable load on the database, and at the same time put forward high requirements for the stability and scalability of the system. With the development of time and business, there will be more and more tables in the database, and the amount of data in the tables will also increase. Accordingly, the overhead of data operations will also increase; in addition, no matter how the hardware is upgraded Resources, the resources of a single server (CPU, disk, memory, network IO, number of transactions, number of connections) are always limited, and ultimately the amount of data and data processing capabilities that the database can carry will encounter bottlenecks. Sub-table, sub-database and read-write separation can effectively reduce the pressure of a single database.

Practical case of MySQL read-write separation high-availability architecture:

ProxySQL+Mysql realizes the separation of database reading and writing in practice

Mysql+Mycat realizes database master-slave synchronization and separation of read and write

MySQL performance monitoring

The indicators of MySQL performance monitoring can be roughly divided into the following four categories:

Query throughput
Query delays and errors
Client connection and error
Buffer pool utilization

For MySQL performance monitoring, the official also provides related service plug-ins: MySQL-Percona, the following briefly introduces the installation of plug-ins


[root@db01 ~]# yum -y install php php-mysql

[root@db01 ~]# wget https://www.percona.com/downloads/percona-monitoring-plugins/percona-monitoring-plugins-1.1.8/binary/redhat/7/x86_64/percona-zabbix-templates-1.1.8-1.noarch.rpm

[root@db01 ~]# rpm -ivh percona-zabbix-templates-1.1.8-1.noarch.rpm

warning: percona-zabbix-templates-1.1.8-1.noarch.rpm: Header V4 DSA/SHA1 Signature, key ID cd2efd2a: NOKEY

Preparing... ################################# [100%]

Updating / installing...

   1:percona-zabbix-templates-1.1.8-1 ################################# [100%]

Scripts are installed to /var/lib/zabbix/percona/scripts

Templates are installed to /var/lib/zabbix/percona/templates

Finally, you can cooperate with other monitoring tools to realize the performance monitoring of MySQL.

MySQL server configuration plugin:

Modify the php script to connect to the monitor@localhost user of MySQL
Modify the MySQL sock file path


[root@db01 ~]# sed -i '30c $mysql_user = "monitor";' /var/lib/zabbix/percona/scripts/ss_get_mysql_stats.php
[root@db01 ~]# sed -i '31c $mysql_pass = "123456";' /var/lib/zabbix/percona/scripts/ss_get_mysql_stats.php
[root@db01 ~]# sed -i '33c $mysql_socket = "/tmp/mysql.sock";' /var/lib/zabbix/percona/scripts/ss_get_mysql_stats.php

Test whether it is available (the monitoring value can be obtained from MySQL)

[root@db01 ~]# /usr/bin/php -q /var/lib/zabbix/percona/scripts/ss_get_mysql_stats.php --host localhost --items gg
gg:12

# 确保当前文件的 属主 属组 是zabbix，否则zabbix监控取值错误。
[root@db01 ~]# ll -sh /tmp/localhost-mysql_cacti_stats.txt
4.0K -rw-rw-r-- 1 zabbix zabbix 1.3K Dec 5 17:34 /tmp/localhost-mysql_cacti_stats.txt

Move the zabbix-agent configuration file to the /etc/zabbix/zabbix_agentd.d/ directory


[root@db01 ~]# mv /var/lib/zabbix/percona/templates/userparameter_percona_mysql.conf /etc/zabbix/zabbix_agentd.d/
[root@db01 ~]# systemctl restart zabbix-agent.service

Import and configure Zabbix template and host: The
default template monitoring time is 5 minutes (the current test is modified to 30s) and the Zabbix template time should also be modified

# 如果要修改监控获取值的时间不但要在zabbix面板修改取值时间，bash脚本也要修改。
[root@db01 scripts]# sed -n '/TIMEFLM/p' /var/lib/zabbix/percona/scripts/get_mysql_stats_wrapper.sh
TIMEFLM=`stat -c %Y /tmp/$HOST-mysql_cacti_stats.txt`
if [ `expr $TIMENOW - $TIMEFLM` -gt 300 ]; then   
# 这个 300 代表 300s 同时也要修改。

The default template version is 2.0.9, which cannot be used in version 4.0. You can export from version 3.0 first, and then import version 4.0.

In fact, in the actual production process, there are still relevant third-party open source software for professional monitoring of databases. Migrant workers have also written related articles before, and they are published today for your reference: Lepus, a powerful open source enterprise-level database monitoring tool

MySQL user behavior safety

Suppose that you are a mysql-DBA of a certain company. Suddenly all the data in the company's database was deleted artificially one day.
Although there is data backup, the loss caused by the service suspension is tens of millions. Now the company needs to find out who did the delete operation.
However, there are many people who have database operation permissions. How to investigate and where is the evidence?
Do you feel powerless?
Mysql itself does not have the function of operation audit. Does that mean that you can only admit that you are unlucky when you encounter this situation?

After learning, you need to go out and practice, and finally give you some corporate interview questions for everyone to practice: 24 database interview questions that must be mastered!

Note: This article was originally submitted by the author. The old A-Operation and Maintenance Back-Guard, from Shandong Province, is engaged in first-line application operation and maintenance and database operation and maintenance. Good at MySQL database, worked in the cloud computing department of Shandong Computing Center (National Supercomputing Jinan Center). Now works for a well-known domestic MySQL database manufacturer. Original: https://dbawsp.com/626.html, migrant workers made relevant adaptations, and the copyright belongs to the original author .

end