There are two types of database applications, OLTP (Online Transaction Processing) and OLAP (Online Analytical Processing). OLAP is generally used in data warehouses (a relational database model in which historical data and metadata from one or more source databases are stored) or data marts (an access layer of the data warehouse, used to extract relevant data from the data warehouse In general, complex SQL statements are required to query; OLTP is mostly used in daily transaction processing applications, and the database capacity is relatively small compared to OLAP.
The InnoDB engine is generally used for OLTP database applications. The characteristics of this application are as follows:
1. The amount of concurrent user operations is large.
2. Transaction processing time is generally relatively short.
3. The query statement is relatively simple, and the index is generally used.
4. Less complex queries.
It can be seen from the above characteristics that OLTP database applications do not require high CPU, because only complex queries can perform CPU-consuming tasks such as comparison, sorting, and connection. It can be said that OLAP is a CPU-intensive operation, and OLTP is an IO-intensive operation (more attention should be paid to improving the IO configuration when purchasing).
In order to get more memory support, the CPU must support 64-bit applications, otherwise it cannot support the installation of 64-bit operating systems.
From the perspective of the design architecture of the InnoDB engine, the main background operations are completed in a single MASTER THREAD, which does not support multi-core applications well. The new InnoDB Plugin version greatly improves the processing performance of multi-core CPUs. , You can also increase the IO thread by modifying the parameters innodb_read_io_threads and innodb_write_io_threads, which can also make full use of the CPU multi-core performance.
In the current version, one SQL statement can only work on one CPU. Because the operation of OLTP database applications is generally very simple, it has little impact on OLTP applications, but multiple CPUs or multi-core CPUs are very helpful for large concurrent requests.
The total size of the data file and index in the above test is 18GB. As the buffer pool increases, the test result TPS (Transaction Per Second) will increase linearly. When the buffer pool increases to 20GB, the database performance is greatly improved. Because the size of the buffer pool is larger than the size of the data file itself, operations on the data file can be performed in memory. At this time, the performance is optimal. If the buffer pool is increased, the database performance can no longer be improved.
Determine whether the database memory has reached the bottleneck: the
meaning of the parameters:
1. Innodb_buffer_pool_reads: the number of times the page is read from the physical disk.
2. Innodb_buffer_pool_read_ahead: the number of pre-reading.
3. Innodb_buffer_pool_read_ahead_evicted: The number of pages that are replaced from the buffer pool without being read. It is generally used to determine the efficiency of pre-reading.
4. Innodb_buffer_pool_read_requests: the number of read requests to the buffer pool.
5. Innodb_data_read: the total number of bytes read in.
6. Innodb_data_reads: the number of read requests initiated, each read may need to read multiple pages.
According to the above parameters, the hit rate of the buffer pool can be calculated (usually it should be guaranteed to be above 99%): the
(innodb_buffer_pool_read_requests) / (Innodb_buffer_read_requests + Innodb_buffer_pool_read_ahead + Innodb_buffer_pool_reads)
average number of bytes read per time:
innodb_data_read / innodb_data_reads
Even if the buffer pool size is larger than the database size, it does not mean that there is no disk operation. The background master thread is also responsible for writing dirty pages to disk asynchronously, and it needs to be written to the redo log file immediately every time a transaction is committed.
The server field generally uses mechanical hard disks with SAS or SATA interfaces. Mechanical hard disks have two important indicators: seek time and rotation speed. Random access to mechanical hard disks requires a long period of head rotation and positioning to find, and the speed of sequential access is much higher than that of random access. Many characteristics of the database are also trying to make full use of the characteristics of sequential access. Multiple mechanical hard disks can be formed into RAID to improve database performance, and data files can also be distributed on different hard disks to achieve load balancing.
The solid state drive is composed of flash memory, which has low latency, low power consumption and shock resistance. Flash memory is a complete electronic device, without the read and write heads of traditional mechanical hard drives, solid state drives can provide consistent random access time. The data in flash memory is not updatable and can only be overwritten by sector. Before overwriting and overwriting, a very time-consuming erasing operation needs to be performed. The erasing operation cannot be completed on the sector, but must be erased. In addition to the completion of the block, the erase block is usually 128KB, but each erase block has a limit on the number of erasing and writing. Now there are some algorithms to solve this problem. Due to writing problems, the solid-state reading speed is higher than the writing speed, and its reading performance should be used to avoid excessive writing operations.
For the optimization of the solid state drive in the InnoDB engine, you can increase the value of the innodb_io_capacity parameter (the number of I/O operations per second (IOPS) available for InnoDB background tasks).
The basic idea of RAID (Redundant Array of Independent Disks) is to combine multiple relatively inexpensive hard disks into a disk array, so that the performance can reach or even exceed that of an expensive hard disk with huge capacity. Since multiple hard disks are combined into a logical sector, RAID looks like a single hard disk or logical storage unit, and the operating system will only treat it as a hard disk.
According to different disk combination methods, common RAID combination methods:
1. RAID 0: Combine multiple disks into a large disk without redundancy, parallel IO, and the fastest speed. RAID 0 is also called a stripe set. When storing data, the data is segmented according to the number of disks, and the data is written to the corresponding disk at the same time. In theory, the performance of multiple disks is equal to 单一磁盘效能 * 磁盘数, but in fact, limited by the bus IO bottleneck and other factors, RAID performance will decrease with the margin.
2. RAID 1: More than two groups of N disks are mirror images of each other. In a multi-threaded operating system, the read speed is very good, but the write speed is slightly lower. Unless the primary disk and the mirror with the same data are damaged at the same time, as long as one disk is normal, it can maintain operation with high reliability. The principle of mirroring is that when data is stored on the main hard disk, the same data is also written on the mirrored hard disk. No matter how many sets of disks are used, RAID 1 only counts it as the capacity of a set of disks, and the disk utilization is low.
3. RAID 5: It is a solution that takes into account storage performance, data security, and storage cost. It uses Disk Striping (hard disk partitioning) technology. RAID 5 requires at least three hard disks (two disks for data, one disk for two Disk parity information), it does not back up the stored data, but stores the data and the corresponding parity information on each of the disks that make up the RAID 5, and stores the data and the corresponding parity information To different disks, when a disk data of RAID 5 is damaged, the remaining data and the corresponding parity information can be used to recover the damaged data (assuming there are three copies of data A1=00000111, A2=00000101 and A3= 00000000, three pieces of data are XORed to get the result Ap=00000010, if A2 fails to access at this time, you can passA1 XOR A3 XOR Ap = 00000101The value of A2 is calculated. The principle is that the result of the exclusive OR of any two identical numbers is 0, and the exclusive OR of 0 and any number is the number itself, and the exclusive OR has a commutative law). RAID 5 can be understood as a compromise between RAID 0 and RAID 1. The security capability of RAID 5 is lower than that of mirroring, and the disk space utilization rate is higher than that of mirroring. The read speed of RAID 5 is close to RAID 0, but the write speed is quite slow due to the additional parity information. Using Write Back can improve performance.
4. RAID 10 and RAID 01: RAID 10 is mirroring first, then striping:
RAID 01 is striping first, then mirroring:
RAID 10 is more secure, assuming Disk0 is damaged, RAID 10 is also on Disk1 at the same time Data is lost only when it is damaged. Simply calculate the damage rate to be 1/3; for RAID 01, it is RAID 0 first, so Disk0 and Disk1 are a logical disk, which is equivalent to both of them are broken. At this time, if Disk2 or If one of Disk3 is damaged (equivalent to Disk2 and Disk3 being damaged as a whole), the data will be damaged. Simply calculate the damage rate to 2/3. The disadvantage of these two methods is that more hard disks are required, and they can only be used when there are at least four even-numbered hard drives.
(5) RAID 50: It is also called mirrored array stripe. It requires at least six hard disks. Its structure is as follows:
RAID Write Back means that the RAID controller puts the data to be written into its own cache and writes their actual data. Arrange to be executed later, so that there is no need to wait for the actual writing of the physical disk to complete, and the performance is significantly improved. This method can significantly improve the performance of operations such as writing to the redo log of the database, writing to the binary log, and refreshing dirty pages when sync_binlog is set to 1.
If an accident occurs to the system, Write Back may occur before the write failure. For this, most hardware RAID cards provide a battery backup unit (BBU, Battery Backup Unit), so you can safely turn on this function.
If Write Back is not enabled, the RAID card setting shows Write Through, which does not buffer writes, so the write performance may be poor, but it is the safest write.
Even if Write Back is turned on, the RAID card may enable Write Through. The prerequisite for the safe use of Write Back is that the RAID card has a battery backup unit. To ensure the validity of the battery, the RAID card will check the battery status regularly, and when the battery is low Charge it. During charging, the Write Back function will be switched to the safest Write Through.
Insert 20W data into the database (through 20W transactions) to test the performance difference between Write Back and Write Through:
In Write Through mode, the performance can also be improved by setting the parameter innodb_flush_log_at_trx_commit to 0. At this time, redo log writing is not When each transaction is submitted, but when the background master thread automatically refreshes every second, the physical disk write request is reduced and the execution speed is improved.
The RAID card can enter a BIOS-like configuration interface for various settings when the server is started. Many manufacturers have also developed software under various operating systems to configure the RAID card.
There is little difference in performance of various file systems, but the functions provided need to be paid attention to. For example, ZFS can support snapshots, so that LVM snapshots are not needed.
Benchmark testing tools can be used to compare the performance of databases or operating systems after tuning. In addition to the excellent tools provided by MySQL itself, there are better and more commonly used sysbench and mysql-tpcc.
sysbench is a modular, cross-platform, multi-threaded benchmark test tool, mainly including the following test methods: 1.
CPU performance.
2. Disk IO performance.
3. Scheduler performance.
4. Memory allocation and transmission speed.
5. POSIX thread performance.
6. Database OLTP benchmark test.
TPC (Transaction Processing Performance Council) is a non-profit organization that evaluates the software and hardware performance of large database systems. TPC-C is formulated by the TPC Association to test the performance of typical complex OLTP systems (commonly used in academia and industry).
TPC-C uses 3NF (Third Normal Form) to simulate a warehouse sales supplier company, including nine basic relationships:
TPC-C performance measurement unit is tpmC (transaction per minute, C stands for TPC C benchmark test), the value is more Larger, higher transaction processing performance.
tpcc-mysql is an open source TPC-C test tool that fully complies with the TPC-C standard.