MySQL storage engine architecture and

MySQL principle chapter

MySQL indexing mechanism

MySQL storage engine architecture and

Detailed MySQL statement execution process

Detailed MySQL implementation plan

MySQL InnoDB buffer pool

MySQL InnoDB transaction

MySQL InnoDB lock

MySQL InnoDB MVCC

MySQL InnoDB achieve high concurrency principle

What is the difference in the RR and RC MySQL InnoDB snapshot read

MySQL architecture

MySQL architecture diagram:

MySQL conceptually divided into four layers, from top to bottom four layers are connected to the network layer, a service layer (core layer), the storage engine layer, a file system layer. Our top-down began to explain.

The network connectivity layer (Client Connectors)

Responsible for connection management, authorization, authentication, security and so on. Each client connection corresponds to a thread on the server. It maintains a thread pool on the server to avoid the destruction of each connection create a thread. When a client connects to a MySQL server, the server to be authenticated. Yes, it can be authenticated by an SSL certificate by user name and password authentication. After login authentication, the server also verifies that the client has permission to perform an operation a query. This layer is not specific to MySQL technology.

Service Layer (MySQL SERVER)

The second layer is the service layer core of MySQL, MySQL's core service layer in this layer, query parsing, SQL execution plan analysis, SQL execution plan optimization, query cache. And across storage engine functions are implemented in this layer: stored procedures, triggers, views, and so on.

  1. Connection pool ( Connection Pool ) needs to be cached demand management, cushioning the user's connection, thread processing:
  2. Management services and tool components ( Services & Utilities ) : system management and control tools, such as backup and recovery, MySQL replication, clustering, etc.  
  3. SQL interfaces ( SQL Interface ) : accept the user's SQL commands, the user need to query and return the results of
  4. Query parser ( Parser ) : SQL command to the parser when the parser is parsing and validation (permissions, grammatical structure)
  5. The query optimizer ( Optimizer ) : SQL statement using the query optimizer in the query before query optimization
  6. Cache ( Caches ) : If you have a query cache hit results, the query can be taken directly to the query cache data 

Engine storage layer (Pluggable Storage Engines) 

It is responsible for storing the extracted data in MySQL. Query execution engine server communicates with the storage engine through the API, an interface screen through the differences between different storage engines. MySQL uses plug-in storage engine. MySQL storage engine provides a number of us, each storage engines have different characteristics. We can according to different business characteristics, choose the most suitable storage engine. If you are not satisfied with the performance of the storage engine, you can get the performance you want to achieve by modifying the source code.

Described below with reference to the respective storage engine.

Features:

  1. MySQL uses plug-in storage engine.
  2. Storage engine is not directed to the table for a library (a library of different tables can use different storage engine), the server communicates with the storage engine through the API, for shielding differences between different storage engines.
  3. No matter what kind of table storage engine in the data area will produce a corresponding one frm file (file description table structure definition)

File system layer (File System NTFS ext4 SAN NAS) 

File system data, log (redo, undo), an index, an error log, query logs, slow query, the detailed description below with reference to the physical file.

The layer such data is stored on the database file system, and complete the interaction with the storage engine. In MySQL indexing mechanism  has a file structure and contents mentioned Myisam InnoDB's.

MySQL storage engine 

The following outlines common MySQL storage engine

CSV Storage Engine

Data is stored in a CSV file.

Features:

  1. You can not define an index, the column must be defined as NOT NULL, you can not set an auto-increment is not applicable to large tables or data processing online
  2. CSV data storage separated by a comma, can directly edit the CSV file layout data, data security is low. After editing, to be effective to use flush table XXXthe command

Scenario:

  1. Quickly export data import
  2. Table directly converted into a form directly converted into CSV 

Archive storage engine

Storage for data compression protocol, the data is stored as ARZ file format.

Features:

  1. Insert and select only supports two modes of operation
  2. Allowing only increment ID column index
  3. Row-level locking
  4. It does not support transactions
  5. Data occupies less disk

Scenario:

  1. Logging system
  2. A large number of data acquisition devices

Memory storage engine 

Data are stored in memory, IO efficiency is much higher than other engines, restart the service data will be lost, only the default memory data table 16M, generally we do not use the Memory storage engine.

Features:

  1. Support hash indexes, B tree index, the default hash (look for the complexity of 0 (1))
  2. The field length is fixed length varchar (32) = char (32)
  3. Does not support large data storage type field such as blog, text
  4. Table-level locking

Scenario:

  1. Find high heat equivalent data
  2. In sorting, grouping, etc. In operation, when the amount of data is less than 16M (default size), a temporary table created by the query optimizer is Memory Type

MyISAM storage engine 

MySQL5.5 previous version of the default storage engine, more system tables still use the storage engine, the system temporary table will be used MyISAM storage engine. 

Features:

  1. select count (*) from table scan data without
  2. Data (MYD) and index (MYI) separately stored
  3. Table-level locking
  4. It does not support transactions

Scenario:

  1. In sorting, grouping, etc. In operation, when the amount exceeds a certain size, the temporary table created by the query optimizer is to MyISAM
  2. Reporting, data warehouse

InnoDB storage engine

MySQL 5.5 and later versions of the default storage engine. 

Features: 

  1. ACID transaction
  2. Row-level locking
  3. Aggregation index (primary key index) mode for data storage
  4. Support foreign key relationships to ensure data integrity

Compared

Feature MyISAM Memory InnoDB Archive NDB
B-tree indexes Yes Yes Yes No No
Backup/point-in-time recovery(note 1) Yes Yes Yes Yes Yes
Cluster database support No No No No Yes
Clustered indexes No No Yes No No
Compressed data Yes (note 2) No Yes Yes No
Data caches No N/A Yes No Yes
Encrypted data Yes (note 3) Yes (note 3) Yes (note 4) Yes (note 3) Yes (note 3)
Foreign key support No No Yes No Yes (note 5)
Full-text search indexes Yes No Yes (note 6) No No
Geospatial data type support Yes No Yes Yes Yes
Geospatial indexing support Yes No Yes (note 7) No No
Hash indexes No Yes No (note 8) No Yes
index caches Yes N/A Yes No Yes
Locking granularity Table Table Row Row Row
MVCC No No Yes No No
Replication support (note 1) Yes Limited (note 9) Yes Yes Yes
Storage limits 256TB RAM 64TB None 384EB
T-tree indexes No No No No Yes
Transactions No No Yes No Yes
Update statistics for data dictionary Yes Yes Yes Yes Yes

Physical file 

Physical file comprising: a log files, data files, configuration files, pid files, socket files

Log Files

MySQL 5.7 introduces the log file: https://dev.mysql.com/doc/refman/5.7/en/server-logs.html

Log Type Information Written to Log
Error log Problems encountered starting, running, or stopping mysqld
General query log Established client connections and statements received from clients
Binary log Statements that change data (also used for replication)
Relay log Data changes received from a replication master server
Slow query log Queries that took more than long_query_time seconds to execute
DDL log (metadata log) Metadata operations performed by DDL statements

Error Log (Error log)

MySQL process is running more serious warning and error messages, as well as detailed information every time you start and shut down MySQL MySQL error logging. MySQL error log is enabled by default.

By MySQL configuration file log-error = / var / log / mysqld.log configuration, modify the configuration of the error log.

You can view details of the error log through the following SQL:

show variables like '%log_err%';

通用查询日志(General query log) 

记录建立的客户端连接和执行的语句。

可以通过如下 SQL 查看当前的通用日志是否开启:

SHOW VARIABLES LIKE '%general%';

/*开启通用日志查询:*/
set global general_log = on;

/*关闭通用日志查询:*/
set global general_log = off;

二进制日志(Binary log) 

MySQL 的二进制日志(binary log)是一个二进制文件,主要用于记录修改数据或有可能引起数据变更的 MySQL 语句。二进制日志(binary log)中记录了对 MySQL 数据库执行更改的所有操作,并且记录了语句发生时间、执行时长、操作数据等其它额外信息,但是它不记录 SELECT、SHOW 等那些不修改数据的 SQL 语句。二进制日志(binary log)主要用于数据库恢复和主从复制,以及审计(audit)操作。

/*删除所有二进制文件:*/
reset master

/*删除部分二进制文件:*/
purge master logs

/*查看是否启用二进制日志:*/
show variables like '%log_bin%';

/*查看所有的二进制参数*/
show variables like '%binlog%';

/*查看文件的位置*/
show variables like '%datadir%';

/*查看当前服务器所有的二进制日志文件*/
show binary logs;
show master logs;

慢查询日志(Slow query log)

记录所有执行时间超过 long_query_time 秒的查询 SQL 或者没有使用索引的查询 SQL,默认情况下,MySQL 不开启慢查询日志,long_query_time 的默认值为10,即运行时间超过 10s 的语句是慢查询语句。

/*查看当前慢查询日志的开启情况:*/
show variables like '%query%';

  • slow_query_log:ON 表示开启慢查询日志,OFF 表示关闭慢查询日志
  • slow_query_log_file:记录慢查询日志的文件地址(默认为主机名.log)
  • long_query_time:指定了慢查询的阈值,单位是秒,即执行语句的时间若超过这个值则为慢查询语句
  • log_queries_not_using_indexes:ON 表示会记录所有没有利用索引来进行查询的语句,前提是 slow_query_log 的值也是 ON,否则,不会奏效,OFF 表示不会记录所有没有利用索引来进行查询的语句。 

配置文件

Linux下 MySQL 的配置文件是 my.cnf,一般会存放在 /etc/my.cnf,/etc/mysql/my.cnf,MySQL 所有的配置信息都存放在该文件中,后面会单独整理一篇 MySQL 的配置内容介绍。

数据文件 

获取硬盘中数据存储的地址:

SHOW VARIABLES LIKE 'datadir';

根目录文件内容:

engine 数据库文件内容:

db.opt 文件

该文件记录这个库的默认使用的字符集和校验规,文件存放在所属数据库的目录下。

FRM 文件

不论使用什么存储引擎,每一张表都会有一个以表名命名的 .frm 文件,与表相关的元数据(meta)信息都存放在此文件中,包括表结构的定义信息等,文件存放在所属数据库的目录下。

MYD 文件 

MyISAM 存储引擎专用,存放 MyISAM 表的数据(data)。每一张 MyISAM 表都会有一个 .MYD 文件,文件存放在所属数据库的目录下。

MYI 文件 

也是 MyISAM 存储引擎专用,存放 MyISAM 表的索引相关信息。每一张 MyISAM  表对应一个 .MYI 文件,文件存放在所属数据库的目录下。

IBD 文件和 IBDATA 文件 

存放 InnoDB 的数据文件(包括索引)。InnoDB 存储引擎有两种表空间方式:独享表空间和共享表空间。

独享表空间:使用 .ibd 文件来存放数据,且每一张 InnoDB 表对应一个 .ibd 文件,文件存放在所属数据库的目录下。

共享表空间:使用 .ibdata 文件,所有表共同使用一个(或多个,自行配置).ibdata 文件。

ibdata1 文件 

系统表空间(数据文件)undo 段,文件存放在 datadir 目录下。 

ib_logfile0、ib_logfile1 文件 

redlog 文件,文件存放在 datadir 目录下。

pid 文件 

pid 文件是 mysqld 应用程序在 Unix/Linux 环境下的一个进程文件,和许多其他 Unix/Linux 服务端程序一样,它存放着自己的进程 id。

socket 文件

socket 文件也是在 Unix/Linux 环境下才有的,用户在 Unix/Linux 环境下客户端连接可以不通过 TCP/IP 网络而直接使用 Unix Socket 来连接 MySQL。

参考

MySQL体系结构

【MySQL】漫谈MySQL体系结构

Chapter 15 Alternative Storage Engines

MySQL5.7 四种日志文件

Guess you like

Origin www.cnblogs.com/yinjw/p/11831702.html