(MySQL experience) The number of rows in a MySQL table should be less than 20 million

As a back-end developer, have you often heard that it is best not to exceed 2000W for a single MySQL table? If a single table exceeds 2000W , data migration must be considered. If the table data reaches 2000W , the query speed will become extremely slow.

1. Table creation operation

Create a table

CREATE TABLE person(
id int NOT NULL AUTO_INCREMENT PRIMARY KEY comment '主键',
person_id tinyint not null comment '用户id',
person_name VARCHAR(200) comment '用户名称',
gmt_create datetime comment '创建时间',
gmt_modified datetime comment '修改时间'
) comment '人员信息表';

Insert a piece of data

insert into person values(1,1,'user_1', NOW(), now());

Use mysqlpseudo column rownumto set the pseudo column starting point to 1

select (@i:=@i+1) as rownum, person_name from person, (select @i:=100) as init;
set @i=1;

Run the following SQL and execute it 20 times continuously, which means that 2 to the 20th power is approximately equal to 1 million data; execute it 23 times, which means 2 to the 23rd power is approximately equal to 800w. If you continue like this, you can insert tens of millions of test data. If you don’t want to turn over If you want to double the data, but want to increase it in small amounts, there is a trick, which is to add a where condition at the end of the SQL, such as id > a certain value to control the amount of added data.

insert into person(id, person_id, person_name, gmt_create, gmt_modified)
select @i:=@i+1,
left(rand()*10,10) as person_id,
concat('user_',@i%2048),
date_add(gmt_create,interval + @i*cast(rand()*100 as signed) SECOND),
date_add(date_add(gmt_modified,interval +@i*cast(rand()*100 as signed) SECOND), interval + cast(rand()*1000000 as signed) SECOND)
from person;

What needs to be noted here is that maybe when you execute nearly 800w or 1000w of data, you will get an error: The total number of locks exceeds the lock table size. This is because your temporary table memory is not large enough and you only need to expand it. Just set the parameters at once.

SET GLOBAL tmp_table_size =512*1024*1024; （512M）
SET global innodb_buffer_pool_size= 1*1024*1024*1024 (1G);

Let’s first look at a set of test data. This set of data is in the version of mysql8.0, and it is on my local machine. Since this machine also runs various tools such as idea and browser, it is not the machine configuration or the use of Database configuration, so test data is for reference only.

Seeing this set of data seems to really correspond to the title. When the data reaches 20 million , the query time increases sharply.

2. What is the limit on the number of single tables?

First, let’s think about the maximum number of rows in a single database table?

CREATE TABLE person(
id int(10) NOT NULL AUTO_INCREMENT PRIMARY KEY comment '主键',
person_id tinyint not null comment '用户id',
person_name VARCHAR(200) comment '用户名称',
gmt_create datetime comment '创建时间',
gmt_modified datetime comment '修改时间'
) comment '人员信息表';

Look at the table creation SQL above. ID is the primary key and is unique in itself. That is to say, the size of the primary key can limit the upper limit of the table. If the primary key declares an int size, which is 32 bits, then 2^32-1 ~~21 is supported. Billion; if it is bigint, then it is 2^62-1? (36893488147419103232), it’s hard to imagine how big this is. Generally, the database may be full before reaching this limit! !

Someone has calculated that if you select unsigned bigint for the auto-increment field when creating a table, then the maximum auto-increment value is 18446744073709551615. At the rate of adding one record per second, when will it be used up?

3. Table space

Now let’s take a look at the structure of the index. By the way, the content we will talk about below is all based on the Innodb engine. Everyone knows that Innodb’s index uses B+ trees internally.

This table data is stored similarly on the hard disk. It is actually placed in a file called person.ibd (innodb data), also called a table space; although in the data table, they appear to be one after another. , but in fact it is divided into many small data pages in the file, and each one is 16K.

It's probably like the following. Of course, this is just an abstraction. There are many concepts such as segments, areas, groups, etc. in the table space, but we need to jump out and see. For what B+ tree is, you can refer to another article.

4. Summary

1. MySQL table data is stored in the form of pages, and pages are not necessarily continuous on the disk.
2. The page space is 16K. Not all the space is used to store data. There will be some fixed information, such as page header, page footer, page number, check code, etc.
3. In the B+ tree, the data structures of leaf nodes and non-leaf nodes are the same. The difference is that leaf nodes store actual row data, while non-leaf nodes store primary keys and page numbers.
4. The index structure will not affect the maximum number of rows in a single table, and 2kw is only a recommended value. Exceeding this value may result in a higher B+ tree level, affecting query performance.