Author: Gao Peng (screen name eight strange), "in-depth understanding of the principles of MySQL master From 32" series author.
Series link: https://www.jianshu.com/nb/43148932
Version: 5.7.29
1. Questions raised
Under normal circumstances, we will use the lsof command to view the use of temporary files currently used by MySQL. This is because these temporary files cannot be displayed using the ls command. In 5.7, the method of creation is to use Liunx api mkstemp to create this file. It is process-specific. After returning the file descriptor, it will be deleted using api unlink.
The source code is mainly this section, which is contained in the create_temp_file function.
org_file=mkstemp(to);
if (mode & O_TEMPORARY)
(void) my_delete(to, MYF(MY_WME));
Some of the following common functions in 5.7 will use this kind of temporary files, such as:
Execution plan filesort file name: beginning with MY
lsof|grep delete
如:/tmp/MYdRH1GW (deleted)
Big transaction binary log cache file name: ML beginning
lsof|grep delete
如:/tmp/MLq9INFu (deleted)
Online DDL involves sorting such as add key, which usually starts with ib
alter table testsort add key(id);
lsof|grep delete
如:
/tmp/ibCxlYQg (deleted)
/tmp/ib51nvZ1 (deleted)
设置 innodb_tmpdir可以将这类文件放到指定的目录
But when we check this kind of file when a MySQLD process just started, we will find many such files, as follows:
[root@mgr2 ~]# lsof|grep delete|grep mysql|wc -l
215
[root@mgr2 ~]#
The size of these files are basically 0, and they all start with ib. A friend once asked me this question. Here is a detailed explanation.
2. Types of temporary files after MySQLD initialization
In fact, in 5.7 such files are created by the function create_temp_file, then we can put a breakpoint on it for analysis when MySQLD starts.
The following files are temporary files beginning with ib.
srv_monitor_file (srv0start.cc:2022): This is the file we usually use when outputting the show engine innodb status command. The main function is srv_printf_innodb_monitor. Stack pin:
srv_dict_tmpfile (srv0start.cc:2032): Mainly related to the output of foreign key creation information.
srv_misc_tmpfile (srv0start.cc:2041): Mainly related to foreign key throwing error output.
lock_latest_err_file (lock0lock.cc:465): Mainly related to deadlock error output.
dict_foreign_err_file (dict0dict.cc:1245): mainly related to foreign key throwing error output.
We can find that in fact, these files will not occupy too much. In fact, these 5 files have been created after initialization.
Three, why the default output of lsof is much more
Obviously we only have 5 files starting with ib above, so what do we see when we lsof (without any parameters)?
The following is the number I see when I initialize it:
[root@mgr2 support-files]# lsof|grep delete|grep mysql|wc -l
215
[root@mgr2 support-files]# lsof|grep delete|grep mysql
mysqld 12916 mysql 10u REG 8,5 9765 17109405 /tmp/ibpuzts2 (deleted)
mysqld 12916 mysql 11u REG 8,5 0 23137683 /tmp/ibA4q9iX (deleted)
mysqld 12916 mysql 12u REG 8,5 0 23137684 /tmp/ibRdXEl4 (deleted)
mysqld 12916 mysql 13u REG 8,5 0 23137685 /tmp/ibe6hfHJ (deleted)
mysqld 12916 mysql 19u REG 8,5 0 23137686 /tmp/ibfb7rps (deleted)
mysqld 12916 12920 mysql 10u REG 8,5 9765 17109405 /tmp/ibpuzts2 (deleted)
mysqld 12916 12920 mysql 11u REG 8,5 0 23137683 /tmp/ibA4q9iX (deleted)
mysqld 12916 12920 mysql 12u REG 8,5 0 23137684 /tmp/ibRdXEl4 (deleted)
mysqld 12916 12920 mysql 13u REG 8,5 0 23137685 /tmp/ibe6hfHJ (deleted)
mysqld 12916 12920 mysql 19u REG 8,5 0 23137686 /tmp/ibfb7rps (deleted)
mysqld 12916 12944 mysql 10u REG 8,5 9765 17109405 /tmp/ibpuzts2 (deleted)
mysqld 12916 12944 mysql 11u REG 8,5 0 23137683 /tmp/ibA4q9iX (deleted)
mysqld 12916 12944 mysql 12u REG 8,5 0 23137684 /tmp/ibRdXEl4 (deleted)
mysqld 12916 12944 mysql 13u REG 8,5 0 23137685 /tmp/ibe6hfHJ (deleted)
mysqld 12916 12944 mysql 19u REG 8,5 0 23137686 /tmp/ibfb7rps (deleted)
mysqld 12916 12945 mysql 10u REG 8,5 9765 17109405 /tmp/ibpuzts2 (deleted)
mysqld 12916 12945 mysql 11u REG 8,5 0 23137683 /tmp/ibA4q9iX (deleted)
mysqld 12916 12945 mysql 12u REG 8,5 0 23137684 /tmp/ibRdXEl4 (deleted)
此处省略....
We mainly focus on the following columns:
PID：is the Process IDentification number of the process.
TID：is the task (thread) IDentification number
FD：is the File Descriptor number of the file
SIZE/OFF：is the size of the file or the file offset in bytes
NAME: is the name of the mount point and file system on which the file resides Refer to man for more details.
From the output we can find
The PIDs are all 12916. This is the PID of mysqld, but their TIDs are different. These are of course the cause of multithreading.
FD and NAME are actually repeated regularly, in fact there are only 5 values. This represents the 5 temporary files we mentioned above
SIZE is relatively small.
Therefore, we can think that although the output is up to 215 lines, in fact the entire MySQLD process only opens 5 files, which is the 5 temporary files we mentioned earlier, because the file descriptor (FD) is shared between threads , Every thread lsof will display these 5 temporary files when viewing it. If our MySQL has a lot of sessions, then there will be a lot of threads, so the number of temporary files seen by lsof is the amount of thread data*5. Therefore, when checking the space usage of temporary files, do not be surprised by excessive output. We only need to output the temporary files of the MySQLD process. The output is as follows:
[root@mgr2 proc]# lsof|grep delete|grep mysql |awk '{if ($3=="mysql") print}'
mysqld 12916 mysql 10u REG 8,5 9765 17109405 /tmp/ibpuzts2 (deleted)
mysqld 12916 mysql 11u REG 8,5 0 23137683 /tmp/ibA4q9iX (deleted)
mysqld 12916 mysql 12u REG 8,5 0 23137684 /tmp/ibRdXEl4 (deleted)
mysqld 12916 mysql 13u REG 8,5 0 23137685 /tmp/ibe6hfHJ (deleted)
mysqld 12916 mysql 19u REG 8,5 0 23137686 /tmp/ibfb7rps (deleted)
或者加上lsof加上参数只看进程的临时文件，而不去查看子线程的临时文件
[root@mgr2 ~]#lsof -p 12916|grep delete
mysqld 12916 mysql 10u REG 8,5 9765 17109405 /tmp/ibpuzts2 (deleted)
mysqld 12916 mysql 11u REG 8,5 0 23137683 /tmp/ibA4q9iX (deleted)
mysqld 12916 mysql 12u REG 8,5 0 23137684 /tmp/ibRdXEl4 (deleted)
mysqld 12916 mysql 13u REG 8,5 0 23137685 /tmp/ibe6hfHJ (deleted)
mysqld 12916 mysql 19u REG 8,5 0 23137686 /tmp/ibfb7rps (deleted)
Note that there are some changes in the names of these temporary files under MySQL8.0.21, and they may be created directly (O_TMPFILE flag) in this way under Linx as follows:
open(dirname_buf, O_RDWR | O_TMPFILE | O_CLOEXEC,
S_IRUSR | S_IWUSR)
我测试得到的名字如下：
[root@mgr4 ~]# lsof|grep a.out|grep delete
a.out 11703 root 3u REG 8,5 0 24237415 /tmp/#24237415 (deleted)
MySQLD启动后查看的临时文件名如下：
[root@mgr4 ~]# lsof -p 11488|grep delete
mysqld 11488 mysql 10u REG 8,5 0 24237413 /tmp/#24237413 (deleted)
mysqld 11488 mysql 11u REG 8,5 0 24237414 /tmp/#24237414 (deleted)
mysqld 11488 mysql 12u REG 8,5 0 24237416 /tmp/#24237416 (deleted)
mysqld 11488 mysql 19u REG 8,5 0 24237417 /tmp/#24237417 (deleted)
Although the temporary file name has changed, the basic purpose is similar to 5.7
Fourth, the file descriptor (FD) between threads is shared
This paragraph borrows Linux system programming and explains it as follows:
Five, test
We can write a simple multi-threaded code to test as follows (without making any wrong judgments):
#include<stdio.h>
#include<stdlib.h>
#include <unistd.h>
#include <string.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <pthread.h>
void* u_sleep(void* t)
{
char file_temp[]="ibXXXXXX";
int fd = mkstemp(file_temp);
unlink( file_temp);
int i = 0;
char* teststr="aasd";
printf("thead tmpfile: %s\n",file_temp);
for(i = 0;i<100000;i++)
{
write(fd,teststr,strlen(teststr));
}
sleep(100);
}
int main(void)
{
int ret = 0;
pthread_t tid[5];
int i = 0;
int fd = 0;
int tid_num = 0;
int tmp_file_num=5;
char file_temp[]="ibXXXXXX";
for(i=0;i<tmp_file_num;i++)
{
strcpy(file_temp,"ibXXXXXX");
mkstemp(file_temp);
printf("%s\n",file_temp);
unlink( file_temp);
}
for (i = 0;i<sizeof(tid)/sizeof(pthread_t);i++)
{
pthread_create(tid+tid_num,NULL,u_sleep,NULL);
tid_num++;
}
for(i = 0;i<sizeof(tid)/sizeof(pthread_t);i++)
{
ret = pthread_join( *(tid+i) , NULL );
}
}
This code actually simulates the establishment of 5 temporary files through mkstemp and unlink in the main process and also starts with ib (simulating MySQL temporary files), and then establishes 5 threads, and each thread creates a temporary file. , Write something in, and then the thread sleeps for a while, the main thread will wait for the thread to reclaim resources, so you can have enough time to observe.
In this way, the observation is whether the file descriptor (FD) is shared between threads. It is expected that the output we can observe is 60 (1 main control thread, 5 child threads, the main control thread establishes 5 temporary files and 6 threads share That is 30, and then each sub-thread creates 1 temporary file and 5 temporary files, and 6 threads share 30 again) The following output:
[root@mgr2 ~]# lsof|grep delete|grep a.out|wc -l
60
[root@mgr2 ~]# lsof|grep delete|grep a.out|awk '{if ($3=="root") print}'
a.out 13460 root 3u REG 8,5 0 43691141 /root/ibpotqYK (deleted)
a.out 13460 root 4u REG 8,5 0 43691142 /root/ibxlgopk (deleted)
a.out 13460 root 5u REG 8,5 0 43691144 /root/ibHOmmQT (deleted)
a.out 13460 root 6u REG 8,5 0 43691145 /root/ibvmykht (deleted)
a.out 13460 root 7u REG 8,5 0 43691146 /root/ib3TViI2 (deleted)
a.out 13460 root 8u REG 8,5 400000 43691147 /root/ibEn9hAb (deleted)
a.out 13460 root 9u REG 8,5 400000 43691148 /root/ibkexi9B (deleted)
a.out 13460 root 10u REG 8,5 400000 43691149 /root/ibJJZi1K (deleted)
a.out 13460 root 11u REG 8,5 400000 43691150 /root/ibRRxKtk (deleted)
a.out 13460 root 12u REG 8,5 400000 43691151 /root/ibWuKsWT (deleted)
[root@mgr2 ~]# lsof|grep delete|grep a.out|more
a.out 13460 root 3u REG 8,5 0 43691141 /root/ibpotqYK (deleted)
a.out 13460 root 4u REG 8,5 0 43691142 /root/ibxlgopk (deleted)
a.out 13460 root 5u REG 8,5 0 43691144 /root/ibHOmmQT (deleted)
a.out 13460 root 6u REG 8,5 0 43691145 /root/ibvmykht (deleted)
a.out 13460 root 7u REG 8,5 0 43691146 /root/ib3TViI2 (deleted)
a.out 13460 root 8u REG 8,5 400000 43691147 /root/ibEn9hAb (deleted)
a.out 13460 root 9u REG 8,5 400000 43691148 /root/ibkexi9B (deleted)
a.out 13460 root 10u REG 8,5 400000 43691149 /root/ibJJZi1K (deleted)
a.out 13460 root 11u REG 8,5 400000 43691150 /root/ibRRxKtk (deleted)
a.out 13460 root 12u REG 8,5 400000 43691151 /root/ibWuKsWT (deleted)
a.out 13460 13461 root 3u REG 8,5 0 43691141 /root/ibpotqYK (deleted)
a.out 13460 13461 root 4u REG 8,5 0 43691142 /root/ibxlgopk (deleted)
a.out 13460 13461 root 5u REG 8,5 0 43691144 /root/ibHOmmQT (deleted)
a.out 13460 13461 root 6u REG 8,5 0 43691145 /root/ibvmykht (deleted)
a.out 13460 13461 root 7u REG 8,5 0 43691146 /root/ib3TViI2 (deleted)
a.out 13460 13461 root 8u REG 8,5 400000 43691147 /root/ibEn9hAb (deleted)
a.out 13460 13461 root 9u REG 8,5 400000 43691148 /root/ibkexi9B (deleted)
a.out 13460 13461 root 10u REG 8,5 400000 43691149 /root/ibJJZi1K (deleted)
a.out 13460 13461 root 11u REG 8,5 400000 43691150 /root/ibRRxKtk (deleted)
a.out 13460 13461 root 12u REG 8,5 400000 43691151 /root/ibWuKsWT (deleted)
省略很多.....
So our statement is proved.
Scan code to add author WeChat
The full text is over.
Enjoy MySQL :)
Teacher Ye's "MySQL Core Optimization" class has been upgraded to MySQL 8.0, scan the code to start the journey of MySQL 8.0 practice