First, this section provides an overview
In the previous article I Finally, you left the question is how to copy data in the two tables. If you can control the number of scanning lines of the source table and locking range is very small, we simply use insert ... select statement can be realized.
Of course, in order to avoid adding to the source table read locks, more secure solution is to first write data to an external text file, and then write back to the target table. At this time, there are two commonly used methods. What follows, I'll take you out in detail what these two methods.
For illustrative purposes, I create a table db1.t, 1000 and insert rows of data, creating a table with the same structure db2.t.
create database db1;
use db1;
create table t(id int primary key, a int, b int, index(a))engine=innodb;
delimiter ;;
create procedure idata()
begin
declare i int;
set i=1;
while(i<=1000)do
insert into t values(i,i,i);
set i=i+1;
end while;
end;;
delimiter ;
call idata();
create database db2;
create table db2.t like db1.t
Suppose we want db1.t data inside a> 900-line lead out, inserted into the db2.t.
Two, mysqldump method
1, mysqldump main parameters of meaning
One method is to use the mysqldump command to export the data into a set of INSERT statement. You can use the following command
mysqldump -h$host -P$port -u$user --add-locks=0 --no-create-info --single-transaction --set-gtid-purged=OFF db1 t --where="a>900" --result-file=/client_tmp/t.sql
The output to a temporary file.
This command, the main parameters are as follows:
- 1. -single-transaction action that does not require additional table lock table db1.t when exporting data, but the use of START TRANSACTION WITH CONSISTENT SNAPSHOT method;
- 2. -add-locks is set to 0, it indicates the result output in the file, without increasing the "LOCK TABLES t WRITE;";
- 3. -no-create-info mean not need to export table structure;
- 4. -set-gtid-purged = off indicates that the output is not associated with the GTID;
- 5. -result-file specifies the path to the output file, where the file is generated to represent client on the client machine.
Through this command generates t.sql mysqldump file contains the INSERT statement shown in FIG.
Partial results output file 1 mysqldump FIG.
You can see, an INSERT statement which will include more value right, which is to use this document to follow when writing data, execution speed can be faster.
If you want to generate a file in an INSERT statement to insert only one row of data, you can in the implementation of mysqldump command, add parameters -skip-extended-insert.
Then, you can use the following this command, these INSERT statements into the db2 library to do it.
mysql -h127.0.0.1 -P13000 -uroot db2 -e "source /client_tmp/t.sql"
2, source is not a SQL statement, but a client command
It should be noted, source is not a SQL statement, but a client command. mysql client to execute the command process is like this:
1. Open the file, the default is a semicolon at the end to read a section of the SQL statement;
2. SQL statements sent to the server to perform.
In other words, not the server-side execution of this "source t.sql" statement, but an INSERT statement. So, whether it is in the slow query log (slow log), or in the binlog, these records are to be truly executed
INSERT statement.
Third, export CSV file
1. Export CSV File Considerations
Another approach is to directly export the results to a .csv file. MySQL provides the following syntax to query results
Export to the server local directory:
select * from db1.t where a>900 into outfile '/server_tmp/t.csv';
When we use this statement, you need to pay attention to the following points.
1. This statement will save the results on the server side. If you execute the command of the client and MySQL server not on the same machine, a temporary directory on the client machine is not going to generate t.csv file.
2. into outfile specified location generated (/ server_tmp /) file, the location parameters must be subject to
limitation of secure_file_priv. Alternatively secure_file_priv parameter values and action are:
-
- If you set to empty, and not to limit the location of the generated file, which is disposed unsafe;
- If set to a string representation of the path, it is required only on the files generated specified directory, or a subdirectory;
- If set to NULL, it means prohibits execution of select on this MySQL instance ... into outfile operation
3. This command will not help you to overwrite the file, so you need to make sure /server_tmp/t.csv this file does not exist, or when the statement is executed will be because of the presence of the same name files and error.
4. This command generates a text file, a data line of a corresponding row on the principle of a text file. However, if the field contains line breaks in the text line break will be generated. However similar line breaks, tabs such symbols, the front
face will keep "\" is the escape character, so that between fields can be separated with the delimiter area between data line.
After obtaining export the .csv file, you can then import the data into the target table db2.t in the load data with the following command.
load data infile '/server_tmp/t.csv' into table db2.t;
2, load data flow statement is executed
After obtaining export the .csv file, you can then import the data into the target table db2.t in the load data with the following command.
This process is performed as shown in the following statement.
1. Open the file /server_tmp/t.csv, by tabs (\ t) as a delimiter between the fields, a newline (\ n) as the separator between the recording, to read the data;
2. Start transaction .
3. Analyzing the number of fields in each row is the same table db2.t:
-
- If not the same, the direct error, the transaction is rolled back;
- If yes, configured in a row, InnoDB engine interface calls, written to the table.
4. Repeat step 3 until the entire file is read into /server_tmp/t.csv completed, the transaction is committed.
After 3, if binlog_format = statement, this load statement recorded in the binlog, how in the standby database replay it?
You may have a doubt, in the future if binlog_format = statement, this load statement recorded in the binlog, how in the standby database replay it?
Because /server_tmp/t.csv files are only stored on the host computer where the main library, if only to put the original written statement binlog, the library at the time of preparation of the implementation, not the document prepared by the library on the local machine, it will lead to stop step standby synchronization
4, the full implementation of the process of load data
Therefore, the complete process of the implementation of this statement, in fact, is below.
1. After completion of the implementation of the main library, the contents of the file written directly /server_tmp/t.csv binlog file.
2. Write the statement to load data local infile binlog file '/ tmp / SQL_LOAD_MB-1-0'INTO TABLE `db2`.`t`.
3. Place the binlog log reached by the library.
4. apply thread library equipment in the execution of this transaction logs:
. SUMMARY A first binlog in t.csv file read out, written to a temporary local directory / tmp / SQL_LOAD_MB-1-0 in;
B then execute the statement load data, to db2.t table prepared with insertion library. the same data in the main database.
Execution process shown in Figure 2:
2 load data synchronization process of FIG.
Note, load data here prepared statement executed inside the library, more than a "local". It means "Customers will execute this command ends of the machine where the local file / tmp / content SQL_LOAD_MB-1-0, and loaded into the target table db2.t in."
5, load data command in two ways
1. Do not add "local", is read a file server, the file must be in secure_file_priv specified directory or subdirectories;
2. add "local", read a client's file, as long as the client has access to mysql file permissions can be. At this time, MySQL client will first pass a local file server, and then execute the above-mentioned load data flow.
Also note that, select ... into outfile method does not generate the file table structure, so we also needed a separate derivative data obtained command table structure definition. -tab mysqldump provides a parameter table can be derived simultaneously junction
structure csv data file and definition file. Use of this command is as follows:
mysqldump -h$host -P$port -u$user ---single-transaction --set-gtid-purged=OFF db1 t --where="a>900" --tab=$secure_file_priv
This command will be under $ secure_file_priv defined directory, create a file saved to build t.sql table statement, creating a CSV data file is saved t.txt.
Fourth, the physical copying method
1, a direct copy .frm file db1.t table and .ibd file to the db2 directory, feasible?
Mysqldump foregoing we mentioned method and a method of deriving the CSV file are logical approach guide data, the data is read out from the table db1.t, the generation of text, and then written in the destination table db2.t.
You may ask, methods of physical data guide it? For example, copy the .frm file directly to the table and .ibd db1.t file to the db2 directory, feasible?
The answer is not acceptable.
Because, a InnoDB table, in addition to containing these two physical files, also need to be registered in the data dictionary. Both direct copy documents, since the data is not in the dictionary db2.t this table, the system will not accept them and recognize the
2, MySQL 5.6 Step physical copy of the table to achieve
However, MySQL 5.6 version introduced a method for transmitting available table space (transportable tablespace) can be derived by way of introduction + tablespace, functional physical copy of the table.
Suppose we now aim at db1 library, a copy of the same table with the table t r, the specific implementation steps are as follows:
1. Run the create table r like t, create an empty table with the same table structure;
2. Execute alter table r discard tablespace, this time r.ibd file is deleted;
3. Implementation flush table t for export, this time db1 directory will generate a next file t.cfg;
4 cp t.cfg r.cfg performed at db1 directory; cp t.ibd r.ibd; both commands (it should be noted that the two copies of the file obtained, MySQL process must have read and write access);
5. perform unlock tables, this time t.cfg file is deleted;
6. perform alter table r import tablespace, this r.ibd file as a new table space table r, since the data content and t.ibd this document are the same, so there will be a table and the table r in the same data t.
Thus, table data copy operation is completed. The execution process is as follows:
FIG physical copy table 3
3, the physical copying method Considerations
About this process copy of the table, there are several points Note:
1. After step 3 after executing flsuh table command, db1.t entire table is read-only, until after the implementation of unlocktables ordered the release read lock;
2. In performing import tablespace in order to allow the file table space id and consistent data dictionary, modify r.ibd table space id. And this table space id exists in each data page. Thus, if a
big file (such as TB level), each page of data need to be modified, so you will see the implementation of the import statement that takes some time. Of course, if compared to a method of introducing a logic, time-consuming Import statement is very short.
V. Summary
Today this article, I introduce you to three and a method to import data table of another table.
Let's compare the advantages and disadvantages of these three approaches.
1. The physical copy fastest way, especially for large copy sheet is the fastest way. If the occurrence of erroneous puncturing table, with a temporary database backup and recovery before mistakenly deleted, then copy sheet to the temporary repository library production, restoring
data the fastest way. However, this approach also has some limitations:
- The table must be a full copy of only some data can not be copied;
- You need to copy the data to the server can not be used in the scenarios users can not log database host;
- Due to the use of physical files by copying is implemented, the source and target tables are using the InnoDB engine
A method comprising using mysqldump INSERT statement file generation can be increased in conditions where the filter parameters to achieve export only part of the data. One deficiency of this approach is that you can not use join such a complex
where conditions wording.
3. select ... into outfile method is the most flexible , it supports all SQL writing. However, one disadvantage of this method is that you can only export the data in a table, and the table structure also requires an additional statement backed up separately.
The latter two are logical backup, it can be cross-engine.
Finally, I leave you with a thought to the bar.
We described earlier binlog_format = statement when, binlog command load data is recorded with the local. Since this command is sent to the standby database to perform, then executed when the standby database is performed locally, why
we need this local it? If written binlog commands without local, what problems will happen?
You can write your analysis in the comments section, I will end next article and you discuss this issue. Thank you for listening, you are welcome to send this share to more friends to read together.
Sixth, on the issue of time
I think the article title last left you, the answer has been done today, the body of this article. Articles on the comments section there are several very good message, I am here to share with you.
@huolang students asked a question: If sessionA get c = Record 5 lock is a write lock, then why can add sessionB and sessionC c = 5 read lock it?
This is because the next-key lock is to lock plus a gap, coupled with record lock. Plus a gap lock is successful, plus record locks will be blocked. If you have questions about this process, you can then go over the first 30 articles of relevant content.
@ Only a great student did an experiment to verify the primary key violation after, insert the statement added clearance lock effect. Examples rollback than that I mentioned in the last article in the body leads to a deadlock of more intuitive, reflects his very good reason for this knowledge
solutions and thinking, it is praise.
@roaming students verified in MySQL 8.0 version, we have been able to deal with a temporary table insert ... select statement written in the original table.
@ Comrade Yang's answer mentions several methods in this article comes to us of.