Adding MySQL data in a CSV file

One. Adding MySQL csv data

This problem is encountered few days before finishing the data when there is data CSVfiles (more than 200 million records), too slow through the python import to the database, then use MySQL the built-in command , more than 30 seconds will be able to complete the second three million of the amount of data import. LOAD DATA INFILE

LOAD DATA INFILE Command allows you to read a text file and then very quickly into the database.

Before you import the file, you need the following elements:

Create a database table the corresponding data.
CSV data file and database tables need to be consistent in the number of columns and data types.
Has written to the database file and insert permission of the account

Suppose we have a discounts table structure is as follows:

We use the [CREATE TABLE statement][1]command to create discounts the table:

CREATE TABLE discounts (
    id INT NOT NULL AUTO_INCREMENT,
    title VARCHAR(255) NOT NULL,
    expired_date DATE NOT NULL,
    amount DECIMAL(10 , 2 ) NULL,
    PRIMARY KEY (id)
);

Discounts.csv the first line of the file as the column name, the other three behavioral data.

The following command causes c:\tmp\discounts.csv the file into the discountsform.

LOAD DATA INFILE 'c:/tmp/discounts.csv' 
INTO TABLE discounts 
FIELDS TERMINATED BY ',' 
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
IGNORE 1 ROWS;

The threshold data file by a comma separated, reflected code , and the data surrounded by double quotation marks, the marked line break in the CSV be described. FIELD TERMINATED BY ',' ENCLOSED BY '" ‘
LINES TERMINATED BY '\n'

Because the first line of the file contains column names, therefore, should not import the data in the table, here IGNORE 1 ROWS ignore the first row to ensure that no import statement.

two. Import data conversion format

Sometimes format of the data format does not meet the target database table column. Simple cases, you can LOAD DATA INFILE set the set option to convert the data

Assume discount_2.csv file expiration time column mm/dd/yyyy format.

When importing data into a table discounts, we must str_to_date () function into a format date MySQL

LOAD DATA INFILE 'c:/tmp/discounts_2.csv'
INTO TABLE discounts
FIELDS TERMINATED BY ',' ENCLOSED BY '"'
LINES TERMINATED BY '\n'
IGNORE 1 ROWS
(title,@expired_date,amount)
SET expired_date = STR_TO_DATE(@expired_date, '%m/%d/%Y');

three. The client import data to a remote MySQL database

Use the command from the client (local computer) to a remote import database data is entirely feasible. LOAD DATA INFILEMySQL

When you use LOAD DATA INFILE the LOCAL read local files option, the client program, and then send it to MySQL server. Files will be uploaded into the respective temporary directory server, such as Windows in C:\windows\temp or linuxin /temp .
This folder can not be configured MySQL or occupied.

We take a look at the following example:

LOAD DATA LOCAL INFILE  'c:/tmp/discounts.csv'
INTO TABLE discounts
FIELDS TERMINATED BY ',' 
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
IGNORE 1 ROWS;

The only difference is that the command LOCAL options. If you need load a large CSVfile, you'll notice LOCAL the option, it will be slower than usual, because more time is wasted in the above data transmission.

When you use the LOCAL option, connect MySQL server account does not require file permissions to import the data.

Use LOAD DATA LOCAL import local files to a remote MySQLserver, you need to pay attention to some security issues , you must be aware of these issues to avoid potential security risks.

This article reprinted from: https://www.cnblogs.com/nju2014/p/5456632.html