Tens of millions of lines of csv large files can be quickly imported into mysql

I recently tried to import csv files, because Taicai used ordinary graphics windows to import before, but the speed was very slow, so I made a summary:

Currently I have two methods here:

1. Window mysql workbench import (scope of application: no database, the csv file to be imported is relatively small, applicable to no more than 10,000 lines) [Slow]

2. Imperative import      (scope of application: understand database commands, import millions of rows of big data) [fast]

1. Graphical

Graphical interface, find the start in the lower left corner of the computer, and find the visualization software wokbench. What is missing is that I did not download its visualization bundled program when I installed mysql, so I can solve it on Baidu.

 After running the program, enter root

 Right click on the blank space in the figure below, Create schema to create a new database

 Then find the left corner of the newly created database, click to expand, right-click on the Table, and select Table Data Import Wizard.

 Select the csv file path and click next

 Then the next page thinks that we have not created a table, so its default option is to create a table for us, you can click next directly

 Then click next again, he will help us choose the type of column, you can also choose yourself, the preview is below

 Continue to click next, and then click next to complete the work. You will find that there are already tables under the database table you created. Remember to manually refresh the list on the left before the table will appear.

 When the blogger was importing, there was too much data, and the import was too slow, so he used imperative import.

2. Imperative import

It only took 13 seconds to test 1 million rows of data. First, find mysql 8.0 Command Line Client in the Windows start menu bar

 Clicking to enter will allow you to enter the password. If you enter the wrong password, it will crash directly. This is what I did. In the end, I had to uninstall and reinstall mysql

Then start typing the command

 first enter

show variables like '%secure%';

Then press Enter, if secure_file_priv=' ' appears, congratulations, you can go directly to the next step,

If it is not empty, for example, some paths are in C:/Program Data/MySQL Server 8.0/Uploads/ Note that the Program Data here is a hidden folder, you need to click the view hidden folder above to see it . You need to modify the my.ini file in the mysql installation directory. The location is shown in the figure below. You can create and modify it on the desktop, and then drag and drop it.

Note: If the my.ini file is not found, there is a high probability that this file is in some of the paths entered above (the orange font refers to something), and you can find the location along some paths . This is mine,

The content of the my.ini file is as follows:

secure_file_priv=''

 Next, restart mysql.

Go back to the command window just now

show variables like 'local_infile';

The result is off and needs to be changed to on.

set global local_infile=on;

Then you need to face your csv file, create a new database in the database, and then create a new table in it by right-clicking the mouse on Table. I personally recommend the above graphical method to let mysql help you identify the type of each column and then create a table for comparison. , otherwise the import will always report an error later. Maybe I don't understand types well enough.

The next step is to start the import command. Here is a suggestion. It is recommended that the csv file be placed in the folder with some orange paths above, which can reduce some unnecessary troubles.

show databases;

view all databases

use 某某数据库;

Select the database, and replace it with your own csv file address below

load data infile 'C:/ProgramData/MySQL/MySQL Server 8.0/Uploads/1.csv' #换成自己地址
into table ship_info #换成你自己表名
fields terminated by','optionally enclosed by '"' escaped by '"'
lines terminated by '\r\n';

Press Enter and you're done, you can go back and refresh to see if there is a table. The success graph is as follows.

Guess you like

Origin blog.csdn.net/qq_43644046/article/details/128934336