LOAD DATA INFILE reading CSV data in ten million to mysql

Work requirements

Build a relationship between the relational model and textbooks in the movies(title,year,length,movietype,studioname,producerC)same relationship, custom name, insert 10 million records in this relationship.

Note: The relationship between moviesthe primary key is (title,year).

Requirements are as follows:

1. Completed in the shortest possible time;
2. Use only native SQL, will be allowed as an embedded SQL language, it is not allowed in other languages, such as C #, Python, etc. to complete;
3. Submit your detailed solutions and results.

In this paper, LOAD DATA INFILEit will contain ten million CSV file data into mysql.

achieve

In addition newcsv.py, other commands are cmd command.

CSV of pieces of data contains 10 million

Relationship with python generate the same movies ten million data structure comprising a CSV file.

newcsv.pyas follows:

The procedure takes about 18 seconds.

import csv
import time

# num_value条数据
num_value = 10000000

# 开始计时
time_start = time.time()

# 生成文件
with open(r'C:\ProgramData\MySQL\MySQL Server 8.0\Uploads\bigdata.csv', 'w', newline='') as f:
    f_csv = csv.writer(f)
    f_csv.writerow(['title', 'year', 'length', 'movieType', 'studioName', 'producerC'])

    for i in range(1, num_value+1):
        f_csv.writerow(['GoGoGo', i, 120, 'sicFic', 'MGM', 100])

# 结束计时
time_end = time.time()

# 输出耗费时间（秒）
print('Time Cost：', time_end - time_start)

Log in mysql

mysql -uroot -p123456

The above rootis my mysql user name, 123456is my mysql password.

Access to the database moviesdb

use moviesdb;

Create a relationship mymovies

Just copy the structure of the relationship between movies and does not copy data

CREATE TABLE mymovies LIKE movies;

CSV file stored in the database

LOAD DATA INFILE 'C:/ProgramData/MySQL/MySQL Server 8.0/Uploads/bigdata.csv' INTO TABLE mymovies FIELDS TERMINATED BY ',' ENCLOSED BY '"' LINES TERMINATED BY '\n' IGNORE 1 ROWS;

On my computer, the file is stored in mysql took 862.646 seconds, about 14 minutes, as shown below:

Reference links

https://www.cnblogs.com/freefei/p/7679991.html

https://blog.csdn.net/qq_22855325/article/details/76087138

https://blog.csdn.net/weixin_44595372/article/details/88723191

https://zhidao.baidu.com/question/185665472.html

https://www.cnblogs.com/zhangjpn/p/6231662.html

https://www.cnblogs.com/wangcp-2014/p/8038683.html

https://blog.csdn.net/gb4215287/article/details/82669785

---

Author: @ smelly salted fish

Please indicate the source: https://www.cnblogs.com/chouxianyu/

Welcome to discuss and share!

---