[MySQL] Import JSONL data into MySQL database

I have been doing some data processing work recently and need to import file data with the suffix ".jsonl" into the MySQL library. Since I haven't tried it before, I will record it as an experience this time.

First of all, you must build a database and a table (these will not be explained in detail), and then you can extract the contents of the jsonl file through the LOAD DATA INFILE command. Since this time we are using MySQL 5.7 or above (MySQL 8), we can directly use the @json user variable to tell MySQL to store the data in JSON format. As shown below:

LOAD DATA INFILE '<<jsonl_path>>'  
INTO TABLE <<target_table>>  
(@json)  
SET field1 = JSON_EXTRACT(@json, '$.field1'),  
    field2 = JSON_EXTRACT(@json, '$.field2'),  
    field3 = JSON_EXTRACT(@json, '$.field3'),
    ...;

However, the "secure-file-priv" parameter exception may occur during this process, as shown below:

Error Code: 1290. The MySQL server is running with the --secure-file-priv option so it cannot execute this statement

At this time, you can query the current parameter status through “show variables like ‘secure_file_priv’”

mysql> show variables like 'secure_file_priv';
+------------------+-------+
| Variable_name    | Value |
+------------------+-------+
| secure_file_priv | NULL  |
+------------------+-------+
1 row in set (0.00 sec)

If Value is NULL, it means that the import and export functions are prohibited. At this time, you need to configure the my.cnf file and specify an import path for it.

# 这里的路径是 Docker 镜像内部的路径
secure_file_priv='/var/lib/mysql/imports'

Then restart the MySQL service. Then re-execute “show variables like ‘secure_file_priv’” to see if the configuration takes effect.

mysql> show variables like 'secure_file_priv';
+------------------+-------------------------+
| Variable_name    | Value                   |
+------------------+-------------------------+
| secure_file_priv | /var/lib/mysql/imports/ |
+------------------+-------------------------+
1 row in set (0.00 sec)

After the configuration takes effect, remember to map the /var/lib/mysql/imports/ path to the host, or use docker cp to put the jsonl file into the container, otherwise a file not found error will be reported.
During the import process, the following may appear:

Error Code: 3141. Invalid JSON text in argument 1 to function json_extract: "Invalid encoding in string." at position 411.

This is caused by the presence of special characters in jsonl. What I encountered this time was the "\" problem. Since character escapes may occur in the data, special characters such as "\n", "\t", "\", etc. need to be converted first, otherwise they cannot be imported into the database. For example, in the jsonl file, replace all "\n" with "@" and all "\t" with "&", and then update them again after importing them into the database.

[MySQL] Import JSONL data into MySQL database

Guess you like