DBAplus community today
The following article comes from the development of Xia dream notes dream of Man
Foreword
With the operation of the system, the amount of data becomes more and more, simply will in MySQL, already can not meet the data storage requirements of the inquiry, this time we introduce a Redis as query caching layer, heat will save business data to Redis , extension of traditional relational database service capabilities, users get through the application directly from the Redis common data quickly, or use Redis to save active user sessions in interactive applications, can greatly reduce the load on back-end relational databases, upgrade user experience.
Shortcomings of the traditional command
Using conventional Redis Client command following defects in the large amount of data into the scene:
Since Redis model is a single-threaded, multi-threading is avoided although the thread switch time-consuming, sequential single execution command quickly, but in large quantities of data import scene, and the reception time sent by the server in response to the results of the command takes time taken will be amplified.
If you need to import data 1,000,000, that just command execution time, it takes 1,000,000 * (t1 + t2).
In addition to sending commands one by one, of course, Redis design will certainly consider this issue, so there pipelining pipeline mode.
But pipelining the command line is not, so we need to write a new handling code to receive the response quantities. But only very little support client-side code, such as php-redis extension does not support asynchronous.
pipelining pipeline mode, in fact, reduces the interaction time of the TCP connection, when a batch of commands is finished, send a one-time result.
The principle is to use its FIFO (First In First Out) queue to ensure that the order of the data.
Only a small portion of the client supports non-blocking I / O, not all clients are able to be an effective way to resolve the answer to maximize throughput.
For these reasons, the preferred method of introducing into the vast amounts of data is formed including Redis Redis protocol data format, the batch is transmitted in the past.
Data Import Redis warm-up
Command to import data using nc
nc is netcat shorthand, nc role are:
1) arbitrary TCP / UDP port listener, after increasing the parameter -l, NC as TCP or UDP server in listening mode designated port.
2) a port scan, nc being the client may initiate a TCP or UDP connection.
3) to transfer files between machines.
4) between the machine speed network.
Import data using the model pipe
However, the use nc listener is not a very reliable way to perform large-scale data import, because netcat do not really know when to transmit all the data, can not check for errors. In version 2.6 or higher in Redis, Redis -cli script supports a new model called pipe pipeline mode, this model is inserted in order to perform large-scale and design. Use pipeline mode command operates as follows:
From the above chart, you can see the results of the command return pipe, how many lines there are in command txt file and return the number is the number of replies, errors indicate the number of execution in which the wrong command.
Redis protocol learning
Protocol format is as follows:
*<参数数量> \r\n
$<参数 1 的字节数量> \r\n
<参数 1 的数据> \r\n
...
$<参数 N 的字节数量> \r\n
<参数 N 的数据> \r\nFor example: Insert a hash of the data type.
HSET id book1 book_description1Redis according to the agreement, a total of four parts, so beginning with * 4, the rest is explained as follows:
Note: HSET command itself is one of the parameters as the protocol to send.
Protocol data structure constructed:
*4\r\n$4\r\nHSET\r\n$2\r\nid\r\n$5\r\nbook1\r\n$17\r\nbook_description1\r\nFormatting about:
*4\r\n
$4\r\n
HSET\r\n
$2\r\n
idvvvv\r\n
$5\r\n
book1\r\n
$17\r\n
book_description1\r\nRESP agreement bulk
The communication protocol used by the client Redis called RESP (Redis serialization protocol) server and Redis.
redis-cli pipe nc command mode and needs as fast and solve the problem nc command does not know when to end the command.
Transmitting data at the same time, it will also respond to read, try to resolve.
Once more data streams to the input is not read, it will send a special 20-bit echo command, identifies the last command has been transmitted if the same data is matched to the response results, described in this batch send a success.
Using this technique, we do not need resolution protocol sent to the server to know how much we send command, you only need to parse the response.
When parsing the response, Redis will resolve the answer be a count, in the last number of sessions that can tell the user to the server to transmit commands a large number of insertion. We use the pipe is above the actual operation mode in response to the results.
The input data source to MySQL
In the above example, we txt to a text input data source, a data import mode pipe.
Based learning and understanding of these agreements, we only need to MySQL data in accordance with established protocol by pipe mode Redis can be imported.
Importing data from MySQL to one million Redis
First-made data
Due to environmental constraints, so there is no real data to achieve the introduction, then we have to use a stored procedure to create one million facts and figures. Stored using the following procedure:
DELIMITER $$
USE `cb_mon`$$
DROP PROCEDURE IF EXISTS `test_insert`$$
CREATE DEFINER=`root`@`%` PROCEDURE `test_insert`()
BEGIN
DECLARE i INT DEFAULT 1;
WHILE i<= 1000000
DO
INSERT INTO t_book(id,number,NAME,descrition)
VALUES (i, CONCAT("00000",i) , CONCAT('book',i)
, CONCAT('book_description',i));
SET i=i+1;
END WHILE ;
COMMIT;
END$$
DELIMITER ;Call a stored procedure:
CALL test_insert();View table data:
According to the agreement to construct the query statement.
According to the above Redis protocol, we use the following SQL construct Protocol Data:
SELECT
CONCAT(
"*4\r\n",
"$",
LENGTH(redis_cmd),
"\r\n",
redis_cmd,
"\r\n",
"$",
LENGTH(redis_key),
"\r\n",
redis_key,
"\r\n",
"$",
LENGTH(hkey),
"\r\n",
hkey,
"\r\n",
"$",
LENGTH(hval),
"\r\n",
hval,
"\r"
)
FROM
(SELECT
"HSET" AS redis_cmd,
id AS redis_key,
NAME AS hkey,
descrition AS hval
FROM
cb_mon.t_book
) AS t limit 1000000Redis.sql save the contents to a file.
Scripting using pipe model import Redis
Writing shell scripts. Since I am on the host through the docker and Redis installed MySQL, the following script for reference:
#!/bin/bash
starttime=`date +'%Y-%m-%d %H:%M:%S'`
docker exec -i 899fe01d4dbc mysql --default-character-set=utf8
--skip-column-names --raw < ./redis.sql
| docker exec -i 4c90ef506acd redis-cli --pipe
endtime=`date +'%Y-%m-%d %H:%M:%S'`
start_seconds=$(date --date="$starttime" +%s);
end_seconds=$(date --date="$endtime" +%s);
echo "脚本执行耗时："$((end_seconds-start_seconds))"s"Screenshot execution:
You can see one million data import Redis, took only seven seconds, efficiency is very high.
Precautions
If MySQL table is particularly large, consider batch import, or split the table, or it may occur during the import process:
lost connection to mysql server during querySince max_allowed_packed and timeout limit, the process of data query, may cause disconnection, so the amount of data in the data table particularly when the need to split the page table or introduced.
to sum up
This article focuses on the next MySQL one million data of the order, how to efficiently migrate to Redis go, and gradually achieve the goal of the course, summed up the following points:
1, Redis single thread of execution command, to avoid the thread switching time consumed, but in order of large data, transmits, in response to receiving the delay can not be ignored.
2, network scenarios nc command, and import data in the presence of disadvantages.
3, understanding and application of Redis RESP agreement.
4 Million Redis order of MySQL data quickly import case.
Author Shu Xia dream
Source Shu Xia dream of Development Notes (ID: xmdevnote)
dbaplus community are welcome to contribute technical personnel, Submission E-mail: [email protected]
