Mysql data transfer Redis

       Recently, if you want to add a redis to your graduation project, you must transfer the data in MySQL to redis. So here comes the question, how to transfer data from a relational database to a non-relational database? Then Baidu MySQL data was migrated to redis, and as expected, each blog maintained a high degree of unity. Looking at the few blogs, almost all of them were reprinted, and there were very few original ones. Of course, I am just a novice, and I have been groping for a long time before I succeeded.

       If the students who have passed Baidu are familiar with the events_all_time table, haha. I don’t want to say more, the original text is okay, not that it’s okay, but the thinking is okay, the grammar is okay, but after the actual operation, an error will be reported, and the reason for this error is unknown. The solution I saw on Stack Overflow does not use redisprotocol. For me, a novice, there are some places in the article that are not explained. It seems a bit strenuous, and I don't understand why it is written like this. Then my article is to explain the redisprotocol first, then talk about the method of converting mysql to redis commands, and finally talk about the format of the multiple data in mysql and how to save them to redis. (Here is the event_all_time 2333 that I don’t know how many times it has been transferred)

CREATE TABLE events_all_time (
  id int(11) unsigned NOT NULL AUTO_INCREMENT,
  action varchar(255) NOT NULL,
  count int(11) NOT NULL DEFAULT 0,
  PRIMARY KEY (id),
  UNIQUE KEY uniq_action (action)
);

       text

       The method of migrating redis from mysql on Baidu, students who don't understand, combined with the explanation on the official website https://redis.io/topics/mass-insert , it is better to eat.

       It is not good to use redis ordinary client to insert a large amount of data, so the official recommended method is to generate a text file that conforms to the redis protocol, and call it uniformly with redis. The text file can write redis command or redis protocol. The protocol format is as follows

*<args><cr><lf>
$<len><cr><lf>
<arg0><cr><lf>
<arg1><cr><lf>
...
<argN><cr><lf>

        <cr> corresponds to '\r' (ASCII code 13), <lf> corresponds to '\n' (ASCII code 10). Then the meaning of the * and $ symbols is. Add a number after the * sign to indicate how many parameters the entire command has in total, including the command itself. Add a number after $ to indicate the corresponding parameter and how many bytes there are.

       Give a chestnut    SET key1 value1

The protocol converted to redis is *3\r\n$3\r\nSET\r\n$4\r\nkey1\r\n$6\r\nvalue1\r\n

*3-->There are set, key1, value1 three

$3-->The following SET has three bytes

$4-->The following key1 has 4 bytes

$6--> There are 6 bytes in the back value1

Isn't it very simple^_^!!!!!

Then look at the events_to_redis.sql that has been transferred many times. After understanding the redis protocol, look at this code, and it will be clear at a glance. The author's intention is to splice the values ​​queried in the database into corresponding commands.

SELECT CONCAT(
  "*4\r\n",
  '$', LENGTH(redis_cmd), '\r\n',
  redis_cmd, '\r\n',
  '$', LENGTH(redis_key), '\r\n',
  redis_key, '\r\n',
  '$', LENGTH(hkey), '\r\n',
  hkey, '\r\n',
  '$', LENGTH(hval), '\r\n',
  hval, '\r'
)
FROM (
  SELECT
  'HSET' as redis_cmd,
  'events_all_time' AS redis_key,
  action AS hkey,
  count AS hval
  FROM events_all_time
) AS t

final use

mysql -h 'ip address' -u'username' -p'password' 'database' --skip-column-names --raw < events_to_redis.sql | redis-cli --pipe

This command means that after mysql login, use the database 'database', --skip-column-names (make mysql output not contain column names) --raw (make mysql not convert newline characters in field values), and then Run events_to_redis.sql, and use the pipeline to pass it into redis-cli and run it.

There is a problem here, that is, the method of *3\r\n$3\r\nSET\r\n$4\r\nkey1\r\n$6\r\nvalue1\r\n is not acceptable. always report an error 

ERR Protocol error: expected '$', got ' '

I don't know the specific reason. If someone solves it, I hope you can comment and let me, a novice, learn it too.

solution

My final solution is actually to modify the sql statement on the basis of the previous understanding, so that the final output format of mysql is the normal SET key value format, and then run it.

SELECT 
  CONCAT(
    redis_cmd,' ',
    redis_key,
    idval,' ',
    ACTION,' ',
    actionval,' ',
    COUNT,' ',
    countval,' '
  ) 
FROM
  (SELECT 
    'HSET' AS redis_cmd,
    'events_all_time' AS redis_key,
    id AS idval,
    'action' AS ACTION,
    ACTION AS actionval,
    'count' AS COUNT,
    COUNT AS countval
  FROM
    events_all_time) AS t

All in all, it doesn't matter if you use mysql, java, python, C/C++, or even notepad. As long as the output format is redis protocol or redis command, this task can be completed by passing it into redis-cli.

Mysql multiple data stored in redis

Then for multiple pieces of data in mysql, I use the hmset command of redis.

HMSET key  field  value  [field  value ..]

The key is the table name + id, so that the data corresponding to one data table can be obtained by simple processing when querying. hmset table name+id column name value column name value...

In fact, it is also possible to use HSET. The key is the table name + id, and then a piece of data can be stored in String, and the data can also be obtained through simple processing.