Build redis sentinel and cluster

repeat

1. Concept

1. The default data is written to the memory, power data will be lost
2. redis is a memory database
3. When selenium operate the browser, the browser needs to pay attention to the release of resources, prevent memory leaks
4. redis persistent: To prevent loss of data, stored in document form
5. Redis is a free open source vmware development of KV type NoSQL caching products

redis properties

1. Redis is an open source (BSD license), the data structure stored in the memory system that can be used as a database, cache and messaging middleware
2. redis is c language, support for data persistence is key-value type of database.
3. Application buffer queue system
4. redis support data backup, that is, master-slave mode

Advantage

1. Redis has good properties, it can provide 100,000 / sec read and write
2. As a cache database, data in memory
3. Alternatively mysql under certain scenarios, such as a social networking app
4. Large-scale system, you can store session information, shopping cart order

2. redis installation

1. Installation

1. yum install, simple to configure source yum
2. Source compiler installation, you can specify the installation path, customized third-party expansion modules function
3. rpm installation, you need to manually resolve dependencies

2. Source Installation

1. Preparing the Environment

wget   http://download.redis.io/releases/redis-4.0.10.tar.gz
tar xzvf redis-4.0.10.tar.gz

• Resolve dependencies

yum install gcc patch libffi-devel python-devel  zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-devel openssl openssl-devel -y

2. The release makefile

• Tell gcc compiler, redis installation path
• If the default is no configure script, but there is Makefile , you can directly make && make install

make && make install

3. compiled

• You can use redis, server start redis

redis-server            # 直接执行
redis-cli               # 测试，ping回显PONG

• redis executable file

./redis-benchmark       # 用于进行redis性能测试的工具
./redis-check-dump      # 用于修复出问题的dump.rdb文件
./redis-cli             # redis的客户端
./redis-server          # redis的服务端
./redis-check-aof       # 用于修复出问题的AOF文件
./redis-sentinel        # 用于集群管理

• Cloud servers in the move

kill -9 pid
# 查看定时任务，并编辑**文件的定时任务
crontab -l
crontab -e
# 全局搜索病毒文件，并删除(注意恶意病毒，篡改了文件名)
find / -name 病毒文件
# 找到后 rm -rf 删除
lsattr filename.txt
# 去掉文件的锁,即可删除
chattr -a  -i  filename.txt

4. Specify the configuration file

• Start redis server security
  1. Change the start port
  2. Add redis password
  3. Safe Mode is turned redis
• The default configuration file, redis.conf

# 过滤出配置文件的有益信息(去掉注释和空白行)
grep -v '^#' redis.conf | grep -v '^$' > myredis.conf

5. Specify startup configuration

• test.py

import redis
# redis默认有 16 个库，0-15
conn = redis.StrictRedis(host='172.16.44.142', port=6379, db=0, password='')
conn.set('name', 'henry')

• myredis.conf

bind 172.16.44.142              # 绑定redis服务器地址
protected-mode yes              # 开启安全模式
port 6800                       # 指定端口
requirepass xxx                 # 设置密码
pidfile /var/run/redis_6379.pid # 进程id文件
loglevel notice                 # 日志等级
daemonize yes                   # 后台运行 

• start up

redis-server myredis.conf
# 测试
redis-cli -p 6800 -h 172.16.44.142
172.16.44.142:6800> ping
(error) NOAUTH Authentication required.
172.16.44.142:6800> AUTH test
OK

3. redis data

1. commonly used commands

keys *
type key                        # 查看key对应的value类型
expire queue seconds            # 设置过期剩余时间
eg. expire test 10              # 10s 后过期
ttl queue                       # 查看剩余时间
persist queue                   # 取消queue的过期时间
exists key                      # 判断key是否存在
del key                         # 删除key，可以删除多个
dbsize key                      # 当前库key的数量
flushdb                         # 清除redis的所有key
flushall                        # 清空所有数据库的所有 key

2. Data Types

• redis is an advanced key: value storage systems, which value supports five data types
  1. Strings (strings)
  2. Hashes (hashes)
  3. List (lists)
  4. Collection (sets)
  5. Ordered set (sorted sets)

1. string (strings)

# string 类型，通过set命令设置
set key value
set 'name' 'xixi'
# 获取指定的key
get 'name'
# key存在，则追加到该key原来值的末尾。
append 'name' 'haha'
# 批量设置和获取
mset 'name' 'henry' 'age' 18
mget 'name' 'age'
# 将已存在 key 的值设为value，并返回key的旧值(old value)
getset name echo
# 返回 key 中字符串值的子字符 value[start, end],包含end
getrange name 1 2
# 获取 name 值的长度
strlen name
# 自增1
incr 'prize'
# 自减1
decr 'prize'

2. hashes (hashes)

# 将哈希表 key 中的字段 field 的值设为 value
hset stu name 'henry' age '18' height '180' 
# 获取存储在哈希表中指定字段的值
hget stu name
# 如果给定字段已经存在且没有操作被执行
hsetnx key field value

# 同时将多个 field-value (域-值)对设置到哈希表 key 中
hmset key field1 value1 field2 value2 ...
hmset stu name echo age 19
# 获取所有给定字段的值
hmget key field1 field2 ...
hmget stu name height

# 删除一个或多个哈希表字段
hdel key field1 field2 ...
# 查看哈希表 key 中，指定的字段是否存在
hexists key field

# 获取所有哈希表中的字段
hkeys key
# 获取哈希表中所有值
hvals key
# 获取在哈希表中指定 key 的所有字段和值
hgetall key
# 获取哈希表中字段的数量
hlen key

3. list (lists)

# 双向队列
# 从list左边(右边)插入
lpush/rpush key value1 value2 ...
lpush/rpush test 1

# 索引查看list, [start, end]
lrange test start end   
# 通过索引设置列表元素的值
lset key index value
# 通过索引获取列表中的元素
lindex key index

# 移出并获取列表的第一个(最后一个)元素
lpop/rpop key
# 移出并获取列表的第一个(最后一个)元素，如果列表为空会阻塞直到等待超时或发现可弹出元素为止
blpop/brpop key1 [key2 ] timeout 

# 获取列表长度
llen key
# 对一个列表进行修剪(trim)，就是说，让列表只保留指定区间内的元素，不在指定区间之内的元素都将被删除
ltrim key start stop                        
# key 存在不处理，不存在添加
lpushx/rpushx key 

4. The set (sets)

• Redis set, is a set of unordered collection of elements in no particular order.

# set 集合类型
# 默认是无序的
sadd url 'http://www.baidu.com' 'http://www.jd.com'
# 查看所有的成员
smembers url
# 删除
srem url 'http://www.jd.com'
# 判断是否是集合的元素
sismember url 'http://www.jd.com'
# 列出 url - url2
sdiff url url2
# 返回集合的交集
sinter url url2
# 返回集合的并集
sunion url url2

4. redis publish and subscribe

# 发布者
publish python 'hello python'
publish python 'hello go'
# 订阅者1
subscribe linux python
# 正则模式订阅
psubscribe python*
# 订阅者2
subscribe go

5. endurance of

• Trigger mechanism
  1. Manually save command
  operations in # 200 10 200 seconds, the modified class over 10 2. triggering condition or the configuration save

1. Method 1: rdb

1. Trigger

1. Creates a compressed binary file, save to disk
2. Profiles
   • myredis_rdb.conf

dbfilename dbmp.conf
# 持久化文件保存位置
dir /data/6379
# 每900s只要有1个修改记录就保存一次
save 900 1
save 60 10000

# 启动
redis-server myredis_rdb.conf

• Manual trigger

save

2. snapshot-based persistence

• Snapshot-based persistence, faster, typically used as a backup, is copied from the master depends on persistence function rdb

2. Second way: aof

• AOF (append-only log file), the modified operation command class, appended to the log file
• AOF file command to all formats redis agreement to save the new command is appended to the end of the file.
• Advantages: maximum program to ensure that data is not lost
• Cons: Very large logging

1. aof profile

• aof_redis.json

daemonize yes
port 6379
logfile /data/6379/redis.log
dir /data/6379
dbfilename  dbmp.rdb
requirepass test
save 900 1
save 300 10
save 60  10000
appendonly yes
appendfsync always          # 总是修改类操作
            everysec        # 每秒保存
            no              # 依赖于系统自带的缓存大小机制

2. Based on the persistence of aof

• Can additionally recorded redis operation log file. Redis can ensure the greatest degree of data security, similar to the mysql binlog

3. rdb way conversion aof

1. Preparing the Environment

• Ensure redis version 2.2 or more
• In redis4.0 herein by config set command, to the service without restarting redis, switching from a RDB persistence AOF

# 版本>4.0
redis -v 
# 准备rdb配置文件
daemonize yes
port 6379
logfile /data/6379/redis.log
dir /data/6379
dbfilename  myredis.rdb
bind  127.0.0.1
requirepass test
save 900 1
save 300 10
save  60  10000
# 启动redis
redis-server redis.conf
# 临时生效
redis-cli
# 开启 aof 持久化方式
config set appendonly yes
# 关闭 rdb 持久化方式
config set save ''
# 永久生效，修改配置文件，/etc/redis.conf
# 此时RDB已经正确切换AOF，注意还得修改redis.conf添加AOF设置，不然重启后，通过config set的配置将丢失
appendonly yes

6. redis master copy from

1. redis multi-instance support

• A machine can run multiple databases simultaneously redis
• Preparing the environment: Run 3 redis database, to achieve a configuration from the master 2
• Main Library: 6379.conf
• From the library
  • 6380.conf, 6381.conf
  • sed 's/6379/6380/' 6379.conf > 6380.conf
• Open master-slave configuration function

1. master library

• redis.conf

daemonize yes
bind  127.0.0.1
port 6379
loglevel notice
logfile /data/6379/redis.log
dir /data/6379
pidfile /data/6379/redis.pid
dbfilename dump.rdb

2. slave library

• 6380.conf 、6381l.conf

# 6380和 6381配置主从
sed 's/6379/6380/g' redis.conf > 6380.conf
sed 's/6379/6381/g' redis.conf > 6381.conf
# 启动数据库
redis-server redis.conf
redis-server 6380.conf
redis-server 6381.conf
# 修改配置文件
echo 'slave of 127.0.0.1 6379' >> 6380.conf
echo 'slave of 127.0.0.1 6379' >> 6381.conf
# 查看数据库信息
redis-cli info replication
redis-cli -p 6380 slaveof 127.0.0.1 6379
redis-cli -p 6381 slaveof 127.0.0.1 6379
redis-cli info replication
# 主库用于插入数据、从库用于读取，实现读写分离

2. master failover

1. Manually switching the primary library 6380

# 1. 关闭从库身份
redis-cli -p 6380 slaveof no one
# 2. 设置其他从库的主库
redis-cli -p 6381 slaveof 127.0.0.1 6380
# 3. 修改配置文件中的主从关系
sed -i 's/slaveof 127.0.0.1 6379//g' 6380.conf
sed -i 's/slaveof 127.0.0.1 6379/slaveof 127.0.0.1 6380/g' 6381.conf

2. implementation details

1. In turn, when the master copy, using the RDB embodiment, the synchronization data from the master
2. After synchronization starts, by way of the spread of the main library commands , active replication to achieve
3. After 2.8 PSYNC realization of mechanism to achieve reconnection

3. redis-sentinel master copy from the HA

1 Introduction

1. Redis-Sentinel is redis official recommended high-availability solutions, when used as a high-availability redis master-slave, if master itself down, redis itself or the client did not achieve the main function from switching.
2. Process redis-sentinel is a stand-alone, for monitoring multiple master-slave cluster, auto-discovery master downtime, automatically switch slave> master.

2. sentinel main function

1. From time to time to monitor whether redis good run, if the node is unreachable nodes will be offline identity
2. If the logo is the master node, and the other will sentinel sentinel node "consultations", if other people can also master node node is unreachable, the election will be a sentinel node to complete automatic failover
3. After the master-slave switch, master_redis.conf, slave_redis.conf and sentinel.conf content will change, that will be more in line slaveof master_redis.conf configuration, monitoring target sentinel.conf will subsequently exchange

3. works (8)

1. Each Sentinel to once per second known frequency to its Master, Slave, and other examples of transmitting a PING command Sentinel

2. If an instance (instance) from the last time the PING command valid responses than down-after-milliseconds value specified by the option, the Sentinel this instance is marked as offline subjective .

3. If a Master is marked as subjective offline , we are monitoring the Master of all Sentinel frequency of once per second to be confirmed Master indeed entered a subjective offline state .

4. When a sufficient number of the Sentinel (not less than the value specified profiles) did enter the confirmation Master subjective offline state within a specified time, the Master is marked as offline objective

5. In general, each of the Sentinel will be once every 10 seconds of the frequency of all other known to the Master, the Slave transmits a command INFO

6. When the Master Sentinel objective is marked offline, the frequency of the Sentinel offline Master Slave sends an INFO command all the time will be changed every second from 10 seconds

7. Without a sufficient number of consent Master Sentinel has gone offline, objective offline Master status will be removed.

8. If the Master valid responses to the Sentinel return to the PING command, Master subjective offline status will be removed.

4. downline subjective and objective offline

• Subjective offline : Subjectively Down , referred SDOWN, referring offline Sentinel determine the current instance of the server made a redis.
• Objective offline : Objectively Down, referred ODOWN, it refers to a plurality of instances Sentinel SDOWN making judgment on Master Server, and interact with each other by a SENTINEL is-master-down-by -addr command later, offline derived Master Server Analyzing and open failover.
• SDOWN suitable for Master and Slave , as long as a Sentinel found Master entered ODOWN, this could be elected Sentinel Sentinel other out, and automatic failover operations off the assembly line of the primary server.
• ODOWN only applies to Master , Slave to the Redis instance, the Sentinel they are determined to be unnecessary consultations before the assembly line, so the Slave Sentinel never reach ODOWN.

4. redis-sentinel use

1. Preparing the Environment

• Three redis database instance, three configuration files

# sentinel_6379.conf
port 6379
daemonize yes
logfile "6379.log"
dbfilename "dump-6379.rdb"
dir "/var/redis/data/"
# sentinel_6380.conf 
port 6380
daemonize yes
logfile "6380.log"
dbfilename "dump-6380.rdb"
dir "/var/redis/data/"
slaveof 127.0.0.1 6379
# sentinel_6381.conf 
port 6381
daemonize yes
logfile "6381.log"
dbfilename "dump-6381.rdb"
dir "/var/redis/data/"
slaveof 127.0.0.1 6379

2. Start the database

redis-server 6379.conf
redis-server 6380.conf
redis-server 6381.conf

3. Configure Sentinel

• Three sentinel surveillance, three configuration files
• sentinel-26379.conf、sentine-26380.conf、sentinel-26381.conf

# sentinel-26379.conf  
port 26379  
dir /var/redis/data/
logfile "26379.log"
# 当前Sentinel节点监控 127.0.0.1:6379 这个主节点
# 2 代表判断主节点失败至少需要 2 个Sentinel节点节点同意
# mymaster 是主节点的别名
sentinel monitor mymaster  127.0.0.1 6379 2
# 每个Sentinel节点都要定期 PING 命令来判断 Redis 数据节点和其余Sentinel节点是否可达，如果超30000毫秒30s且没有回复，则判定不可达
sentinel down-after-milliseconds mymaster 30000
# 当Sentinel节点集合对主节点故障判定达成一致时，Sentinel领导者节点会做故障转移操作，选出新的主节点，
# 原来的从节点会向新的主节点发起复制操作，限制每次向新的主节点发起复制操作的从节点个数为1
sentinel parallel-syncs mymaster 1
# 故障转移超时时间为180000毫秒，3分钟
sentinel failover-timeout mymaster 180000
#加一个后台运行
daemonize yes 

# sentinel-26380.conf、sentinel-26381.conf
sed 's/26379/26380/g' sentinel-26379.conf > sentinel-26380.conf
sed 's/26379/26381/g' sentinel-26379.conf > sentinel-26381.conf

4. Start Sentinel

• After the first startup configuration file is automatically modified
• daemonize yes

redis-sentinel sentinel-26379.conf
redis-sentinel sentinel-26380.conf
redis-sentinel sentinel-26381.conf
# 测试哨兵是否启动成功
redis-cli -p 26379 info sentinel
redis-cli -p 26380 info sentinel
redis-cli -p 26381 info sentinel

5. Test

• Check main cut off from the main library state

kill -9 主库的pid
redis-cli -p 6379 info replication
redis-cli -p 6380 info replication
redis-cli -p 6381 info replication

7. redis-cluster configuration

0. outcome and the amount of data concurrency issues

1 Introduction

1. redis official generate up to 100,000 / per second, the implementation of 100,000 commands per second
2. A server memory is normally 16 ~ 256G, 500G if your business needs memory
3. Data overload
   • A server memory is normally 16 ~ 256G, 500G if your business needs memory, Sina Weibo as the world's largest redis store more than 1TB of data, where to buy such a large memory?
   • Major companies have their own solutions, launched their own clustering capabilities, the core idea is to redis in multiple instances, each piece is a redis instance data slice (sharding) storage.
4. Major companies clustering solutions:
   • twemproxy open source by Twitter
   • Codis by the pea pods developed, based on GO and C development
   • redis-cluster after the official version 3.0 cluster scheme

2. The client fragment

• redis3.0 cluster using P2P mode, completely to the center, all the key into the redis 16384 slots, each slot redis instance is responsible for a portion, all the information is updated by the cluster nodes exchange data.
• Examples redis main idea is to cluster the redis key data hashing, by the function of the particular hash key is mapped to a specified node redis

3. Data distribution theory

• Distributed Database primary solution to the entire data set is mapped in accordance with zoning rules to issue multiple nodes, namely the division of data sets across multiple nodes, each node is responsible for a subset of the entire data.
• Common zoning regulations have hash partitioning and partition order . Redis ClusterUsing hash partitioning rules , so the next rule will discuss hash partitioning.
  1. Node modulo partition
  2. Consistent hashing partition
  3. Virtual Partition grooves (redis-cluster mode used)

4. hash partitioning

• Node modulo

# 例如按照节点取余的方式，分三个节点
# 1~100的数据对3取余，可以分为三类
1. 余数为0
2. 余数为1
3. 余数为2

# 那么同样的分4个节点就是hash(key)%4
# 节点取余的优点是简单，客户端分片直接是哈希+取余

• Consistent hashing

  • Client fragmentation, clockwise modulo hash +
• Virtual slot partition
  1. Redis Cluster` virtual partition groove
  2. Virtual slot partition clever use of the Hash space, good dispersion of hash function used to map the data to all integers within a fixed range, the integer is defined as a groove (slot).
  3. Redis Cluster grooves range from 0 to 16,383.
  4. Slot is the cluster of data management and migration of basic units. The main purpose of using a wide range of grooves is to facilitate splitting and expansion of clusters of data,
  5. Each node is responsible for a certain number of slots.

Setting up an redis cluster

1. Preparing the Environment

• 配置 文件: Return-7000.conf, Redi-7001.conf, Redi-7002.conf, Redi-7003.conf, Redi-7004.conf, Redi-7005.conf
• Only the difference port

port 7000
daemonize yes
dir "/opt/redis/data"
logfile "7000.log"
dbfilename "dump-7000.rdb"
cluster-enabled yes                     #开启集群模式
cluster-config-file nodes-7000.conf     #集群内部的配置文件
# redis cluster需要16384个slot都正常的时候才能对外提供服务，换句话说，只要任何一个slot异常那么整个cluster不对外提供服务。 因此生产环境一般为no
cluster-require-full-coverage no　　  

sed 's/7000/7001/g' redis-7000.conf > redis-7001.conf
sed 's/7000/7002/g' redis-7000.conf > redis-7002.conf
sed 's/7000/7003/g' redis-7000.conf > redis-7003.conf
sed 's/7000/7004/g' redis-7000.conf > redis-7004.conf
sed 's/7000/7005/g' redis-7000.conf > redis-7005.conf

2. Start

redis-server redis-7000.conf
redis-server redis-7001.conf
redis-server redis-7002.conf
redis-server redis-7003.conf
redis-server redis-7004.conf
redis-server redis-7005.conf

3. Assign redis slot slot

• C language manual distribution
• Use of a ruby ​​redis module automatically assigned
• Configuration ruby ​​scripting environment

# 安装
yum install -y ruby
# 使用 ruby和gem

• Download ruby ​​module operation redis

wget http://rubygems.org/downloads/redis-3.3.0.gem

• Use ruby ​​package management tools installed gem

gem install -l redis-3.3.0.gem

• A key slot allocated by ruby

# 找到redis-trib.rb命令
find / -name redis-trib.rb
# 开启集群开启槽位
/opt/redis-4.0.10/src/redis-trib.rb create --replicas 1 127.0.0.1:7000 127.0.0.1:7001 127.0.0.1:7002 127.0.0.1:7003 127.0.0.1:7004 127.0.0.1:7005
# 分配好集群后，向集群中写入数据
redis-cli -c -p 7000
set name 'xiake'

python operation redis

1. Install

# win
下载redis到指定目录，配置PATH即可
# mac
brew install redis

2. Use

1. Use redis key: value stored, the hash storage structure {key: value}
2. Repeatedly set the same key will be overwritten

# 终端
redis-cli
# 总共 16 个库，0-15，用来数据隔离
select 8                    # 切换 8 号库，默认 0 号库
set key value               # 设置一个健值对，哈希存储结构{key:value}
keys pattern                # 查询当前数据库中所有的key,如keys * 查询当前数据库中所有key
         a*                 # 查询以 a开头
     *n*                    # 包含 n
...
get key                     # 查询 key 对应的 value

3. python operation redis

from redis import Redis
redis_cli = Redis(host='127.0.0.1', port=6379, db=6)
redis_cli.set('name', 'echo')