codis -- redis cluster solution

1. Online redis cluster architecture diagram

haproxy: forwarding and load balancing on the client side (qb, uc, etc.)

codis-config: manage codis and provide dashboard

codis-proxy: the proxy of redis, which implements the redis protocol

codis: the redis service, developed based on redis2.8.13

zookeeper: manage and coordinate codis-proxy node information

codis-ha: Monitor the codis service and pull up the service in time when the codis service exits abnormally

codis-proxy-ha: Monitor the codis-proxy service and pull up the service in time when the codis-proxy service exits abnormally

 

There are 16 physical machines online, each of which is deployed with codis, of which codis-proxy is deployed separately from zookeeper, codis-config, and haproxy.

 

Second, the performance of codis cluster

The maximum pressure of a single pressure machine can reach 20w OPS.

 

3. High availability of codis cluster

Continued 7*24 hours of pressure testing with 13w OPS, the total key operations reached 90 billion, 500 were missed, and the hit rate exceeded 99.99%.

During the stress test, codis-proxy exited abnormally 10 times, but no exception occurred in codis.

===================================================================================

 

Most of the redis in the line network are switched to the codis cluster

However, there are some restrictions in the use process, please pay attention and check whether there is any problem with your code.

 

Some online applications have been switched to codis (redis) clusters. Due to the characteristics of codis, please help to popularize the following points in the development team:

1. The keys used by each application need naming conventions to avoid key conflicts, because for all online applications, a redis is used

Naming convention: http://x.qbao.com/pages/viewpage.action?pageId=10716303

2. Only the db0 library can be used, other libraries are not supported

3. Considering the design of codis itself, try to avoid the occurrence of small keys, but very large values ​​(>1M)

4. Some commands are not supported. The list of unsupported commands is as follows: https://github.com/wandoulabs/codis/blob/master/doc/unsupported_cmds.md

Command Type Command Name

Keys	KEYS
	MIGRATE
	MOVE
	OBJECT
	RANDOMKEY
	RENAME
	RENAMENX
	SCAN
Strings	BITOP
	MSETNX
Lists	BLPOP
	BRPOP
	BRPOPLPUSH
Pub/Sub	PSUBSCRIBE
	PUBLISH
	PUNSUBSCRIBE
	SUBSCRIBE
	UNSUBSCRIBE
Transactions	DISCARD
	EXEC
	MULTI
	UNWATCH
	WATCH
Scripting	SCRIPT
Server	BGREWRITEAOF
	BGSAVE
	CLIENT
	CONFIG
	DBSIZE
	DEBUG
	FLUSHALL
	FLUSHDB
	LASTSAVE
	MONITOR
	RESTORE
	SAVE
	SHUTDOWN
	SLAVEOF
	SLOWLOG
	SYNC
	TIME
Codis Slot	FINAL CHECK
	SLOTSDEL
	SLOTSINFO
	SLOTSMGRTONE
	SLOTSMGRTSLOT
	SLOTSMGRTTAGONE
	SLOTSMGRTTAGSLOT

===================================================================

Naming conventions for Redis keys

 

Overview:

为了降低各系统的耦合度，后面redis根据应用域名进行分区存储，各应用使用各自的redis。老的redis保留，兼容老的应用程序。

命名规范：“域:业务:功能:特性”

“域:业务”是必须有的，“功能:特性”是扩展字段

以宝筹项目为例：存储商品id为1的买5交易数据，key为“baochou:trade:buy5:1”；用户还在委托中的单子数量，key为 “baochou:trade:pending:2015-01-26_102661_1”

各应用可以提供一个命名的帮助类，核心方法如下：

//key分隔符

public final static String REDIS_NAMESPACE_SEPARATOR = ":";

//静态key部分

public final static String REDIS_KEY_USER="user:register:id";

 

//动态key生成方法

@Override
public String generateCacheKey(String namespace, Object... keys) {
        StringBuffer out=new StringBuffer();
        out.append(namespace);
        out.append(REDIS_NAMESPACE_SEPARATOR);
        for(int i=0;i<keys.length;i++){
                out.append(keys[i]);
                if(i!=keys.length-1){ 
                        out.append("_“）；
              }
        }
        return out.toString();
}

例如：“baochou:trade:pending:2015-01-26_102661_1”，“baochou:trade:pending“即为namespace，”2015-01-26_102661_1“为“日期_用户id_商品id”，整个key描述了“用户某天某个商品的正在委托中的数量” 

目前整理的业务应用：

各业务后面提供单独的redis

• 后台：backoffice & 前台www & 接口m

• gamepay & 订单order （目前未用reds）
• bizorder （微商订单，目前用的qianbao redis）
• 宝筹：bc  & bcbo 宝筹后台
• 商品goods & goodsba
• im
• leipai
• 用户user
• 登陆passport
• 拍卖paimai & pmba  拍卖后台
• qhb 抢红包
• t  有票 & ticketbo 有票后台
• tpush  有票推送

现在的应用中有同一redis实例中跨应用使用同一key的情况，后面分应用存储以后，坚决杜绝此情况，有跨应用使用的情况，请提供http接口。

各业务redis key查询列表：

各业务自己添加，表头两项：“key名称” “key含义”，例如：

key名称

key含义

baochou:trade:buy5:1	存储商品id为1的买5交易数据
baochou:trade:pending:2015-01-26_102661_1	用户某天某个商品的正在委托中的数量

=====================================================

线上大部分的应用服务使用的redis服务已经切换到了新的codis（基于redis2.8.13开发）集群（除了抢红包和签到所用的redis没有迁移），测试环境也需要搭建相应的codis服务，

鉴于codis的一些特性，大家在测试过程中有些地方需要注意：

1、线上大部分应用都使用同一codis集群，所以各个应用模块的key命名需要规范，避免key冲突的情况。

2、只能使用db0 的库，其它库均不支持

3、部分命令不支持，不支持的命令列表如下：

Command TypeCommand Name

KeysKEYS

MIGRATE

MOVE

OBJECT

RANDOMKEY

RENAME

RENAMENX

SCAN

StringsBITOP

MSETNX

ListsBLPOP

BRPOP

BRPOPLPUSH

Pub/SubPSUBSCRIBE

PUBLISH

PUNSUBSCRIBE

SUBSCRIBE

UNSUBSCRIBE

TransactionsDISCARD

EXEC

MULTI

UNWATCH

WATCH

ScriptingSCRIPT

ServerBGREWRITEAOF

BGSAVE

CLIENT

CONFIG

DBSIZE

DEBUG

FLUSHALL

FLUSHDB

LASTSAVE

MONITOR

RESTORE

SAVE

SHUTDOWN

SLAVEOF

SLOWLOG

SYNC

TIME

Codis SlotSLOTSCHECK

SLOTSDEL

SLOTSINFO

SLOTSMGRTONE

SLOTSMGRTSLOT

SLOTSMGRTTAGONE

SLOTSMGRTTAGSLOT

 

codis最简版搭建已经放到知识库了：《最简版codis（redis）搭建》

运维也会在预发布搭建简易的codis集群环境。

 

一、zookeeper安装：

1、下载解压

2、将conf下的zoo_sample.cfg改为zoo.cfg，配置：

tickTime=2000

initLimit=10

syncLimit=5

dataDir=/opt/zookeeper/data

dataLogDir=/opt/zookeeper/log

clientPort=2181

server.1=192.168.7.21:2888:3888

 

注意，建data和log目录

3、data下新建myid文件，每个zookeeper写入不同的id

4、./bin/zkSever.sh start启动

 

二、部署codis

四、服务启动，集群初始化

1、启动dashboard（codis-config上操作）

# cat config.ini

zk=192.168.7.21:2181

product=codis-cluster

proxy_id=proxy_1

net_timeout=30

dashboard_addr=0.0.0.0:18087

coordinator=zookeeper

 

nohup ../bin/codis-config -c config.ini -L ./log/dashboard.log dashboard --addr=:18087 --http-log=./log/requests.log &

 

 

2、初始化slots（codis-config上操作）

../bin/codis-config -c ./config.ini slot init

 

 

3、启动codis（等同于redis，基于redis2.8.13开发，加入了slot支持和原子的数据迁移指令，且codis-proxy和codis-config只能和这个版本进行交互）(在codis-server上操作)

../bin/codis-server ./redis_conf/6379.conf

 

4、添加redis server group（可以在dashboard上操作），每个group最多有一个master，可以有多个slave（在codis-config上操作）

命令模版：codis-config server add <group_id> <redis_addr> <role>

添加两个 server group, 每个 group 有两个 redis 实例，group的id分别为1和2， redis实例为一主一从。

../bin/codis-config -c ./config.ini server add 1 192.168.7.215:6379 master

 

 

5、设置每个slot由那个sever group来提供服务（可以在dashboard上操作），默认slots分成1024份（0-1023）（codis-config上操作）

../bin/codis-config -c ./config.ini slot range-set 0 1023 online

 

6、启动codis-proxy

# cat config.ini

zk=192.168.7.21:2181

product=codis-cluster

proxy_id=proxy_103

net_timeout=30

dashboard_addr=0.0.0.0:18087

coordinator=zookeeper

注意：proxy_id在同一个集群里是唯一的

 

nohup ../bin/codis-proxy -c ./config.ini -L /var/log/codis/codis-proxy.log --log-level=warn --cpu=8 --addr=0.0.0.0:1900 --http-addr=0.0.0.0:11000 &

 

After the proxy is started, it needs to be online before it can provide services to the outside world. You can go online on the dashboard, or use the command to go online.

../bin/codis-config -c config.ini proxy online proxy_113

 

After the setup is complete, the configuration of redis.properties in the environment is changed to the ip of codis-proxy: the port of codis-proxy (1900)

example：

Before changing:

redis.ip=192.168.7.217

redis.port=6379

Change to new codis:

redis.ip=192.168.7.103 

redis.port=1900

 

The codis management page can be accessed at http://ip:18087/admin.