KubeSphere installs redis cluster, the whole process is super exciting

foreword

redisIs often used in the development process 缓存中间件, in 生产环境order to consider 稳定性and 高可用generally for 集群模式the deployment.

Conventional deployment on virtual machines is cumbersome to configure and requires manual restart of nodes, while K8Scluster redisdeployment has the following advantages:

• Easy installation : use a mirror image or yamlconfiguration file to install it in one piece
• Automatic scheduling : After the container hangs up, it will automatically schedule restart and resource allocation
• Convenient scaling and expansion : Needless to say, the advantages in terms of, one-click 扩容scaling缩容
• Stable and efficient : k8sScheduling is performed on the entire cluster, as long as the entire cluster does not hang up, it will always be scheduled to a suitable node to restart the container service

1. Install Redis cluster

I created a test-projectnew project space here to Redisplace all the installation resources of the cluster. DNSThe name of the project space will be used later on, and this part will be marked. You need to pay attention to using your own project space name.

Installing a cluster is roughly divided into the following steps:

1. configuration redis.confdictionary
2. create redisservice
3. Container group configuration
4. storage settings
5. advanced settings

Now start with the first step.

1.1 Configure redis.conf dictionary

Create a configuration dictionary in 配置→ 配置字典→ in the project space创建

The name is called redis-confand the next step is to add key-value pair data

keyThe content of the value is redis.conf, valuethe value is:

cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
cluster-require-full-coverage no
cluster-migration-barrier 1
appendonly yes

1.2 Create Redis service

Create a Redis service in 应用负载→ 服务→ in the project space创建

Select 有状态服务, services that require data storage are basically stateful services

In the basic settings, the name is called redis-clusterand then the highlight, the next step容器组配置

1.3 Container Group Configuration

The core of this step is to configure the Redis container. The number of clusters is conventional, and 三主三从the number of copies of the container is 6.

Adjust the number of copies of the container group to 6, click Add Container

docker hubThe version in the image selection , and choose to use the default port, and can be reserved selectively, if not reserved, it is scheduling public resources redis.6.2.3CPU内存

If selected 使用默认端口, the following port settings will be used as shown in the above figure 6379, and there is also a configuration startup command.

Configure as shown above:

• Order:redis-server
• parameter:/etc/redis/redis.conf

The parameter points to the content of the previous dictionary configuration, but 存储设置the dictionary needs to be configured in the next step before it can be used.

There is nothing to configure in other content, select the tick to complete the container configuration.

The update strategy is recommended 滚动更新, and there is nothing else that needs to be modified. Click Next to configure 存储设置.

1.4 Storage settings

In this step there are two operations

• 添加持久卷声明模板
• 挂载配置字典

1.4.1 Add Persistent Volume Claim Template

PVC name prefix : redis-pvc

Capacity : 10G

Mount path :

• Permissions : read and write
• Address : /data
  
  is mainly to select the mount path, after configuration, click the check mark to complete the configuration

1.4.2 Mount configuration dictionary

This step is to mount the dictionary we configured before redis-conf, which is also redisthe parameter content of our startup command

Selected redisconfiguration dictionary

The mount authority is: 只读, and the address is: /etc/redis; corresponding to the configuration of the above command parameters.

Select the specific key redis.confwith the same name behind it redis.conf, and click the check mark to return to the storage setting after completion

After the configuration is complete, enter the above picture, click Next to enter the final高级设置

高级设置Here are some additional configurations, you can choose to adjust the configuration according to your own scene, click to create the Redis cluster container after the adjustment is completed创建

2. Initialize the Redis cluster

Redis服务After creation , click on redisthe service name to enter redisthe service details, as shown in the figure below:

The 6 rediscontainer groups have all started successfully, and the next step is to initialize the cluster; because redisthe service we configured is 有状态服务(Headless)so the access mode can be through internal DNS, the access format is: ( 容器名称) (.Container DNS ).svc.cluster.local

According to the example in the above figure, for example, the access 集群1 节点address is redis-cluster-v1-1Add DNSAddress redis-cluster.test-projectAdd svc.cluster.local, the complete address is as follows:

redis-cluster-v1-1.redis-cluster.test-project.svc.cluster.local

In redis集群other node terminals other than the first node, visit through this address to verify whether they are interoperable, and enter 3节点, 终端as shown in the following figure:

Enter the terminal and execute the command:

redis-cli -h redis-cluster-v1-1.redis-cluster.test-project.svc.clusterredis.local

If you can jump to v1-1the node as shown in the figure below, it means that the two nodes communicate with each other

Execute the command: cluster infoview the cluster status of the node

Mainly look at the two parameters in the above figure. nodesIn order to 1indicate that there is only 1one node at present, cluster_sizeit indicates that there are no masternodes at present, so it is not a cluster structure at present. infoThe detailed explanation of the attributes is listed here:

• cluster_state: The ok state indicates that the cluster can accept query requests normally. The fail state indicates that at least one hash slot is not bound (indicating that a hash slot is not bound to any node), or in the wrong state (the node can provide services but has a FAIL flag), or the node Unable to contact most master nodes
• cluster_slots_assigned: The number of hash slots allocated to cluster nodes (not the number not bound). All 16384 hash slots are allocated to the cluster nodes is a necessary condition for the normal operation of the cluster
• cluster_slots_ok: number of hash slot status not FAIL and PFAIL
• cluster_slots_pfail: The hash slot status is the number of PFAIL. As long as the hash slot status is not upgraded to FAIL status, these hash slots can still be processed normally. The PFAIL state means that we are currently unable to interact with the node, but this state is only a temporary error state
• cluster_slots_fail: The number of hash slots whose status is FAIL. If the value is not 0, then cluster nodes will not be able to serve queries unless cluster-require-full-coverage is set to no.
• cluster_known_nodes: The number of nodes in the cluster, including nodes that are in the handshake state and have not yet become official members of the cluster
• cluster_size: The number of master nodes that contain at least one hash slot and can provide services
• cluster_current_epoch: The value of the cluster local Current Epoch variable. This value is useful during node failover process, it is always incremented and unique.
• cluster_my_epoch: The Config Epoch value of the node currently in use. This is the version value associated with this node
• cluster_stats_messages_sent: number of messages sent over the node-to-node binary bus
• cluster_stats_messages_received: number of messages received over the node-to-node binary bus

2.1 IP address initialization cluster

First try to use ip + portthe method to initialize the cluster, but k8sthe service ipwill change in the middle, so the final result still needs to use DNSthe method to initialize the cluster.

执行本步后再想修改为DNS地址初始化需要从来一遍，如果不想麻烦的同学可以直接跳过。

Record redisall of the cluster ip+port, the initialization command is as follows:

redis-cli --cluster create 10.233.70.30:6379 10.233.70.32:6379 10.233.90.41:6379 10.233.90.43:6379 10.233.96.47:6379 10.233.96.50:6379 --cluster-replicas 1

Enter the cluster to execute the above command redison any node终端

As shown in the figure above, the cluster initialization is completed, and then enter the command redis-clito enter the command terminal, and then execute cluster infoto view the cluster information

Now we have 6one cluster node and masterthree nodes, the cluster is established, and the subsequent operation selects masterthe node to operate.

When verifying the cluster nodes, if you encounter the error in the above figure, (error) MOVED 2589 10.233.70.30:6379it is because redis-clithe cluster mode is not enabled, and you can redis-cli -cswitch to the cluster mode by changing the command to .

2.2 Initialize with internal DNS

The method of using the address will change after ipeach k8sscheduling , so it is not appropriate to use the method to initialize the cluster in the cluster , but if you use the internal method to initialize the cluster directly as above, an error will occur, because the support for domain names is not very good. So it can be used at this time .redisipk8sipDNSredisRedis-tribe

2.2.1 Create Redis-tribe service

To create a custom service, choose编辑YAML

namespaceWrite the project name as the parameter :

The specific YAMLcontent is as follows:

apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: test-project
  labels:
    app: redis-cluster-tools
  name: redis-cluster-tools
spec:
  replicas: 1
  selector:
    matchLabels:
      app: redis-cluster-tools
  template:
    metadata:
      labels:
        app: redis-cluster-tools
      name: pos-redis
    spec:
     containers:
     - name: pos-redis
       image: sunnywang/redis-tools-ubuntu:v0.5.1
       imagePullPolicy: IfNotPresent
       args:
       - /bin/bash
       - -c
       - sleep 3600

Found in the container group after creationredis-cluster-tools

2.2.2 Initialize the cluster

Click on the container name to enter the container details and then enter the terminal

First execute the following command to initialize masterthe node, then the previous internal DNSdomain name will be useful

redis-trib.py create `dig +short redis-cluster-v1-0.redis-cluster.test-project.svc.cluster.local`:6379 `dig +short redis-cluster-v1-1.redis-cluster.test-project.svc.cluster.local`:6379 `dig +short redis-cluster-v1-2.redis-cluster.test-project.svc.cluster.local`:6379

The execution result is as follows:

Next, bind each masternode to the corresponding copy node, a total of three:

0 nodes -> 3 nodes

redis-trib.py replicate --master-addr `dig +short redis-cluster-v1-0.redis-cluster.test-project.svc.cluster.local`:6379 --slave-addr `dig +short redis-cluster-v1-3.redis-cluster.test-project.svc.cluster.local`:6379

1 node -> 4 nodes

redis-trib.py replicate  --master-addr `dig +short redis-cluster-v1-1.redis-cluster.test-project.svc.cluster.local`:6379 --slave-addr `dig +short redis-cluster-v1-4.redis-cluster.test-project.svc.cluster.local`:6379

2 nodes -> 5 nodes

redis-trib.py replicate --master-addr `dig +short redis-cluster-v1-2.redis-cluster.test-project.svc.cluster.local`:6379 --slave-addr `dig +short redis-cluster-v1-5.redis-cluster.test-project.svc.cluster.local`:6379

The execution results are as follows:

2.2.3 Verification

Just enter the terminal of a cluster node, or execute cluster infothe command to view the cluster information

Use basic commands to verify, and verify the cluster mode redis-clineeds to be added-c

Verify that the cluster mode can be used normally, and then you should play freely

If you think it's okay, all the judges will give you a thumbs up with your little hand that made a fortune,未来的运气能爆棚哦。