Redis is a high-performance key-value storage system that is widely used in microservice architecture. If we want to use the advanced features provided by the Redis cluster mode, we need to modify the client code, which brings some difficulties in application upgrade and maintenance. Using Istio and Envoy, we can implement client-insensitive Redis Cluster data sharding without modifying the client code, and provide advanced traffic management functions such as read-write separation and traffic mirroring.
Redis Cluster
A common use of Redis is as a data cache. By adding a Redis cache layer between the application server and the database server, a large number of read operations on the database by the application server can be reduced, and the risk of slow response or even downtime of the database server under heavy pressure can be avoided, and the robustness of the entire system can be significantly enhanced. The principle of Redis as a data cache is shown in the figure:
In a small-scale system, the single Redis shown in the figure above can well implement the function of the cache layer. When the amount of data that needs to be cached in the system is large, one Redis server cannot meet the caching requirements of all application servers; at the same time, when a single Redis instance fails, a large number of read requests will be sent directly to the back-end database server, resulting in transient database server Excessive pressure affects the stability of the system. We can use Redis Cluster to fragment the cached data and put different data into different Redis fragments to improve the capacity of the Redis cache layer. In each Redis shard, multiple replica nodes can also be used to share the load of cached read requests and achieve high availability of Redis. The system using Redis Cluster is shown below:
As you can see from the figure, in the Redis Cluster mode, the client needs to send read and write operations of different keys to different Redis nodes in the cluster according to the fragmentation rules of the cluster, so the client needs to understand the topology of the Redis Cluster. This makes it impossible for us to smoothly migrate an application using Redis independent node mode to Redis Cluster without modifying the client. In addition, since the client needs to understand the internal topology of Redis Cluster, it will also lead to the coupling of client code and Redis Cluster operation and maintenance. For example, to achieve read-write separation or traffic mirroring, it is necessary to modify the code of each client and redeploy. .
In this scenario, we can place an Envoy proxy server between the application server and the Redis Cluster, and Envoy is responsible for routing the cache read and write requests issued by the application to the correct Redis node. There are a large number of application processes that need to access the cache server in a microservice system. To avoid single points of failure and performance bottlenecks, we deploy an Envoy proxy for each application process in the form of Sidecar. At the same time, in order to simplify the management of these proxies, we can use Istio as the control plane to uniformly configure all Envoy proxies, as shown in the following figure:
In the subsequent part of this article, we will introduce how to manage Redis Cluster through Istio and Envoy to realize client-insensitive data partitioning, as well as advanced routing strategies such as read-write separation and traffic mirroring.
Deploy Istio
The Redis protocol is already supported in Pilot, but the function is weak. It can only configure a default route for the Redis proxy, and does not support the Redis Cluster mode. It is impossible to realize the Redis filter data fragmentation, read-write separation, traffic mirroring and other advanced traffic management Features. In order to allow Istio to deliver Redis Cluster related configuration to Envoy Sidecar, we modified the EnvoyFilter configuration related code to support EnvoyFilter's "REPLCAE" operation. The revised PR Implement REPLACE operation for EnvoyFilter patch has been submitted to the Istio community, merged into the main branch, and will be released in subsequent versions of Istio.
At the time of writing this article, the latest Istio release version 1.7.3 has not yet incorporated this PR. So I built a Pilot image to enable the "REPLACE" operation of EnvoyFilter. When installing Istio, we need to specify the Pilot image in the istioctl command, as shown in the following command line:
$ cd istio-1.7.3/bin
$ ./istioctl install --set components.pilot.hub=zhaohuabing --set components.pilot.tag=1.7.3-enable-ef-replaceNote: If your Istio version is newer than 1.7.3 and the PR has been incorporated, you can directly use the default Pilot mirroring in the Istio version.
Deploy Redis Cluster
Please download the relevant code used in the following example from https://github.com/zhaohuabing/istio-redis-culster :
$ git clone https://github.com/zhaohuabing/istio-redis-culster.git
$ cd istio-redis-culsterWe create a "redis" namespace to deploy the Redis Cluster in this example.
$ kubectl create ns redis
namespace/redis createdDeploy the Statefulset and Configmap of the Redis server.
$ kubectl apply -f k8s/redis-cluster.yaml -n redis
configmap/redis-cluster created
statefulset.apps/redis-cluster created
service/redis-cluster createdVerify Redis deployment
Confirm that the Redis node is up and running normally:
$ kubectl get pod -n redis
NAME READY STATUS RESTARTS AGE
redis-cluster-0 2/2 Running 0 4m25s
redis-cluster-1 2/2 Running 0 3m56s
redis-cluster-2 2/2 Running 0 3m28s
redis-cluster-3 2/2 Running 0 2m58s
redis-cluster-4 2/2 Running 0 2m27s
redis-cluster-5 2/2 Running 0 117sCreate Redis Cluster
In the above steps, we deployed 6 Redis nodes using Statefulset, but currently these 6 nodes are still independent of each other and do not form a cluster. Here we use the Redis cluster createcommands these nodes a Redis Cluster.
$ kubectl exec -it redis-cluster-0 -n redis -- redis-cli --cluster create --cluster-replicas 1 $(kubectl get pods -l app=redis-cluster -o jsonpath='{range.items[*]}{.status.podIP}:6379 ' -n redis)
Defaulting container name to redis.
Use 'kubectl describe pod/redis-cluster-0 -n redis' to see all of the containers in this pod.
>>> Performing hash slots allocation on 6 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
Adding replica 172.16.0.72:6379 to 172.16.0.138:6379
Adding replica 172.16.0.201:6379 to 172.16.1.52:6379
Adding replica 172.16.0.139:6379 to 172.16.1.53:6379
M: 8fdc7aa28a6217b049a2265b87bff9723f202af0 172.16.0.138:6379
slots:[0-5460] (5461 slots) master
M: 4dd6c1fecbbe4527e7d0de61b655e8b74b411e4c 172.16.1.52:6379
slots:[5461-10922] (5462 slots) master
M: 0b86a0fbe76cdd4b48434b616b759936ca99d71c 172.16.1.53:6379
slots:[10923-16383] (5461 slots) master
S: 94b139d247e9274b553c82fbbc6897bfd6d7f693 172.16.0.139:6379
replicates 0b86a0fbe76cdd4b48434b616b759936ca99d71c
S: e293d25881c3cf6db86034cd9c26a1af29bc585a 172.16.0.72:6379
replicates 8fdc7aa28a6217b049a2265b87bff9723f202af0
S: ab897de0eca1376558e006c5b0a49f5004252eb6 172.16.0.201:6379
replicates 4dd6c1fecbbe4527e7d0de61b655e8b74b411e4c
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
.
>>> Performing Cluster Check (using node 172.16.0.138:6379)
M: 8fdc7aa28a6217b049a2265b87bff9723f202af0 172.16.0.138:6379
slots:[0-5460] (5461 slots) master
1 additional replica(s)
M: 4dd6c1fecbbe4527e7d0de61b655e8b74b411e4c 172.16.1.52:6379
slots:[5461-10922] (5462 slots) master
1 additional replica(s)
S: 94b139d247e9274b553c82fbbc6897bfd6d7f693 172.16.0.139:6379
slots: (0 slots) slave
replicates 0b86a0fbe76cdd4b48434b616b759936ca99d71c
M: 0b86a0fbe76cdd4b48434b616b759936ca99d71c 172.16.1.53:6379
slots:[10923-16383] (5461 slots) master
1 additional replica(s)
S: ab897de0eca1376558e006c5b0a49f5004252eb6 172.16.0.201:6379
slots: (0 slots) slave
replicates 4dd6c1fecbbe4527e7d0de61b655e8b74b411e4c
S: e293d25881c3cf6db86034cd9c26a1af29bc585a 172.16.0.72:6379
slots: (0 slots) slave
replicates 8fdc7aa28a6217b049a2265b87bff9723f202af0
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.Verify Redis Cluster
We can use the cluster infocommand to view the configuration information and Redis Cluster Cluster member nodes in order to verify that the cluster is created successfully.
$ kubectl exec -it redis-cluster-0 -c redis -n redis -- redis-cli cluster info
cluster_state:ok
cluster_slots_assigned:16384
cluster_slots_ok:16384
cluster_slots_pfail:0
cluster_slots_fail:0
cluster_known_nodes:6
cluster_size:3
cluster_current_epoch:6
cluster_my_epoch:1
cluster_stats_messages_ping_sent:206
cluster_stats_messages_pong_sent:210
cluster_stats_messages_sent:416
cluster_stats_messages_ping_received:205
cluster_stats_messages_pong_received:206
cluster_stats_messages_meet_received:5
cluster_stats_messages_received:416Deploy the test client
We deploy a client to send test commands:
$ kubectl apply -f k8s/redis-client.yaml -n redis
deployment.apps/redis-client createdDistribute Envoy configuration related to Redis Cluster through Istio
In the following steps, we will send Redis Cluster related configuration to Envoy Sidecar through Istio to enable the advanced functions of Redis Cluster without changing the client, including data fragmentation, read-write separation, and traffic mirroring.
Create Envoy Redis Cluster
Envoy provides an Envoy Cluster of type "envoy.clusters.redis" to connect to the back-end Redis Cluster. Envoy will obtain the topology of the back-end Redis Cluster through the Cluster, including how many shards, and each shard is responsible Which slots and which nodes are included in the shard to distribute requests from clients to the correct Redis nodes.
Use EnvoyFilter to create the required Envoy Redis Cluster:
$ kubectl apply -f istio/envoyfilter-custom-redis-cluster.yaml
envoyfilter.networking.istio.io/custom-redis-cluster createdCreate Envoy Redis Proxy
Istio's default LDS is configured with TCP proxy filter, we need to replace it with Redis Proxy filter.
Since the "REPLACE" operation of EnvoyFilter is not yet supported in 1.7.3, we first need to update the CRD definition of EnvoyFilter before we can create the EnvoyFilter:
$ kubectl apply -f istio/envoyfilter-crd.yaml
customresourcedefinition.apiextensions.k8s.io/envoyfilters.networking.istio.io configuredUse EnvoyFilter to replace TCP proxy filter with Redis Proxy filter so that Envoy can proxy Redis operation requests from clients:
$ sed -i .bak "s/\${REDIS_VIP}/`kubectl get svc redis-cluster -n redis -o=jsonpath='{.spec.clusterIP}'`/" istio/envoyfilter-redis-proxy.yaml
$ kubectl apply -f istio/envoyfilter-redis-proxy.yaml
envoyfilter.networking.istio.io/add-redis-proxy createdVerify Redis Cluster function
Now that everything is ready, let's verify the various functions of Redis Cluster.
Redis data sharding
After we send the configuration defined in EnvoyFilter to Envoy through Istio, Envoy can automatically discover the topology of the back-end Redis Cluster, and automatically distribute the request to the correct node in the Redis Cluster according to the key in the client request.
According to the command line output in the previous step of creating Redis Cluster, we can see the topology of the Redis Cluster: There are three shards in the Cluster, and each shard has a Master node and a Slave (Replica) node. The client accesses the Redis Cluster through the Envoy Proxy deployed in the same Pod, as shown in the following figure:
The master and slave node addresses of each shard in Redis Cluster:
Shard[0] Master[0] redis-cluster-0 172.16.0.138:6379 replica redis-cluster-4 172.16.0.72:6379 -> Slots 0 - 5460
Shard[1] Master[1] redis-cluster-1 172.16.1.52:6379 replica redis-cluster-5 172.16.0.201:6379 -> Slots 5461 - 10922
Shard[2] Master[2] redis-cluster-2 172.16.1.53:6379 replica redis-cluster-3 172.16.0.139:6379 -> Slots 10923 - 16383Note: If you deploy this example in your own K8s cluster, the IP address and topology of each node in Redis Cluster may be slightly different, but the basic structure should be similar.
We try to send a number of different key to Rdeis Cluster from client setrequests:
$ kubectl exec -it `kubectl get pod -l app=redis-client -n redis -o jsonpath="{.items[0].metadata.name}"` -c redis-client -n redis -- redis-cli -h redis-cluster
redis-cluster:6379> set a a
OK
redis-cluster:6379> set b b
OK
redis-cluster:6379> set c c
OK
redis-cluster:6379> set d d
OK
redis-cluster:6379> set e e
OK
redis-cluster:6379> set f f
OK
redis-cluster:6379> set g g
OK
redis-cluster:6379> set h h
OKFrom the client perspective, all requests are successful, we can use the scancommand to view the data in each node on the server side:
View the data in shard[0], the master node is redis-cluster-0 and the slave node is redis-cluster-4.
$ kubectl exec redis-cluster-0 -c redis -n redis -- redis-cli --scan
b
f
$ kubectl exec redis-cluster-4 -c redis -n redis -- redis-cli --scan
f
bCheck the data in the shard Shard[1], the master node is redis-cluster-1 and the slave node is redis-cluster-5.
$ kubectl exec redis-cluster-1 -c redis -n redis -- redis-cli --scan
c
g
$ kubectl exec redis-cluster-5 -c redis -n redis -- redis-cli --scan
g
cCheck the data in the shard [2], the master node is redis-cluster-2 and the slave node is redis-cluster-3.
$ kubectl exec redis-cluster-2 -c redis -n redis -- redis-cli --scan
a
e
d
h
$ kubectl exec redis-cluster-3 -c redis -n redis -- redis-cli --scan
h
e
d
aAs can be seen from the above verification results, the data set by the client is distributed to the three shards in the Redis Cluster. The data distribution process is automatically implemented by Envoy Redis Proxy. The client does not perceive the backend Redis Cluster. For the client, the interaction with the Redis Cluster is the same as the interaction with a single Redis node.
Using this method, we can seamlessly migrate Redis in the system from single node to cluster mode when the scale of application business is gradually expanding and the pressure on a single Redis node is too high. In the cluster mode, the data of different keys is cached in different data shards. We can increase the number of Replica nodes in the shards to expand a shard, or increase the number of shards to expand the entire cluster , In order to cope with the increased data pressure due to business expansion. Since Envoy can perceive the Redis Cluster cluster topology, and the data distribution is completed by Envoy, the entire migration and expansion process does not require a client, and will not affect the normal operation of online business.
Redis read-write separation
In a Redis shard, there is usually one Master node and one or more Slave (Replica) nodes. The Master node is responsible for writing operations and synchronizing data changes to the Slave node. When the read operation pressure from the application is high, we can add more replicas to the shard to share the load of the read operation. Envoy Redis Rroxy supports setting different read strategies:
- MASTER: Only read data from the Master node. This mode is required when the client requires strong data consistency. This mode puts a lot of pressure on the Master, and multiple nodes cannot be used to load share the read operation in the same shard.
- PREFER_MASTER: Read data from the Master node first. When the Master node is unavailable, read data from the Replica node.
- REPLICA: Only read data from the Replica node. Since the data replication process from Master to Replica is performed asynchronously, it is possible to read expired data in this way, so it is suitable for scenarios where the client does not have high data consistency requirements. In this mode, multiple Replica nodes can be used to share the read load from the client.
- PREFER_REPLICA: Read data from the Replica node first, and read from the Master node when the Replica node is unavailable.
- ANY: Read data from any node.
In the EnvoyFilter issued earlier, we set the read strategy of Envoy Redis Proxy to "REPLICA", so client read operations should only be sent to the Replica node. Let us use the following command to verify the read-write separation strategy:
By initiating a series of key client is "b" of getand setactions:
$ kubectl exec -it `kubectl get pod -l app=redis-client -n redis -o jsonpath="{.items[0].metadata.name}"` -c redis-client -n redis -- redis-cli -h redis-cluster
redis-cluster:6379> get b
"b"
redis-cluster:6379> get b
"b"
redis-cluster:6379> get b
"b"
redis-cluster:6379> set b bb
OK
redis-cluster:6379> get b
"bb"
redis-cluster:6379> In the previous Redis Cluster topology, we have learned that the key "b" belongs to the Shard[0] shard. We can command redis-cli monitorto view the command slice Master and Replica node received.
Master node:
$ kubectl exec redis-cluster-0 -c redis -n redis -- redis-cli monitorSlave node:
$ kubectl exec redis-cluster-4 -c redis -n redis -- redis-cli monitorCan be seen from the figure, all getrequests are sent to the Replica Envoy node.
Redis traffic mirroring
Envoy Redis Proxy supports traffic mirroring, that is, requests sent by the client are simultaneously sent to a mirrored Redis server/cluster. Traffic mirroring is a very useful function. We can use traffic mirroring to import online data from the production environment into the test environment to use the online data to simulate the application as realistically as possible without affecting the line. Normal use by users on the Internet.
We create a single-node Redis node to be used as a mirror server:
$ kubectl apply -f k8s/redis-mirror.yaml -n redis
deployment.apps/redis-mirror created
service/redis-mirror createdUse EnvoFilter to enable the mirroring strategy:
$ sed -i .bak "s/\${REDIS_VIP}/`kubectl get svc redis-cluster -n redis -o=jsonpath='{.spec.clusterIP}'`/" istio/envoyfilter-redis-proxy-with-mirror.yaml
$ kubectl apply -f istio/envoyfilter-redis-proxy-with-mirror.yaml
envoyfilter.networking.istio.io/add-redis-proxy configuredBy initiating a series of key client is "b" of getand setactions:
$ kubectl exec -it `kubectl get pod -l app=redis-client -n redis -o jsonpath="{.items[0].metadata.name}"` -c redis-client -n redis -- redis-cli -h redis-cluster
redis-cluster:6379> get b
"b"
redis-cluster:6379> get b
"b"
redis-cluster:6379> get b
"b"
redis-cluster:6379> set b bb
OK
redis-cluster:6379> get b
"bb"
redis-cluster:6379> set b bbb
OK
redis-cluster:6379> get b
"bbb"
redis-cluster:6379> get b
"bbb"By command redis-cli monitorview Command Master, Replica and mirrored node received respectively.
Master node:
$ kubectl exec redis-cluster-0 -c redis -n redis -- redis-cli monitorSlave node:
$ kubectl exec redis-cluster-4 -c redis -n redis -- redis-cli monitorMirror node:
$ kubectl exec -it `kubectl get pod -l app=redis-mirror -n redis -o jsonpath="{.items[0].metadata.name}"` -c redis-mirror -n redis -- redis-cli monitorCan be seen from the figure, all setrequests are sent to the Envoy a mirror node.
Implementation principle
In the above steps, we created two EnvoyFilter configuration objects in Istio. These two EnvoyFilter modify the configuration of the Envoy proxy, which mainly includes two parts: Redis Proxy Network Filter configuration and Redis Cluster configuration.
The EnvoyFilter below replaces the TCP Proxy Network Filter in the Listener created by Pilot for Redis Service, and replaces it with a "type.googleapis.com/envoy.config.filter.network.redis_proxy.v2.RedisProxy" type Network Filter. The default route of this Redis Proxy points to "custom-redis-cluster", and the read-write separation strategy and traffic mirroring strategy are configured.
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: add-redis-proxy
namespace: istio-system
spec:
configPatches:
- applyTo: NETWORK_FILTER
match:
listener:
name: ${REDIS_VIP}_6379 # Replace REDIS_VIP with the cluster IP of "redis-cluster service
filterChain:
filter:
name: "envoy.filters.network.tcp_proxy"
patch:
operation: REPLACE
value:
name: envoy.redis_proxy
typed_config:
"@type": type.googleapis.com/envoy.config.filter.network.redis_proxy.v2.RedisProxy
stat_prefix: redis_stats
prefix_routes:
catch_all_route:
request_mirror_policy: # Send requests to the mirror cluster
- cluster: outbound|6379||redis-mirror.redis.svc.cluster.local
exclude_read_commands: True # Mirror write commands only:
cluster: custom-redis-cluster
settings:
op_timeout: 5s
enable_redirection: true
enable_command_stats: true
read_policy: REPLICA # Send read requests to replicaThe following EnvoyFilter creates a "envoy.clusters.redis" type Cluster in the CDS issued by Pilot: "custom-redis-cluster", the Cluster will use the CLUSTER SLOTS command to query a random node in the Redis cluster for the cluster The topology structure and save the topology structure locally to distribute the requests from the client to the correct Redis nodes in the cluster.
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: custom-redis-cluster
namespace: istio-system
spec:
configPatches:
- applyTo: CLUSTER
patch:
operation: INSERT_FIRST
value:
name: "custom-redis-cluster"
connect_timeout: 0.5s
lb_policy: CLUSTER_PROVIDED
load_assignment:
cluster_name: custom-redis-cluster
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address:
address: redis-cluster-0.redis-cluster.redis.svc.cluster.local
port_value: 6379
- endpoint:
address:
socket_address:
address: redis-cluster-1.redis-cluster.redis.svc.cluster.local
port_value: 6379
- endpoint:
address:
socket_address:
address: redis-cluster-2.redis-cluster.redis.svc.cluster.local
port_value: 6379
- endpoint:
address:
socket_address:
address: redis-cluster-3.redis-cluster.redis.svc.cluster.local
port_value: 6379
- endpoint:
address:
socket_address:
address: redis-cluster-4.redis-cluster.redis.svc.cluster.local
port_value: 6379
- endpoint:
address:
socket_address:
address: redis-cluster-5.redis-cluster.redis.svc.cluster.local
port_value: 6379
cluster_type:
name: envoy.clusters.redis
typed_config:
"@type": type.googleapis.com/google.protobuf.Struct
value:
cluster_refresh_rate: 5s
cluster_refresh_timeout: 3s
redirect_refresh_interval: 5s
redirect_refresh_threshold: 5summary
This article describes how to use Envoy to provide client-agnostic Redis data sharding for microservice applications, and how to use Istio to uniformly manage the Redis Cluster configuration of multiple Envoy agents in the system. We can see that the use of Istio and Envoy can greatly simplify the coding and configuration of the client using Redis Cluster, and the operation and maintenance strategy of Redis Cluster can be modified online to achieve advanced traffic management such as read-write separation and traffic mirroring. Of course, the introduction of Istio and Envoy did not reduce the complexity of the entire system, but concentrated the maintenance of Redis Cluster from the scattered application code to the service grid infrastructure layer. Corresponding to the majority of application developers, its business value mainly comes from application code, and it is not cost-effective to invest a lot of energy into such infrastructure. It is recommended to directly adopt the cloud-native Service Mesh service TCM (Tencent Cloud Mesh) on Tencent Cloud to quickly introduce the traffic management and service governance capabilities of Service Mesh for microservice applications without having to pay attention to the installation, maintenance, and upgrade of the Service Mesh infrastructure itself. And other matters.