All roads lead to Rome series - use Hiredis-cluster to connect Amazon ElastiCache for Redis cluster

foreword

Amazon ElastiCache for Redis is an ultra-fast in-memory data store that delivers sub-millisecond latency to support real-time applications. ElastiCache for Redis is built on open source Redis, is compatible with the Redis API, works with Redis clients, and uses the open Redis data format to store data. ElastiCache for Redis combines the speed, simplicity, and versatility of open source Redis with the manageability, security, and scalability of Amazon for gaming, ad tech, e-commerce, healthcare, financial services, and the Internet of Things Realm supports the most demanding real-time applications.

The Amazon cloud technology developer community provides developers with global development technology resources. There are technical documents, development cases, technical columns, training videos, activities and competitions, etc. Help Chinese developers connect with the world's most cutting-edge technologies, ideas, and projects, and recommend outstanding Chinese developers or technologies to the global cloud community. If you haven't paid attention/favorite yet, please don't rush over when you see this, click here to make it your technical treasure house!

With ElastiCache for Redis, customers can create and run managed Redis clusters with multiple partitions. The Redis environment must be extended when encountering the following three main scenarios. First, if the total memory size of Redis data exceeds or is expected to exceed the memory capacity of a single virtual machine. Second, if the application writes data to Redis the write throughput exceeds the capacity of a single virtual machine. Third, if the data is to be distributed across multiple partitions, so that any issues encountered by a single node have less impact on the overall Redis environment.

We have launched a series of blogs showing how to use different clients supporting ElastiCache cluster mode to connect and operate ElastiCache clusters in different languages. A previous blog introduced "Using redission to connect Amazon ElastiCache for redis clusters", Today's topic is to discuss how to use C language to connect Elasticache for redis cluster. Check the existing clients on the redis official website. The C language supports redis cluster mainly including Hiredis-cluster, hiredis-vip, etc. The C language supports redis cluster mainly includes redis-plus-plus, xredis, etc. This blog will mainly introduce the use of Hiredis-cluster.

2. Build a test environment

Under this explanation, the test was carried out in Ningxia, and the main contents of the test include:

Verify read, write and Failover in the ElastiCache cluster with TLS and Auth enabled
Verify read and write and Failover in the ElastiCache cluster that does not enable TLS and Auth, because for clusters that do not enable TLS and Auth, the client-side code implementation will be different.

2.1. Create a test environment

Create an ElastiCache cluster

First create a 3 shards and 6-node ElastiCache cluster with TLS and Auth enabled, the instance type is cache.t3.small, the version is 5.0.6, and the configuration endpoint is obtained in the following manner. Here, it is assumed that its value is:

The specific way to obtain the endpoint is as follows:

Find the configuration endpoint for a Redis (cluster mode enabled) cluster

Log in to the Amazon Management Console and open the ElastiCache console ( https://console.aws.amazon.com/elasticache/ ).
From the navigation pane, choose Redis.
A list is displayed with clusters running any version of Redis.
From the list of clusters, select the checkbox to the left of the cluster running "Cluster Redis".
The screen expands to show details about the selected cluster.
Find the Configuration endpoint

2. Create a 3 shards and 6-node ElastiCache cluster with TLS turned off and no auth, the instance type is cache.t3.medium, the version is 5.0.6, and the configuration endpoint is obtained according to the above method. Here, it is assumed that its value is:

Create an EC2 test client

1. Create an EC2 in the same VPC and configure the corresponding security group. The EC2 I use is Ubuntu 18.04, and the installation environment is some Ubuntu reference commands.

2. Prepare the environment on EC2. Hiredis-cluster is written based on the hiredis extension. Our environment preparation will also involve the hiredis part.

C Compiler installation, here I use GCC, you can refer to this How to install GCC Compiler on Ubuntu 18.04
libssl-dev installation, sudo apt-get install libssl-dev pay attention to install before CMake, otherwise it will report an error "Could NOT find OpenSSL, try to set the path to OpenSSL root folder in the system variable OPENSSL_ROOT_DIR (missing: OPENSSL_CRYPTO_LIBRARY OPENSSL_INCLUDE_DIR)"
CMake installation, I use CMake to compile, CMake official website provides the installation method Installing | CMake
libevent-dev installation, sudo apt install libevent-dev, direct installation may report the following error, the solution is

CMake Error at /usr/local/share/cmake-3.23/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
 Could NOT find PkgConfig (missing: PKG_CONFIG_EXECUTABLE)
Call Stack (most recent call first):
 /usr/local/share/cmake-3.23/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
 /usr/local/share/cmake-3.23/Modules/FindPkgConfig.cmake:99 (find_package_handle_standard_args)
 tests/CMakeLists.txt:40 (find_package)

Download and compile hiredis, because hiredis-cluster is written based on hiredis extension, so you need to install and compile hiredis. The following command, because SSL will be used, so I use make USE_SSL=1, you can directly use make instead.

git clone https://github.com/redis/hiredis.git
cd hiredis
make USE_SSL=1
sudo make install

The hiredis-cluster environment is prepared, because ssl will be tested, so it is specified by the option, Redis

mkdir build; cd build
cmake -DCMAKE_BUILD_TYPE=RelWithDebInfo -DENABLE_SSL=ON ..
make

3. After the above steps are completed, your environment will look like this:

related to hiredis, the hiredis_ssl lib file will be generated in </usr/local/lib/> and the hiredis.a file in </usr/local/lib/>

, the hiredis_cluster lib file is generated

2.2 ElastiCache cluster test without in-transit encryption (TLS) and auth enabled

load test data

Use redis-cli to log in to the ElastiCache cluster that has not enabled TLS and auth, and use the cluster nodes command to obtain the distribution of slots in shards. For the specific method of connecting to the cluster through the redis-cli tool and performing operations, see Connecting to a Redis Cluster Using Redis- cli .

redis-cli -c -h -p

Generate testka, testb, and testc three keys respectively, and through their slot values combined with the above slot distribution information on each shard, it can be judged that the three keys are just distributed on shard1, shard2, and shard3

C code

1. The following is the code written to the ElastiCache cluster, which is written for the three keys test1, test2, and test3 on the three shards of Elasticache. Please note that the configured address is the configuration endpoint of the ElastiCache cluster, and modify the different keynames.

#include "hircluster.h"
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

int main(int argc, char **argv) {
    UNUSED(argc);
    UNUSED(argv);
    struct timeval timeout = {1, 500000}; // 1.5s

    redisClusterContext *cc = redisClusterContextInit();
    redisClusterSetOptionAddNodes(cc, "<No TLS and auth ElastiCache configuration endpoint>:<port>");    
    redisClusterSetOptionConnectTimeout(cc, timeout);
    redisClusterSetOptionRouteUseSlots(cc);
    redisClusterConnect2(cc);
    if (cc && cc->err) {
        printf("Error: %s\n", cc->errstr);
        // handle error
        exit(-1);
    }

    int count = 0;
    while(count < 10000){
        redisReply *reply = (redisReply *)redisClusterCommand(cc, "SET %s %d", "test1", count);
        printf("SET: %d\n", count);
        count++ ;
        if (cc && cc->err) {
                printf("Error: %s\n", cc->errstr);
        }
        freeReplyObject(reply);
        sleep(1);
    }


    redisClusterFree(cc);
    return 0;
}

2. The following is the code to read the ElastiCache cluster. It reads the three keys test1, test2, and test3 on the three shards of Elasticache. Please note that the configured address is the configuration endpoint of the ElastiCache cluster, and modify the different keynames.

#include "hircluster.h"
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

int main(int argc, char **argv) {
    UNUSED(argc);
    UNUSED(argv);
    struct timeval timeout = {1, 500000}; // 1.5s

    redisClusterContext *cc = redisClusterContextInit();
    redisClusterSetOptionAddNodes(cc, "<No TLS and auth ElastiCache configuration endpoint>:<port>");    
    redisClusterSetOptionConnectTimeout(cc, timeout);
    redisClusterSetOptionRouteUseSlots(cc);
    redisClusterConnect2(cc);
    if (cc && cc->err) {
        printf("Error: %s\n", cc->errstr);
        // handle error
        exit(-1);
    }

    int count = 0;
    while(count < 10000){
        redisReply *reply = (redisReply *)redisClusterCommand(cc, "GET %s", "test1");
        if(reply && reply->str) {
                printf("GET: %s\n", reply->str);
        
        }
        freeReplyObject(reply);
    if (cc && cc->err) {
                printf("Error: %s\n", cc->errstr);
        }
        count++;
        sleep(1);
    }

    redisClusterFree(cc);
    return 0;
}

Basic Functional Test

1. Use the following command to compile the code:

gcc hiredisc-naw1.c -o hiredisc-naw1 -L./build -lhiredis_cluster /usr/local/lib/libhiredis.a -L~/hiredis -lhiredis_ssl

2. Run the program and monitor the CurrConnections indicators of ElastiCache without TLS and Auth clusters. You can see that the load is evenly sent to 6 nodes, that is, hiredis-cluster can complete the read-write separation and load balancing work:

My code is under ~/hiredis-cluster path, so

cd hiredis-cluster/ &&./hiredisc-naw1
cd hiredis-cluster/ &&./hiredisc-naw2
cd hiredis-cluster/ &&./hiredisc-naw3
cd hiredis-cluster/ &&./hiredisc-nar1
cd hiredis-cluster/ &&./hiredisc-nar2
cd hiredis-cluster/ &&./hiredisc-nar3

2.3. ElastiCache cluster test with encryption in transit (TLS) and auth enabled

Amazon ElastiCache in-transit encryption is an optional feature that allows you to increase the security of your data at its most vulnerable point (while in transit from one location to another). Enabling encryption in transit has some performance impact due to the processing required to encrypt and decrypt data at the endpoint. Benchmarking should be done with and without data encrypted in transit to determine the performance impact on the use case. For details about TLS encryption, please refer to All Roads Lead to Rome - Using redission to connect to Amazon ElastiCache for redis cluster.

load test data

1. Use redis-cli to log in to the ElastiCache cluster with TLS and auth enabled and use the cluster nodes command to obtain the distribution of slots in shards. For the specific method of connecting to the cluster through the redis-cli tool and operating it, see Connecting to Redis using redis-cli cluster.

redis-cli -c -h -p

2. Generate testka, testb, and testc three keys respectively. You can judge that the three keys are just distributed on shard1, shard2, and shard3 by combining their slot values with the slot distribution information on each shard above.

C code

1. The following is the code written to the ElastiCache cluster. It is written for the three keys test1, test2, and test3 on the three shards of Elasticache. Please note that the configured address is the configuration endpoint of the ElastiCache cluster, set the password, and modify the different keyname. It should be noted here that hiredis-cluster supports SSL in two ways, one is to specify the SSL certificate path, and the other is to specify the certificate. For the case of EC2 accessing ElastiCache, just specify the certificate path. Refer to the documentation of hiredis .

#include "hiredis_cluster/hircluster.h"

#include "hiredis/hiredis_ssl.h"
#include <stdio.h>
#include <stdlib.h>

#define CLUSTER_NODE_TLS ":<Enable TLS and auth ElastiCache configuration endpoint>:<port>"
#define CLUSTER_PASSWORD "<PASSWORD>"

int main(int argc, char **argv) {
    UNUSED(argc);
    UNUSED(argv);

    redisSSLContext *ssl;
    redisSSLContextError ssl_error;

    redisInitOpenSSL();
    ssl = redisCreateSSLContext(NULL, "/etc/ssl/certs", NULL, NULL,
                                NULL, &ssl_error);
    if (!ssl) {
        printf("SSL Context error: %s\n", redisSSLContextGetError(ssl_error));
        exit(1);
    }

    struct timeval timeout = {5, 500000}; // 5.5s

    redisClusterContext *cc = redisClusterContextInit();
    redisClusterSetOptionAddNodes(cc, CLUSTER_NODE_TLS);
    redisClusterSetOptionConnectTimeout(cc, timeout);
    redisClusterSetOptionRouteUseSlots(cc);
    redisClusterSetOptionEnableSSL(cc, ssl);
    redisClusterSetOptionPassword(cc, CLUSTER_PASSWORD);
    redisClusterConnect2(cc);
    if (cc && cc->err) {
        printf("Error: %s\n", cc->errstr);
        // handle error
        exit(-1);
    }

    int count = 0;
    while(count < 10000){
        redisReply *reply = (redisReply *)redisClusterCommand(cc, "SET %s %d", "test1", count);
        printf("SET: %d\n", count);
        if (cc && cc->err) {
                printf("Error: %s\n", cc->errstr);
        }
    count++ ;
        freeReplyObject(reply);
        sleep(1);
    }

    redisClusterFree(cc);
    redisFreeSSLContext(ssl);
    return 0;
}

2. The following is the code to read the ElastiCache cluster. It reads the three keys test1, test2 and test3 on the three shards of Elasticache. Please note that the configured address is the configuration endpoint of the ElastiCache cluster, and modify the different keynames.

#include "hircluster.h"
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>

int main(int argc, char **argv) {
    UNUSED(argc);
    UNUSED(argv);
    struct timeval timeout = {1, 500000}; // 1.5s

    redisClusterContext *cc = redisClusterContextInit();
    redisClusterSetOptionAddNodes(cc, "<No TLS and auth ElastiCache configuration endpoint>:<port>");    
    redisClusterSetOptionConnectTimeout(cc, timeout);
    redisClusterSetOptionRouteUseSlots(cc);
    redisClusterConnect2(cc);
    if (cc && cc->err) {
        printf("Error: %s\n", cc->errstr);
        // handle error
        exit(-1);
    }

    int count = 0;
    while(count < 10000){
        redisReply *reply = (redisReply *)redisClusterCommand(cc, "GET %s", "test1");
        if(reply && reply->str) {
                printf("GET: %s\n", reply->str);
        
        }
        freeReplyObject(reply);
    if (cc && cc->err) {
                printf("Error: %s\n", cc->errstr);
        }
        count++;
        sleep(1);
    }

    redisClusterFree(cc);
    return 0;
}

Basic Functional Test

1. Use the following command to compile the code:

gcc hiredisc-naw1.c -o hiredisc-naw1 -L./build -lhiredis_cluster /usr/local/lib/libhiredis.a -L~/hiredis -lhiredis_ssl

My code is under ~/hiredis-cluster path, so

cd hiredis-cluster/ &&./hiredisc-naw1
cd hiredis-cluster/ &&./hiredisc-naw2
cd hiredis-cluster/ &&./hiredisc-naw3
cd hiredis-cluster/ &&./hiredisc-nar1
cd hiredis-cluster/ &&./hiredisc-nar2
cd hiredis-cluster/ &&./hiredisc-nar3

2.3. ElastiCache cluster test with encryption in transit (TLS) and auth enabled

load test data

redis-cli -c -h -p

C code

#include "hiredis_cluster/hircluster.h"

#include "hiredis/hiredis_ssl.h"
#include <stdio.h>
#include <stdlib.h>

#define CLUSTER_NODE_TLS ":<Enable TLS and auth ElastiCache configuration endpoint>:<port>"
#define CLUSTER_PASSWORD "<PASSWORD>"

int main(int argc, char **argv) {
    UNUSED(argc);
    UNUSED(argv);

    redisSSLContext *ssl;
    redisSSLContextError ssl_error;

    redisInitOpenSSL();
    ssl = redisCreateSSLContext(NULL, "/etc/ssl/certs", NULL, NULL,
                                NULL, &ssl_error);
    if (!ssl) {
        printf("SSL Context error: %s\n", redisSSLContextGetError(ssl_error));
        exit(1);
    }

    struct timeval timeout = {5, 500000}; // 5.5s

    redisClusterContext *cc = redisClusterContextInit();
    redisClusterSetOptionAddNodes(cc, CLUSTER_NODE_TLS);
    redisClusterSetOptionConnectTimeout(cc, timeout);
    redisClusterSetOptionRouteUseSlots(cc);
    redisClusterSetOptionEnableSSL(cc, ssl);
    redisClusterSetOptionPassword(cc, CLUSTER_PASSWORD);
    redisClusterConnect2(cc);
    if (cc && cc->err) {
        printf("Error: %s\n", cc->errstr);
        // handle error
        exit(-1);
    }

    int count = 0;
    while(count < 10000){
        redisReply *reply = (redisReply *)redisClusterCommand(cc, "SET %s %d", "test1", count);
        printf("SET: %d\n", count);
        if (cc && cc->err) {
                printf("Error: %s\n", cc->errstr);
        }
    count++ ;
        freeReplyObject(reply);
        sleep(1);
    }

    redisClusterFree(cc);
    redisFreeSSLContext(ssl);
    return 0;
}

#include "hiredis_cluster/hircluster.h"

#include "hiredis/hiredis_ssl.h"
#include <stdio.h>
#include <stdlib.h>

#define CLUSTER_NODE_TLS "<Enable TLS and auth ElastiCache configuration endpoint>:<port>"
#define CLUSTER_PASSWORD "<PASSWORD>"

int main(int argc, char **argv) {
    UNUSED(argc);
    UNUSED(argv);

    redisSSLContext *ssl;
    redisSSLContextError ssl_error;

    redisInitOpenSSL();
    ssl = redisCreateSSLContext(NULL, "/etc/ssl/certs", NULL, NULL,
                                NULL, &ssl_error);
    if (!ssl) {
        printf("SSL Context error: %s\n", redisSSLContextGetError(ssl_error));
        exit(1);
    }

    struct timeval timeout = {5, 500000}; // 5.5s

    redisClusterContext *cc = redisClusterContextInit();
    redisClusterSetOptionAddNodes(cc, CLUSTER_NODE_TLS);
    redisClusterSetOptionConnectTimeout(cc, timeout);
    redisClusterSetOptionRouteUseSlots(cc);
    redisClusterSetOptionEnableSSL(cc, ssl);
    redisClusterSetOptionPassword(cc, CLUSTER_PASSWORD);
    redisClusterConnect2(cc);
    if (cc && cc->err) {
        printf("Error: %s\n", cc->errstr);
        // handle error
        exit(-1);
    }

    int count = 0;
    while(count < 10000){
        redisReply *reply = (redisReply *)redisClusterCommand(cc, "GET %s", "test1");
        if(reply && reply->str) {
        printf("GET: %s\n", reply->str);
        
    }
    freeReplyObject(reply);
if (cc && cc->err) {
                printf("Error: %s\n", cc->errstr);
        }
    count++;
        sleep(1);
    }

    redisClusterFree(cc);
    redisFreeSSLContext(ssl);
    return 0;
}

Basic Functional Test

1. Use the following command to compile the code:

gcc hiredisc-aw1.c -o hiredisc-aw1 -L./build -lhiredis_cluster /usr/local/lib/libhiredis.a -L~/hiredis -lhiredis_ssl

2. Run the program and monitor the CurrConnections indicators of ElastiCache to enable TLS and Auth clusters. You can see that the load is sent to 6 nodes in a balanced manner, that is, hiredis-cluster can complete the work of read-write separation and load balancing:

My code is under ~/hiredis-cluster path, so

cd hiredis-cluster/ &&./hiredisc-aw1
cd hiredis-cluster/ &&./hiredisc-aw2
cd hiredis-cluster/ &&./hiredisc-aw3
cd hiredis-cluster/ &&./hiredisc-ar1
cd hiredis-cluster/ &&./hiredisc-ar2
cd hiredis-cluster/ &&./hiredisc-ar3

Level 3 heading

3.1. ElastiCache failover test without TLS and auth enabled

Log in to the console, perform failover on Shard1, and then observe the output of our C program. This process lasts about 12s ((observed)),

It can also be seen from the screenshot of the CurrConnections indicator of each node in the ElastiCache cluster that hiredis-cluster sends all read and write requests to the new primary node of shard1:

3.2. ElastiCache failover test with TLS and auth enabled

Log in to the console, perform failover on Shard1, and then observe the output of our C program. This time is about 10s (observed),

It can also be seen from the screenshot of the CurrConnections indicator of each node in the ElastiCache cluster that hiredis-cluster sends all read and write requests to the new master node of shard1:

Problems and solutions in failover tests

When doing failover, I found a problem. During the failover process, sometimes the client directly disconnects the link, and sometimes it succeeds. Later, after analyzing the code of hiredis, I learned that the timeout time needs to be adjusted, so it is adjusted from 1.5s to 5.5s, note that this value is not the best practice, it is based on tests.

4 Conclusion

In this blog, the C language is mainly used to connect and operate the ElastiCache cluster through hiredis-cluster. From the simple demo, we can see that hiredis-cluster can well support the ElastiCache cluster to open TLS and auth functions, and automatically complete reading and writing Separation, load balancing, failover and other work help us to use ElastiCache conveniently and efficiently.

5. Reference

Hiredis-cluster 文档：GitHub - Nordix/hiredis-cluster: C client library for Redis Cluster. This project is used and sponsored by Ericsson. It is a fork of the now unmaintained hiredis-vip.

Hiredis Documentation: GitHub - redis/hiredis: Minimalistic C client for Redis >= 1.2

Amazon Elasticache ： Amazon ElastiCache for Redis

All roads lead to Rome - use redisson to connect Amazon ElastiCache for redis cluster: All roads lead to Rome - use redisson to connect Amazon ElastiCache for redis cluster | Amazon AWS official blog

All roads lead to Rome - use redis-py to access Amazon ElastiCache for redis cluster

All roads lead to Rome - use go-redis to connect Amazon ElastiCache for Redis cluster

The author of this article

Feng Qiushuang

Amazon solution architect, responsible for multinational enterprise-level customers' technology architecture design, consulting and design optimization based on Amazon. Before joining Amazon, he worked in IT companies such as IBM and Oracle, and accumulated rich practical experience in program development and database.

Article source: https://dev.amazoncloud.cn/column/article/630a21a42ecbae73705ffb40?sc_medium=regulartraffic&sc_campaign=crossplatform&sc_channel=CSDN