Redis memory analysis method [Shared redis cache analysis scheme for old projects]

background

In order to unify development standards in the early stage of construction, most companies or projects will improve delivery consistency by sharing redis.

It is often encountered online that users want to know the memory distribution of data in their Redis instance.

In order not to affect the use of online instances, we generally use bgsave to generate dump.rdb files, and then combine redis-rdb-tools and sqlite for static analysis.

create backup

Self-built Redis can execute bgsave on the client to generate rdb files.
The Redis version of Alibaba Cloud Database can perform data backup and download operations on the console, and the downloaded data is an rdb format file.

Features

redis-rdb-tools is a python tool for parsing rdb files. It mainly has three functions:

Generate a memory snapshot
Dump into json format
Compare two dump files using standard diff tools

After analyzing the use of memory, we mainly use its function of generating memory snapshots.

redis-rdb-tools installation

GitHub - sripathikrishnan/redis-rdb-tools: Parse Redis dump.rdb files, Analyze Memory, and Export Data to JSON

There are two installation methods for redis-rdb-tools, just choose one.
Install using PyPI

pip install rdbtools

Install from source

git clone https://github.com/sripathikrishnan/redis-rdb-tools
cd redis-rdb-tools
sudo python setup.py install

Use redis-rdb-tools to generate memory snapshots

The command to generate a memory snapshot is:

rdb -c memory dump.rdb > memory.csv

Generate a memory report in CSV format. The included columns are: database ID, data type, key, memory usage (byte), and encoding. Memory usage contains key, value and other values.
Note: The memory usage is a theoretical approximation and in general, slightly lower than the actual value.
memory.csv example:

$head memory.csv
database,type,key,size_in_bytes,encoding,num_elements,len_largest_element
0,string,"orderAt:377671748",96,string,8,8
0,string,"orderAt:413052773",96,string,8,8
0,sortedset,"Artical:Comments:7386",81740,skiplist,479,41
0,sortedset,"pay:id:18029",2443,ziplist,84,16
0,string,"orderAt:452389458",96,string,8,8

Analyzing memory snapshots

SQLite is a lightweight database. After we can import the previously generated csv into the database, we can use sql statements to conveniently perform various analyzes on the memory data of Redis.
Import method:

sqlite3 memory.db
sqlite> create table memory(database int,type varchar(128),key varchar(128),size_in_bytes int,encoding varchar(128),num_elements int,len_largest_element varchar(128));
sqlite>.mode csv memory
sqlite>.import memory.csv memory

After the data is imported, you can analyze it however you want. Here are a few simple examples:

Query the number of keys

sqlite>select count(*) from memory;

Query the total memory usage

sqlite>select sum(size_in_bytes) from memory;

Query the top 10 keys with the highest memory usage

sqlite>select * from memory order by size_in_bytes desc limit 10;

Query a list with more than 1000 members

sqlite>select * from memory where type='list' and num_elements > 1000 ;

Summarize

By using redis-rdb-tools + sqlite, it is convenient to statically analyze the memory of the redis instance. The whole process is relatively simple, after obtaining rdb

rdb -c memory dump.rdb > memory.csv;
sqlite3 memory.db
sqlite> create table memory(database int,type varchar(128),key varchar(128),size_in_bytes int,encoding varchar(128),num_elements int,len_largest_element varchar(128));
sqlite>.mode csv memory
sqlite>.import memory.csv memory

can

In actual use, I have found that a List has accumulated more than 10G of content, and I have also found more than 43M of string type values, which can not only answer users' doubts, but also help users eliminate potential risk points in the business and find out business performance. bottleneck.

In general, the whole analysis process is simple and practical, and it is a method worth mastering for every Redis user.