In this article I lecture in June 23 to participate in the Shenzhen GIAC technology conference transcript.
Ladies and gentlemen, I am a palm reading engineer Qian products, Nuggets brochure "Redis depth Adventure" from the author. Today I bring that share the theme: Redis optimization practice in vast amounts of data and high concurrency. Redis for engineers engaged in Internet technology is not new, almost all medium and large enterprises are using Redis as a cache database, but for the vast majority of businesses will only use its most basic KV caching feature, there are many Redis advanced features may have never seriously practiced. Today, in this one hour I will be around Redis, sharing nine classic case encountered in the usual daily business development, we hope that through this sharing can help you better apply advanced features Redis to everyday business development in the past.
The total amount of users palm reading e-book reading software ireader is probably about 500 million monthly living 5kw, Nikkatsu nearly 2kw. 1000 Redis a plurality of server instances, the cluster 100 + memory 20g in each instance the following controls.
KV Cache
The first is the most basic and most commonly used is the KV function, we can use Redis to cache user information, session information, product information and so on. The following code is common buffer read logic.
def get_user(user_id):
user = redis.get(user_id)
if not user:
user = db.get(user_id)
redis.setex(user_id, ttl, user) // 设置缓存过期时间
return user
def save_user(user):
redis.setex(user.id, ttl, user) // 设置缓存过期时间
db.save_async(user) // 异步写数据库
复制代码The expiration time is very important, and it is usually proportional to the length of a single user session, try to ensure that users have the cache and data can be used within a single session. Of course, if your company's financial strength, but also pay attention to the ultimate performance experience, you can even set a long time points simply do not set an expiration time. When the growing amount of data, use Codis or Redis-Cluster to cluster expansion.
In addition Redis also provides cache mode, Set instruction does not have to set the expiration time, it also can use these key-value pairs according to certain policies eliminated. Open instruction cache mode is: config set maxmemory 20gb, so that when memory reaches 20gb, Redis will begin phase-out strategy to the new key-value pairs to make room. This strategy also provides a variety of Redis, summed up the strategy is divided into two: the elimination algorithm delineated out of range of options. For example, our strategy is to use online allkeys-lru. This means that all the key allkeys are likely to be eliminated internal Redis, with or without a belt expiration date, and volatile eliminated only with an expiration time. Redis elimination function like when companies need to tighten their belts to meet the economic winter need to be a brutal winter optimize the talent. It will choose optimized only temporary, or is it the equality of all people could be optimized. When this range delineated, will be selected from a number of places, how to choose it, this is the elimination algorithm. The most commonly used is the LRU algorithm, it has a weakness, that is superficial people who do well can escape optimization. For example, you took the opportunity to rush in front of the boss a good look at the performance, then you'll be safe. So to Redis 4.0 which introduces LFU algorithm, the results are to be of the usual assessment, only superficial it was not enough, you usually depends on the ground are not diligent. Finally, a very unusual method - a random wave number algorithm which is likely to be put out of the CEO, it is generally not use it.
Distributed Lock
Here we see a second function - distributed lock, this is in addition to the KV caching the most commonly used features another. For example, a very competent senior engineer, development efficiency quickly, code quality is also high, is the star of the team. So many product manager should bother him, so he needs to do for himself. If the same time to a bunch of product managers are looking for him, it's the idea of it will fall into chaos, then good programmer, brain concurrent capacity is not much better. So he was in the door of his office hung a Do Not Disturb sign when a product manager to the janitor to look at this brand on there, if it can not find engineers come in to talk about needs, before talking about should sign hanging up, then talk over the brand plucked. Other such product manager should bother him, if you see this sign hanging in there, you can choose to wait or go to sleep busy with other things. The case of star engineers have won the peace.
The use of distributed lock is very simple, it is to use the Set command parameters are as follows extensions
# 加锁
set lock:$user_id owner_id nx ex=5
# 释放锁
if redis.call("get", KEYS[1]) == ARGV[1] then
return redis.call("del", KEYS[1])
else
return 0
end
# 等价于
del_if_equals lock:$user_id owner_id
复制代码Be sure to set the expiration time, because of special circumstances - such as earthquakes (the process is kill -9, or machine downtime), product manager may choose to jump from the window, no chance delisted, resulting in a deadlock hunger, let the good engineers became a big idlers, causing serious waste of resources. Also note that owner_id, it represents who added the lock - Product Manager job number. In case you accidentally lock someone else take off. To match this owner_id when the lock is released, in order to match the success of the release of the lock. This owner_id is usually a random number, stored in the ThreadLocal variable in (a stack variable).
In fact, the official does not recommend this approach, because it can cause problems lock lost in clustered mode - when the main switch from the place in. Official recommendation distributed lock called RedLock, authors believe that this algorithm is more secure, we recommend use. However, palm reading here has been used above the simplest distributed lock, why we do not use RedLock it, because it's operation and maintenance costs will be higher, it requires a separate instance Redis three or more, with some of them to be cumbersome. In addition it Redis cluster occurrence probability from the main switch is not high, even if the probability of occurrence of emergence from the main switch lock lost is very low, because the Lord has often switched from a process, the process time often exceed the lock expired time, the lock will not occur abnormal loss. There is the opportunity it distributed lock lock conflicts encountered not many companies which, as a star programmer is relatively limited, as always lock queue that met the need to optimize the structure.
Latency Queuing
Here we continue to look at the third function - delay queue. We mentioned earlier can choose a variety of strategies when product manager in the face of "Do Not Disturb" sign, a dry bed waiting 2. 2. 3. surrendering rest a stay for another. Dry is waiting spinlock, which will burn CPU, soared high Redis of QPS. Sleep is to sleep for a while and try again, this wastes thread resources, will increase the long response. Abandon not do it is to inform the user front end will try to be, and now the system pressure a bit busy, affecting the user experience. The last one do now is talk about strategy - will come to be, it is in the real world the most common strategy. This strategy is generally used in consumer message queue, this lock conflicts when it came to how to do? Does not abandon the process is not suitable for immediate retries (the spinlock), this time delay can be thrown into the message queue, the processing again after a while.
There are many professional messaging middleware supports latency messaging, such as RabbitMQ and NSQ. Redis can, we can use zset to achieve this delay queue. zset value is stored inside / value pairs score, we will be stored as a message value task serialized task message, the score is stored as a message task run time (DEADLINE), then the score is greater than the polling zset now to deal with.
# 生产延时消息
zadd(queue-key, now_ts+5, task_json)
# 消费延时消息
while True:
task_json = zrevrangebyscore(queue-key, now_ts, 0, 0, 1)
if task_json:
grabbed_ok = zrem(queue-key, task_json)
if grabbed_ok:
process_task(task_json)
else:
sleep(1000) // 歇 1s
复制代码When consumers are multi-threaded or multi-process, where there will be competition waste. Obviously the current thread will task_json poll out from the zset, but to compete by zrem Shique not grab the hand. Then you can use the LUA script to solve this problem, the polling and the competition for atomic operations, so you can avoid wasting competition.
local res = nil
local tasks = redis.pcall("zrevrangebyscore", KEYS[1], ARGV[1], 0, "LIMIT", 0, 1)
if #tasks > 0 then
local ok = redis.pcall("zrem", KEYS[1], tasks[1])
if ok > 0 then
res = tasks[1]
end
end
return res
复制代码Why do I want to talk about distributed queue and delay lock with it, because early on a fault line out. Redis burst length of a queue table line when a fault occurs, leading to the failure to implement many of asynchronous tasks, business data is a problem. Later find out the reason is because there is no good use of distributed lock has led to a deadlock, and the face of the lock fails to sleep infinite retries result has led to the asynchronous task completely into the sleep state can not handle the task. That was how this distributed lock use it? Use is setnx + expire, the results at the time of the service upgrade process stops directly led to the implementation of individual requests setnx, but expire not been implemented, so he brought individual users deadlock. But then there is another asynchronous background task processing, but also the need for the user to lock, lock failure will retry infinite sleep, so once users hit the front of the deadlock, the asynchronous thread completely stalled. Since the accident we have the correct form of today's distributed lock and queue delay invention, there is an elegant stop, because if there is a logical shutdown elegant, the service upgrade will not only lead to a request to perform a half It was interrupted, unless the process is kill -9 or downtime.
Regular tasks
Distributed regular tasks implemented in many ways, the most common one is the master-workers model. master is responsible for managing time, the point of the mission will still message to messaging middleware, and then worker who is responsible for monitoring these message queues to consume messages. The famous Python regular tasks framework Celery is so dry. But Celery has a problem that master is the single point, if the master hung up, the timing of the entire mission system stops working.
Another implementation is multi-master model. What does that mean this model, similar to Java inside the Quartz framework, the database locks to control concurrent tasks. There will be multiple processes, each process will manage time, time is up on the use of the database lock to compete for task execution right, grab the process will get the chance to perform the task, and then began to perform the task, this would resolve the master of single-point problem. This model has a drawback that waste can cause competition problems, but generally most of the regular tasks of business systems and not so much, so this competition is not serious waste. Another problem is that it depends on the consistency of distributed machine time, if the time on multiple machines inconsistency will cause the task to be performed multiple times, which can be alleviated by increasing the database lock time.
Now, with Redis distributed lock, then we can achieve a simple timing task framework on top of Redis.
# 注册定时任务
hset tasks name trigger_rule
# 获取定时任务列表
hgetall tasks
# 争抢任务
set lock:${name} true nx ex=5
# 任务列表变更(滚动升级)
# 轮询版本号，有变化就重加载任务列表，重新调度时间有变化的任务
set tasks_version $new_version
get tasks_version
复制代码If you feel internal codes of Quartz people to understand complex, distributed the document had little hard to toss, you can try Redis, using it makes you have lost some hair.
Life is Short，I use Redis
https://github.com/pyloque/taskino
复制代码Frequency Control
If you did know the community, always inevitably encounter spam. Home suddenly wake up you will find some inexplicable advertising posts will refresh the. Without adequate mechanisms to control the user experience will receive a serious impact.
Control your garbage stickers strategy is very much high point by AI, the easiest way is through keyword scanning. There are more commonly used way is to control the frequency, limiting the production rate of a single user content, different users have different levels of frequency control parameter.
Frequency control can be achieved using Redis, we will understand user behavior as a time series, we want to ensure that limit the length of the time series of individual users within a certain period of time, more than the behavior of this length would prohibit users. It can be used Redis of zset to achieve.
The green sector is a period of time-series information that we want to keep the gray segment will be cut off. Statistics time series in the green section of the number of records to know whether exceeds the threshold frequency.
The following code control ugc user behavior most N times per hour
hist_key = "ugc:${user_id}"
with redis.pipeline() as pipe:
# 记录当前的行为
pipe.zadd(hist_key, ts, uuid)
# 保留1小时内的行为序列
pipe.zremrangebyscore(hist_key, 0, now_ts - 3600)
# 获取这1小时内的行为数量
pipe.zcard(hist_key)
# 设置过期时间，节约内存
pipe.expire(hist_key, 3600)
# 批量执行
_, _, count, _ = pipe.exec()
return count > N
复制代码Service Discovery
Technology maturity slightly higher infrastructure services business will have discovered. Normally we would prefer to use zookeeper, etcd, consul and other distributed configuration database as a storage service list. They have a very timely notification mechanism to inform the customer service changes occurred in the list of services. How do we use Redis service discovery to do it?
Here we want to use data structures zset again, we use zset to save a single list of services. Multiple listing service on the use of multiple zset to store. zset the value of time and score address and heartbeat services are stored. Service providers need to report their own survival heartbeat, called once zadd every few seconds. Service provider when you stop the service, use zrem to remove yourself.
zadd service_key heartbeat_ts addr
zrem service_key addr
复制代码This is not enough, because there may be an abnormal termination of service, there is no opportunity to perform hook, you need to use an additional thread to clean up the expired items in the list of services
zremrangebyscore service_key 0 now_ts - 30 # 30s 都没来心跳
复制代码Then there is another important issue is how to inform consumers has been changed at the list of services, where we use the same version number of polling mechanism. When the service list of changes, incrementing the version number. Consumers polled by version number changes to reload the list of services.
if zadd() > 0 || zrem() > 0 || zremrangebyscore() > 0:
incr service_version_key
复制代码But there is a problem, if a lot of consumers rely on the list of services, then it needs a lot of polling version number, such as IO efficiency will be relatively low. Then we can add a global version number, when any service list version number is changed, the global version number is incremented. Under normal circumstances such consumers only need to poll the global version number on it. When the global version number changed and then one by one sub-version number than the reliance on the list of services, then load a list of changes to the service.
bitmap
Palm reading of the sign system to do earlier, when the number of users has not yet come up, relatively simple design, is to sign status of the user with the hash structure Redis to store, sign once recorded a in the hash structure, the sign has three states not sign, sign and has retroactive, three integer values 0,1,2 respectively.
hset sign:${user_id} 2019-01-01 1
hset sign:${user_id} 2019-01-02 1
hset sign:${user_id} 2019-01-03 2
...
复制代码This user is a waste of space, to later sign Nikkatsu tens of millions of times, Redis storage problems beginning to show a direct memory Biao to 30G +, we usually had a line of examples 20G began to alarm, 30G has a serious excessive.
This time we started to address this problem, to optimize storage. We chose to use the bitmap information recording sign, a sign-on state requires two bits to record, one month of storage requires only 8 bytes. This enables the use of a short character string records stored in the user check month.
The effect of the optimization is very clear, direct memory down to 10 G. Because the query sign state throughout the month of frequent API calls, so the traffic interface also followed a lot smaller.
However, the bitmap has a disadvantage that it is the underlying string of continuous storage space bitmap will automatically expand, such as a large 8m bits bitmap, only the last bit is 1, the other bits are zero it will take up storage space 1m, such waste is very serious.
So there will be a roaring bitmap data structure, its large bitmaps has been segmented memory, the whole bit segment zeros can not exist. Also designed sparse storage structure for each segment, if no more than 1 set of this segment, they can store only integer offsets. Such storage space bitmap is obtained a very significant compression.
The roar bitmap accurate count in the field of big data is very valuable, interested students can find out.
Fuzzy count
The aforementioned attendance systems, product manager need to know if this sign of Nikkatsu month living how to do it? Usually we throw pot directly - please find the data sector. But the data sector is not very often in real time, often the previous day's data needs to come out the next day, off-line calculation is usually once a day timing. Then how to achieve a real-time active count?
The simplest solution is to maintain a set collection Redis inside to a user, sadd about the size of a final set of numbers that we need the UV. But this is very serious waste of space, just to be a digital store such a huge collection seems to be very worth when. so what should I do now?
Then you can use Redis provided HyperLogLog fuzzy counting function, it is a probability count, a certain degree of error, the error is approximately 0.81%. However, a small footprint, which is the bottom of a bitmap, it will only take up to 12k of storage space. And the count value is small when using a sparse bitmap memory, even smaller footprint.
# 记录用户
pfadd sign_uv_${day} user_id
# 获取记录数量
pfcount sign_uv_${day}
复制代码Read the number of micro-channel public number of articles you can use it, UV web statistics it can be done. However, if the product manager extremely concerned about the accuracy of the figures, such as a statistical needs and directly linked to money, then you can consider roar bitmap mentioned earlier. It will use more complex, the need to advance the user ID of the sequence of integers. Redis does not provide native functionality roaring bitmap, but there is an open-source Redis Module can be used to ready to use.
Bloom filter
Finally, we talk about the Bloom filter, if a system is about to have a lot of influx of new users, it will be very valuable, can significantly reduce cache of penetration, reducing the pressure on the database. This influx of new users is not necessarily the business of large-scale roll system, it may be because the cache from external penetration attacks.
def get_user_state0(user_id):
state = cache.get(user_id)
if not state:
state = db.get(user_id) or {}
cache.set(user_id, state)
return state
def save_user_state0(user_id, state):
cache.set(user_id, state)
db.set_async(user_id, state)
复制代码The above example is the user query the state of the business system interface code, a new user is now over, and it will go to cache query has no such user state data, because it is a new user, it is certainly not the cache. Then it is going to check the database, the database did not result. If such an instant influx of new users in large quantities, it can be foreseen database pressures will be relatively large, there will be a lot of empty query.
We very much hope that Redis there such a set, it is stored for all users of the id, so this set by querying the set know is not new to the user. When the user very large amount of time, maintaining such a collection of required storage space is great. This time we can use the Bloom filter, which is equivalent to a set, but it is different from the set, it needs to be much smaller storage space. For example, you need to store a user id 64 bytes, and stores a Bloom filter requires only one user id byte points. But it does not save the user id, user id but fingerprints, so there will be some small probability of miscarriage of justice, it is a vessel equipped with a fuzzy filtering capabilities.
When a user id say it is not in a container, then it certainly is not. When it says user id in the container, 99% probability that it is correct, there is a 1% chance it produced a miscarriage of justice. However, in this case, this miscarriage of justice does not create problems, but the cost of misjudgment cache penetrate it, the equivalent of 1% of the new users are not protected Bloom filter penetration directly to the database query, and left 99% of new users that can be effectively blocked by the Bloom filter, to avoid penetration of the cache.
def get_user_state(user_id):
exists = bloomfilter.is_user_exists(user_id)
if not exists:
return {}
return get_user_state0(user_id)
def save_user_state(user_id, state):
bloomfilter.set_user_exists(user_id)
save_user_state0(user_id, state)
复制代码Principle Bloom filter has a good analogy, it is on the ground in the winter a snow-covered, if you come from above, it will leave your footprints. If you have your footprints on the ground, it can be concluded that a high probability that you came to this place, but not necessarily, maybe just someone else's shoes and you wear exactly the same. But if you no footprints on the ground, then you can conclude that you have never been 100% this place.