Redis-like mass storage pika released the official version 2.2

Infrastructure & DBA team 360 cloud computing
pika is a large-capacity Redis storage developed by the 360 WEB platform department DBA and the infrastructure team. It strives to be fully compatible with the Redis protocol and inherit the Redis convenient operation and maintenance design through persistent storage. Redis has problems in large-capacity scenarios, such as slow recovery time, high cost of master-slave synchronization, relatively fragile single thread, limited data load, high memory cost, etc.

List of improvements and bug fixes in pika2.2

Update rocksdb to 5.0.1 and plug-in it, then you can dynamically upgrade the rocksdb version
Added geo interface and hyperloglog interface (thanks to Leviathan1995 for the support)
Optimize the underlying network framework pink to solve the problem of fd overflow that may occur in the scenario of extremely slow requests and a large number of short connections.

Note: The current version of pika will block a thread when it encounters a very slow request such as keys * (similar to Redis hang). At this time, subsequent requests will always be in the queue of the worker thread, with more connections Eventually caused the process to crash with too many fd handles. After optimization, we increased the limit on the number of fd handles and optimized the thread task allocation logic. In version 2.2, even if there are slow requests, subsequent connections will be allocated to idle threads and will not appear again. This kind of problem

4. Optimize the write performance, enable parallel write at the engine layer, and the write interface performance has been improved to a certain extent, for example, the set performance is increased by 30%

5. Improved the performance of the range interface

6. The residual meta information after the expiration of multiple data structures can be recycled through the compact command (if your instance has a large number of expired multiple data structures, executing the compact command after this upgrade will free up more space for you)

7. Fix the problem of scan blocking due to bugs during the scan process

8. Fix the problem that the key that does not exist may not be read out in the range operation

9. Fixed the problem that there is a very small probability of deadlock when bgsave and info are executed at the same time

10. Virtual dbnum now has an upper limit, up to 16 (db0-db15), in order to solve the problem of some Redis management tools stuck in select because pika has no upper limit of dbnum

11. Complete support for rocksdb multi-threaded compact, improve compaction speed

12. The configuration file adds new parameters max-background-flushes, max-background-compactions, max-bytes-for-level-multiplier, please refer to the wiki for specific functions

github: https://github.com/Qihoo360/pika
(You can also click "Read the original to jump" below)
pika technical exchange group: 294254078 For
more information about pika, please click on the public account name on the upper left to follow.
Welcome to the public through HULK Number feedback pika related bugs and submit improvement points

If you still don’t know what pika is, please browse the introduction below

Pika is a large-capacity Redis storage jointly developed by the DBA of the 360 WEB Platform Department and the Infrastructure Group. It strives to solve its problems in large-capacity scenarios through persistent storage under the premise of compatibility with the Redis protocol and operation and maintenance methods, such as recovery Slow time, high cost of master-slave synchronization, relatively fragile single thread, limited data load, and high memory cost. It has the following characteristics:

1. Pika is fully compatible with the Redis protocol, so it does not have its own dedicated client, please use the Redis client to connect to pika directly, so it is almost unnecessary to modify the program code when migrating from Redis to pika

2. Pika is compatible with most of the Redis interfaces (above 80%). Please refer to the wiki for the specific interface compatibility list. The use of individual interfaces is slightly different. In addition, we have been continuously increasing the number of interface compatibility.

3. Pika is a persistent storage, no additional settings are required for the data to be automatically dropped, and there is no need to worry about what to do if the data is suddenly cut

4. Pika supports multi-threading and optimizes the allocation of worker threads. In extreme scenarios, pika has a higher survival rate than Redis. For example, a key command that allows Redis to completely hang in the 12-threaded pika you need to send at least continuously 12 keys can achieve the same hanging effect, and the number of pika threads can be set according to your preferences

5. Pika is cheap, it occupies very little memory resources and has its own compression function. In the actual test, the Redis data imported into pika has a compression ratio of 3 to 8 times, that is, only 80G of Redis memory data is imported into pika. 10~30G disk data is very economical, and pika supports multiple compression algorithms for you to choose

6. Pika inherits the convenient operation and maintenance design of Redis. If you have Redis operation and maintenance experience, pika can easily get started. In addition, we have made some improvements to some operation and maintenance related functions, such as:

Slaveof creates a master-slave structure with one click (full automatic synchronization and automatic conversion to incremental synchronization)
Also supports Redis's cascading cluster mode
There is no need to re-import data after restarting, and the data will be automatically loaded in seconds
Synchronization uses binary logs as the medium. This persistent log synchronization solution makes the slave database no longer have headaches such as Redis slave database "restart redo", "network disconnected for too long to redo", and disaster recovery from the database Then you can continue to synchronize through the binlog location before the disaster
There is no need to refill the data from the slave database after the failure of the master, and the slave database to the new master database can be mounted in seconds, and the synchronization and automatic transmission will be resumed, which greatly improves the recovery time of the entire cluster after the failure.
Non-blocking snapshot full hot backup, while the backup speed is extremely fast (completed in milliseconds), the backup only takes up a small space, and we provide an incremental backup tool
Pika provides multi-user (administrator/ordinary user) and multi-user "command blacklist" functions, which can shield users from risky commands, while the administrator account is not restricted
Other minor improvements will not be explained one by one.
7. Pika's status output is as close to Redis as possible in the design, so some of your current Redis monitoring strategies can be directly used in pika without modification.

8. Pika is compatible with codis (Thank you left2right of Huanxin Company for supporting pika codis)

9. Whether migrating to pika or migrating from pika is very simple, we provide the following tools:

Redis -> pika migration tool, which is based on AOF, can send the data in AOF to pika in full + auto increment
ssdb -> pika migration tool
pika -> Redis migration tool
Other "features":

Pika has three part-time product managers responsible for function improvement, BUG tracking, performance and functional testing, and their other identity is DBA
It is said that the pika development group (WEB platform infrastructure group) may be one of the "best knowledge" rocksdb teams in China
There are no guinea pigs in the pika community. All new versions of pika will run through nearly 1500 test cases and then prioritize internal use, and will only release to the public after reaching a stable level.
The version used internally by pika is exactly the same as the open source version

Pika is positioned as a supplement to Redis scenarios, we have selected some scenarios that may be suitable for pika

Using Redis leads to high server costs
The amount of data is huge (exceeding the upper limit of the single-machine memory) but it is not easy to fragment
Frequent requests with high time complexity make Redis block intermittently
Hope that the data can be truly persisted instead of AOF
Store a lot of data in Redis but can't stand the long data reloading time after the disaster
Read and write separation and do not want to fail to switch to the master after the shadow to the slave library

Current status of pika users

360 inside


实例数量：已达500+

承载请求：400亿次/天

总数据量：4T（此处为单份数据未记入冗余部分，例如一个主从集群2个实例每个实例20G共40G，此处只按一个实例的数据量20G计算），换算到Redis大致相当于20T内存

Community user

Please refer to the community user LOGO on github or join the QQ group to learn more