After two years, the Pika community officially released a version that was developed by more than 50 people in the community and verified in the 360 production environment. v3.5.0While improving performance, the new version also supports Codis cluster deployment, BlobDB KV separation, and added new features such as Exporter .
We will detail the important new features introduced in this release.
1 Remove Rsync
Prior to version v3.5.0, Pika used the Rsync tool to synchronize the stock data in the engine, and an Rsync child process was created when the Pika process started. This synchronization method has some problems in actual use, including the failure of normal synchronization after the Pika process crashes and the Rsync process exits without reason during the synchronization process. In the v3.5.0 version released this year, we have made important improvements in the full synchronization solution, abandoned the Rsync used in the past, implemented a new data synchronization solution, supported breakpoint resumes, and dynamically adjusted transmission speed limits and other features , to ensure a more stable and controllable synchronization process. These improvements not only enhance the reliability of synchronization, but also provide users with a better experience.
- Remove the Rsync process and use the self-developed full synchronization method
- Realize breakpoint resume transmission, transmission speed limit function
- When Pika master-slave synchronization, check the master run_id
2 Compatible with more Redis commands
In the v3.5.0 version, we have taken a bigger step, improved the compatibility of Redis commands, and provided wider support for Redis commands. The improvements in this version make Pika perform better in integration with the Redis ecosystem, providing users with richer functions and wider possibilities. Our expansion of command support provides users with more flexibility to meet the needs of different scenarios.
- Support UNLINK command
- Support INFO COMMANDSTATS command
- Support HELLO, SETNAME commands
- Support BLPOP, BRPOP commands
- Added Pika original DISKRECOVERY command
3 RocksDB version upgrade and hierarchical compression
In the v3.5.0 version, we made an important upgrade, upgrading the RocksDB engine to v8.1.1 version, and realized the integration of the hierarchical compression function. This upgrade is not only a technological leap, but also a reflection of our continued focus on system performance and optimization. Through this upgrade, we have added a higher level of data management capabilities to Pika, while also allowing the system to better adapt to different compression needs, providing users with greater flexibility and efficiency in data storage and retrieval.
- Upgrade RocksDB version to v8.1.1
- Implement RocksDB hierarchical compression
- The new RocksDB cache configuration item num-shard-bits can be read from the configuration file
4 Support BlobDB
In the v3.5.0 release, we introduced a compelling innovation - the separation of the BlobDB and KV storage layers, breathing new life into our system. This version upgrade makes Pika more flexible and efficient in terms of data storage. By supporting BlobDB KV separation, we provide a more optimized data storage structure and bring deeper advantages to users' data management and query operations. This important improvement will demonstrate its strong potential in more application scenarios.
- Support BlobDB KV separation
5 Codis-based cluster mode
In the v3.5.0 version, we actively introduced the Codis cluster mode. In addition, we not only integrated the Codis cluster mode into the system, but also provided command support for migrating slots, thus realizing more intelligent cluster management. This major change not only expands the scope of application of Pika in large-scale data storage scenarios, but also further improves the scalability and high availability of the system. By introducing the Codis cluster mode, we provide a more optimized solution for user data processing and management.
- Introduce Codis to Pika
- Introducing Codis's CI
- Support Codis migration slot command
- Add whether it is in the slotmigrate state of reload
6 Observability
In version v3.5.0, we introduced an innovative tool --pika_exporter to improve the observability of the Pika database. The addition of this tool is not just a reflection of our continued enhancements to our system monitoring capabilities. In the subsequent updates of the version, we further enriched the indicators and continuously enriched the observability of Pika. It provides users with more comprehensive and accurate data insight.
- Added Pika observable system pika_exporter
- Added network I/O traffic monitoring indicators
- Added command statistics time-consuming indicators
- Added estimate_pending_compaction_bytes metric to analyze fragmentation rate metrics
- Added RocksDB metrics
7 Containerized deployment
In the v3.5.0 version, we introduced an innovative milestone -- the pika-operator mvp version, which technically achieves an important goal: migrating the Pika single instance service to the Kubernetes (K8s) platform Rapid deployment. This is not only a reflection of our continuous attention to the development of the industry, but also our pursuit of continuous improvement of user experience. Through pika-operator, we provide users with a more convenient deployment solution, integrating Pika's high-performance database engine with the flexibility of Kubernetes, so as to bring more efficient and flexible support to users' application environments.
- Realize the rapid deployment of Pika singleton service on K8s
- Implemented the deployment of Pika in the MiniKube environment
- Add E2E test to pika-operator
8 Cross-platform compilation
In the v3.5.0 version, Pika has shown a comprehensive and vigorous development trend, and can demonstrate its superiority on different operating system platforms. The breakthrough of this release is that Pika implements full compilation and usage support for the major platforms of MacOS, CentOS and Ubuntu. This initiative not only reflects our focus on diverse technical environments, but also aims to maximize the expansion of the user base and provide flexible and efficient database solutions for a wide range of user groups. This enhancement of cross-platform compatibility pushes Pika into a broader technology ecosystem.
- Support MacOS platform
9 Multi-platform CI, Go integration test, TCL unit test, PythonE2E test, CTest unit test
In the v3.5.0 release, we've taken an impressive step forward with not only support on several major OS platforms, but also a comprehensive upgrade in the testing area. We have built continuous integration (CI) environments for the three major platforms of Ubuntu, CentOS and MacOS to ensure the integrity and stability of the system. In terms of testing, we have introduced more extensive coverage, including Go language integration tests, TCL unit tests, and Python end-to-end (E2E) tests. With these test strategy upgrades, we have taken a bigger step towards ensuring system performance and reliability.
- Add CI in CentOS environment
- Added CI in MacOS environment
- Added E2E test framework
- Added the addition of CMake compilation environment in Github CI Workflow
- Added the populate method in the TCL script to simulate the Redis debug populate method to fill the test data
- Add CMake file in blackwidow, add unit test for blackwidow
- Porting Redis test script
10 Others
If you have any questions, you are sincerely welcome to scan the QR code on WeChat, join our communication group, and have in-depth discussions with like-minded members. The journey adds bricks and tiles. In this group, we will build a technical community of mutual learning and continuous improvement with excellent wisdom and interactive cooperation spirit.
