When you are in electronic business platform spike goods or brush when hot topic in the social network, you can clearly feel the terror of the current network data traffic, hundreds of thousands of goods just getting robbed, one second to none sold out; what a big star derailment news appeared, and instantly read and transfer times can reach hundreds of millions. As we may think of the end-user, the service system is how to survive in such harsh traffic conditions.
In fact, the service system architecture There are many clever design to deal with this problem, and in this one, the system will usually set up a caching system to alleviate the impact of the massive data access request brought by high-performance access requirements.
Meanwhile, with the development of micro-cloud services and technologies, the demand for distributed architectures become more common, coupled with the type of data on the Web today is no longer a single, but the amount of data also showed explosive growth, the traditional structured storage scheme has not keep pace, the database SQL operations are no longer meet the requirements, then NoSQL appears.
Several solutions integrate these technologies, we can design a distributed NoSQL caching system, this type of system currently has some of the more powerful open source solutions, such as Memcached and Redis, their availability for the entire service system, scalability and performance play a crucial role.
Tencent recently distributed an open-source NoSQL storage systems DCache, its typical application scenarios in distributed cache. According to the official introduction, DCache TARS-based micro-management service program, which supports kv, kk-row, list, set zset with a variety of data structures, memory-based data storage, while the support of Access DB data persistence. DCache with rapid horizontal scaling capabilities, while supporting a Web platform for efficient operation and maintenance of operation and maintenance operations.
We interviewed the first time DCache R & D team silver treasure mountain, hoping to have a better understanding of the R & D background and technical details of the project.
The current open-source distributed caching system, Memcached and Redis is a very popular choice, the Tencent Why make yourself a system?
Shan Bao silver introduced, although Memcached and Redis itself has a very strong ability, but there are difficulties in operation and maintenance, lack of challenges clustering scheme can not cope with the problems caused by micro-services trend.
For example, micro-current service is a big trend, we are said to do micro service, which allows decoupling between computing and storage to achieve lightweight communications. Service does not need to micro-manage the same period in life, but as a system component Redis is not, "we want to do logic and data layers separated design services architecture, but if you use Redis as cache, cache-coherent data between DB issues, as well as a cache miss how to solve issues such as the user needs to do the relevant business logic processing, which adds a certain complexity and difficulty, but also increases the degree of coupling logic layer and data layer. "
On the other hand, silver treasure mountain introduction, initially face of massive and high-performance data access requirements, internal Tencent each team actually developed their own caching system, however, the agreement between these systems are not uniform, diverse service model, does not have universal fault-tolerant, scalable uneven, so the team set out to develop a set of common Cache DCache this system, hoping to solve the overall business, development, operation and maintenance and monitoring of the challenges faced.
So it can be seen that, at present DCache has been applied to the internal Tencent multiple services, including QQ browser, application treasure map Tencent, Tencent computer butler, housekeeper and Tencent mobile phone games
SQL, NoSQL distributed with the trade-offs
SQL refers to the structured query language database, which is the operation command set of the database, traditional relational database using standard SQL statements to manipulate data processing. Distributed architecture model is a software system, distributed system, a plurality of hardware or software components distributed on different computers, communicate with each other by passing messages, external performance as a whole, to provide a unified service.
There is a widespread view is that there is a natural antagonism between SQL databases and distributed, silver treasure mountain understanding is: "distributed systems because the data scattered in different nodes, like the joint operation of SQL tables, and other needs of global affairs the lock protection, this treatment more complicated, and affect performance. "
There are trade-offs with SQL to NoSQL, refers to a class NoSQL databases, primarily for ultra-high-performance processing huge amounts of data, it is a major feature of a simple data structure to key-value-based, non-correlation between the data, easily do horizontal scaling.
Literally, SQL and NoSQL seems to be contradictory, do NoSQL seems to mean giving up SQL, NoSQL but in fact the intention is Not Only SQL, it is not just SQL, then it can contain SQL capabilities. "NoSQL is not necessarily have to give up SQL, in fact the agent layer can increase SQL parsing, computational logic to implement SQL operations, but it will affect the performance, we still look at scenarios and business needs."
Shanbao silver is our simple analysis of the significance DCache "distributed NoSQL" is. In SQL processing, distributed there appears to be a disadvantage, however, means you can link distributed more cheap computer, make full use of force count, low-cost way to deal with concurrent access request high strength, in addition, there are many distributed architecture advantages, such as avoiding total failure of the system of a single point lead, to achieve high availability. On the other hand, silver treasure mountain also said: "DCache because the main goal is a high-performance, SQL operations are not mainly want to solve the problem, so DCache not implemented SQL functions."
DCache distributed strategy and capabilities
DCache particle group is external service provider, is responsible for a part of a group of data pieces, each data group as to which services to make the range of the hash key mapping according to which the data is determined.
key DCache data will be mapped to the 0 ~ 4294967295 (unsigned int) range by the hash algorithm, then the range of 0 to 4294967295 evenly divided into different group. For example, two group, the value of the hash key to do it in the range of 0 to 2,147,483,647 distributed to group1, in the range 2147483648 to 4294967295 would distribute to group2.
Within a Group, used standby architecture, only the master node receives the write request, the data consistency can be guaranteed, and when the host is unavailable, triggers automatic standby switch available to ensure continued service.
DCache architecture
We rely on strong doubts DCache seem etcd and TARS middleware, that its own core features and capabilities is reflected in where?
Shan Bao silver explanation, DCache not strong dependence etcd, "etcd involves only the selected primary route service RouterServer if RouterServer single point of deployment are also available, and RouterServer downtime will not affect the data read and write access, since all the Proxy and cache service has local route cache ", adopted on TARS, he said:" because TARS is a very good service development framework, which shields the details of the underlying communications network, and comes with a name, and many other services service capabilities required for DCache, using existing TARS framework can be better done as a service, we do not need to repeat-create the wheel. "
As for the ability DCache itself, silver treasure mountain introduction: "DCache own storage engine with high performance, and the support of Access DB, for the user, do not need to be concerned about the consistency of data between DB and cache, as well as cache miss caused a series of problems. "
Specifically, DCache persistence and Redis is not the same, which is only the data in memory on the local disk to do a backup, ensure data recovery after the restart Redis.
"Redis persistence mainly to back-end data backup .DCache later with DB, back-end business logic and data can be completely separated, DCache itself will handle data consistency between the cache and DB .DCache will continue to landing back-end data in the DB Cache, Cache if storage space is not enough, the cold data will already landed the DB eliminated. in the process of data query, if the query Cache misses, and re-reads from DB to store Cache in order to ensure that hot spots and hit rate of the data in Cache, while penetrating questions and DB Cache has also been resolved. in addition, the ability to back-end data persistence for some DB needs to be done off-line data analysis business scenario is relatively convenient short things you totally do not care about the data, just need to write data to the Cache, landing back-end processing by the DCache. "
DCache properties
In addition, DCache distributed clustering, remote image deployment, disaster recovery fault tolerance will provide a very high value in practical applications online.
Useless
As a distributed storage system, DCache application scenarios there is no limit on the cache, Shan Bao silver introduced to the scene with a high performance NoSQL storage needs, you can use DCache, but also because DCache have the capacity and the elimination of outdated automatic cleaning function data , the need to store hot data (such as popular articles) scene with temporary data (if timeliness chats) can also provide good support.
Shan Bao silver also provides DCache performance data:
Currently Tencent QQ browser, including internal applications treasure, map Tencent, Tencent computer butler, housekeeper and Tencent mobile phone games, including nearly a hundred businesses have access to a DCache, large body mass of these businesses can imagine, silver treasure mountain supplement : "in addition to this simple set of data provided, DCache in efficiently and reliably support the operation of nearly a hundred businesses, the average daily call volume is too trillion, which is also from the side of the DCache performance and stability in the production environment . "
And in addition to their high performance, scalability, availability and security of data outside the system design, Web visualization platform for efficient operation and maintenance has become an indispensable DCache ability. Memory NoSQL storage systems will have a huge overhead on the operation and maintenance based, it requires in-depth understanding of the relevant technology, and resolute to make good decisions in a pinch.
DCache based TARS development, so the operation and maintenance platform DCache with TARS service management are united on one module, Shanbao silver introducing the operation and maintenance platform will greatly improve efficiency, while reducing operation and maintenance threshold, on the deployment of services, on-line , migration, expansion, monitoring and configuration of these operations can be easily achieved.
Guest Introduction
Shan Bao silver, Tencent senior engineer background, focus on technology research and development of distributed NoSQL storage areas, more involved in the development of Tencent self-developed storage system, has rich experience in the field of distributed systems, high availability and high performance services .