Article Directory
foreword
Database Mesh, this concept was first proposed in 2018 by Zhang Liang, the author of the open source software shardingsphere. Its meaning is that Database Mesh uses a meshing layer to uniformly manage databases scattered in every corner of the system. The web of interactions between applications and databases gathered through the meshing layer is as complex and orderly as a spider's web. This technology focuses on how to organically connect distributed data access applications and databases. It pays more attention to interaction, which is to effectively sort out the interaction between messy applications and databases. At present, this technology is still in a relatively early stage, and only a few companies have practiced it. I will talk about my understanding of this technical issue later in this article.
1. Service Mesh: Service Mesh
Before we start discussing Database Mesh, let's take a look at the more well-known Service Mesh. The former is also derived from this concept. The so-called Service Mesh is a service mesh, which is a proprietary infrastructure layer. The basic idea is to insert a network of proxies between services to abstract the network layer. In turn, it can make the communication between services within the microservice architecture more reliable, fast and secure; help developers solve the interaction challenges between microservices.
Let's take a look at the implementation mechanism of Service Mesh using Istio, a relatively popular open source service mesh project. Istio is an open source project jointly developed by Google, IBM and Lyft, which aims to provide a unified, secure, manageable, and monitorable way to connect microservices. The following figure is the architecture diagram of Istio:
From a large perspective, the entire architecture is divided into two planes: data plane and control plane. The upper part of the data platform is mainly composed of Proxy components; the lower part is the control plane, mainly composed of Mixer, Pilot, Galley, and Citadel components. The former can efficiently forward traffic, realize service discovery, load balancing, secure transmission, multi-protocol support, circuit breaker, health check, diversion routing, fault injection, system measurement and other capabilities, while the latter provides configuration management capabilities from a global perspective of data. By abstracting the above components, the typical structure of Service Mesh can be obtained:
The core of the above is a mesh data plane formed by the Sidecar proxy (discussed later), through which the traffic between all services is processed and observed. The data plane plays a role to establish, secure and control traffic through the mesh. The corresponding management component responsible for how the data plane executes is called the control plane. The control plane is the brain of the service mesh and provides exposed APIs for mesh users to manipulate network behavior more easily. The business application system only needs to communicate with the SideCar agent deployed in the same machine or Pod to implement many governance services that the above microservice environment relies on.
Service Mesh Advantages
So why is service grid technology beneficial to solve various problems in microservice governance?
Previous governance capabilities for microservices were often built within the application itself (as shown on the left), which includes traffic management, circuit breakers, retries, client load balancing, security, and observability, etc. As our requirements for governance functions continue to increase, the logic of this part of the code also needs to be continuously enhanced. But there is a problem here, that is, even if my application logic has not changed, the entire application needs to be constantly changed in order to meet the needs of service governance capabilities. With the growth of applications and the complexity of business, the above service governance functions will also become very complicated. This is a simple idea of decoupling, that is, decoupling service governance capabilities from applications. Through the method on the right side of the figure above, the service governance capability is SideCar-like. These SideCar capabilities do not need to depend on a specific technical framework and can support multiple programming languages. The service governance functions that originally needed to be implemented in the code base were abstracted into common components, and gradually sank into the agent. The standardization and unification of these service governance capabilities can solve the problems of large differences and lack of commonality faced in the implementation of complex system microservices, and can well support different programming languages and different frameworks. By sinking the abstraction of application service communication capabilities to the infrastructure, developers can focus more on the development of business applications themselves, and let the infrastructure provide these common capabilities.
2. Database service grid: Database Mesh
Similar to Service Mesh, microservices require governance, as do database applications. Its basic governance capabilities include database discovery, access routing, data sharding, read-write separation, load balancing, fusing, link collection, observability, and more. In order to achieve the above capabilities, generally by what means? There are two common ones: Client mode and Proxy mode. The former is a client mode, which has the advantage of a decentralized architecture and the disadvantage of being unable to support heterogeneous languages. The latter is an agent-side model. The advantage is the support of heterogeneous languages, but the disadvantage is still the centralized architecture. Obviously, there are certain problems in the above two modes, and the corresponding SideCar mode is logical. The figure below is an excerpt from ShardingSphere's conception of database Mesh implementation.
The business code accesses the SideCar of the Database Mesh through the SideCar of the Service Mesh, and then accesses the database cluster. For the management side, you can still use the Proxy mode to access directly by disguising the database. Then, for these three modes, ShardingSphere also made a comparison, and it can be seen from the figure below that the SideCar mode is superior in many aspects.
In addition, as a cloud-native deployment practice, Sidecar is created and dies along with the life cycle of the host, and is completely dynamic and elastic. The corresponding elastic capabilities are provided synchronously with the elastic capabilities of the application. I believe that with the development of cloud native and Kubernetes, the Sidecar model will become more mature. The benefits it brings can benefit from execution efficiency, resource cost, management ease of use, and performance improvement. It is also expected that through a unified database service grid, the business can use standard technical components, reduce learning and maintenance costs, and only focus on business development and innovation.
3. Data service grid: Data Mesh
In the same way, in terms of database service governance capabilities, SideCar can be used to meet future needs. By extension, can the data service governance capability be solved in the same way? From basic metadata acquisition, data quality assurance, data storage and processing, infrastructure and modeling, etc. The figure below is an excerpt from the latest data management summary of the International Data Management Association DAMA2. The functional domains listed below are possible potential governance directions. If the core of Service Mesh is "flow" and the core of Database Mesh is "connection", is it feasible to say that Data Mesh can abstract the basic elements of data services? For example, the unified data access language SQL...
