Data Architecture Design in Microservice Development

foreword

Microservices are a very popular technical framework at present. Through the miniaturization and atomization of services, as well as the elastic scaling and high availability of distributed architecture, loose coupling between services, flexible adjustment and combination of services, and high availability of the system can be achieved. It provides a good basic platform for business innovation and business continuity. This article shares the design ideas and design points of the data architecture under this technical architecture. This article includes the following contents.

• Multi-tier data architecture design in microservice technology framework

• Key points in data architecture design

• Point 1: Data Ease of Use

• Point 2: Primary, secondary data and data decoupling

• Point 3: Sub-library and sub-table

• Point 4: Multi-source data adaptation

• Point 5: Multi-source data caching

• Point 6: Data Marts

For easy understanding, this article uses a simplified sales model to illustrate, as shown below. Figure 1 shows the relationship between customers, sellers, commodities, pricing, and orders (other elements such as payment, logistics, etc. are omitted here).

Figure 1 Sales Model

In this sales model, sellers provide products, set prices, and customers choose products to buy and form sales orders. According to the conceptual design of microservices, it can be divided into customer service, seller service, commodity service, pricing service, order service, and public service (such as authentication, permission, notification, etc.), as shown in Figure 2.

Figure 2 Microservice function

Multi-tier data architecture design in microservice architecture

The distributed architecture generally divides the system into three layers: Saas (Software-as-a-Service), Paas (Platform-as-a-Service), and Iaas (Infrastructure as a Service). The Saas layer is responsible for providing external business services, the Paas layer provides the basic application platform, and the Iaas layer provides the infrastructure. Microservices are vertically embedded in these three layers of services and are independent of each other. Therefore, when designing the data architecture, it is necessary to consider the focus of the three-tier service on the data, and also consider the independence of the microservice.

Hierarchical Design of Data Architecture

Figure 3 Microservice technology framework

As shown in Figure 3, the Iaas layer provides the physical basic environment in which the program runs (a lot of hardware and network content is involved here, which is omitted in this article). The Pass layer is subdivided into three layers, the basic service layer, which is mainly responsible for data storage and processing; the transaction framework layer, which is mainly responsible for the registration and scheduling management of microservices and distributed transaction processing; the application service layer, which mainly implements the APIs of each microservice for Direct calls of other microservices and service calls of the Saas layer. Saas services are business services that are publicly provided to the outside world.

Here I recommend an exchange and learning group to everyone: 575745314, which will share some videos recorded by senior architects: Spring, MyBatis, Netty source code analysis, high concurrency, high performance, distributed, principles of microservice architecture, JVM performance Optimizing these becomes an essential knowledge system for architects. You can also receive free learning resources, which are currently benefiting a lot

The data architecture is correspondingly divided into Raw Data layer, Logic Data (inner) layer and Logic Data (outer) layer from bottom to top (Iaas is mainly based on the basic hardware environment, which is omitted in this article). The Raw Data layer is based on database, file or other forms of data content. The Logic Data (inner) layer is the logical data used by the microservice API, such as customer data, order data, and so on. The Logic Data (outer) layer provides data for external services, such as customer order data. Therefore, the hierarchical result of our data architecture is shown in Figure 4.

Figure 4 Data Layer Architecture

In addition, a lot of information will be displayed in the form of pictures or reports. Therefore, on top of Logic Data (outer), Information Block (commonly used information block) can be constructed, and after setting the View type (display mode), the final View is displayed.

As shown in Figure 4, the closer to the external service layer, the greater the influence of the customer on the designer, and the more consideration must be given to usability, ease of use, and applicability. On the contrary, the farther away from the external service layer, the more concerned about data storage in design.

The advantage of the data three-tier architecture is to realize the layer-by-layer transition of data from system implementation to business implementation, and to achieve loose coupling between business data and system data. At the same time, the flexible expansion of the business and the flexible expansion of the system are realized.

Key points in data architecture design

The layered design of the data architecture is described above, and the main points in the design of the data architecture are described below.

Point 1: Data Ease of Use

No matter how data is implemented, its ultimate purpose is for business (or customer) use. Therefore, when providing external services, the ease of use of data is very critical.

Figure 5 Data Ease of Use

As shown in Figure 5, in the Logic Data (inner) layer, for the sake of data flexibility and non-redundancy, the customer information is divided into several sub-tables for storage. For example, the personnel address table can store an unlimited amount of customer address information. The advantage of this is that each time the personnel address is updated, instead of directly updating the personnel address, a new address data is generated, and the original address information is saved as historical data, which is easy for quick data recovery and historical information tracking. However, when the Logic Data (outer) layer provides external data, the first consideration is to provide enough information for one-time performance (after all, the operation of query is much higher than the operation of modification), so as to reduce unnecessary information in business scenarios. For example, when only three common addresses are provided to general customers, address 1, address 2 and address 3 in the data design are placed in one table.

Point 2: Primary, secondary data and data decoupling

It is unrealistic for the data of each microservice API to be completely independent. For example, the order needs to have data such as products, customers (including consignees), sellers, and prices. If these data are managed in the order service API, then the changes of customer information, price adjustments and other information must be synchronized to the data in the order API, and the coupling degree of data will become very high. When designing data, it is necessary to consider reducing the interdependence between data. Therefore, the primary and secondary data for each microservice API needs to be determined first. Master data refers to the core data of the microservice API. The addition, deletion and modification of this data are mainly concentrated in a microservice API, such as the order data in the order service API. Secondary data refers to data that references or maps other microservice APIs, such as product data and price data in the order service API. Secondly, in order to reduce the coupling between the data, the data association table is used to represent the relationship between the data. If you want to remove the association between the data, you can directly remove the association table, which has no effect on the data itself. Specifically, as shown in Figure 6.

Figure 6 Primary, secondary data and data decoupling

Point 3: Sub-library and sub-table

As the amount of business data continues to increase, a large amount of data will be accumulated in a single database or single data table, such as order data. With the passage of time and the increase in the number of customers, more and more order data will be generated. When the data accumulates to a certain level, the performance of data operations will drop significantly, which is what we often call the database "unable to carry". Therefore, data sub-database and sub-table should be considered in the data architecture design stage.

As shown in Figure 7, the database is divided, that is, we divide the order data into the current data application database, the historical database, and the historical archive database. The current data application library is used to support the generation of new orders and the addition, deletion and modification of orders in execution. The historical database (for example, it is divided into the last 3 months and the last 1 year) is only used when customers want to view past orders. Historical archived data (archived by year) are not directly disclosed to customers in principle, and are used for reference and statistical analysis. For the current data application database, you can continue to subdivide the database according to the customer number range. In this way, the size of each database can be effectively controlled. Partitioning, that is, storing a piece of information in two or more tables. For example, the order information is divided into tables according to the basic information and the detailed information, which can be applied to the basic information query of the order and the order detailed information query. In short, the core of sub-database and sub-table is to control the load (data volume and data information volume) of a single database, and to cope with the growth of business data volume through multiple tables and multiple databases.

Figure 7 Sub-table and sub-library

Point 4: Multi-source data adaptation

In addition to the traditional relational database, there are a variety of data sources, such as multimedia data files or data streams such as images, sounds, and videos, and various heterogeneous data such as CSV, TXT, Doc, Excel, PDF, and XML. These data need to be processed accordingly and converted into manageable data information. Therefore, when designing the data architecture, it is necessary to configure the corresponding read and write adapters for data sources of different natures, and at the same time, there needs to be a place for unified scheduling, as shown in Figure 8.

Here I recommend an exchange and learning group to everyone: 575745314, which will share some videos recorded by senior architects: Spring, MyBatis, Netty source code analysis, high concurrency, high performance, distributed, microservice architecture principles, JVM performance Optimizing these becomes an essential knowledge system for architects. You can also receive free learning resources, which are currently benefiting a lot

Figure 8 Multi-source data adaptation

Point 5: Multi-source data caching

In addition to the complexity of processing logic, a large part of the performance of data processing is the operation time of the target data (including reading and writing to hardware disk devices and network transmission). The network speed, especially the use of optical fibers, has been greatly improved, but the read and write efficiency of machine disks has not been significantly improved. Therefore, reducing disk read and write is an important way to improve efficiency. Data caching is to put frequently used data (data that does not change frequently) and recently used data into memory. In this way, the operating overhead of the system on the hardware disk device can be greatly reduced, and the performance of the entire data system can be improved, as shown in Figure 9.

Figure 9 Data cache

Point 6: Data Marts

Data marts are a big topic. When the existing data cannot be simply associated with several table data and can be used for business after simple processing, it is necessary to consider building a data mart. The data mart analyzes and processes data from the viewpoint of data application, and provides a usable and stable data source for the business after a series of data operations such as multi-source data import, cleaning, processing, and view creation. For example, in sales analysis, what kind of products are preferred by customers, the impact of price on sales amount, and the relationship between sales amount and regional date, etc., the concept of data mart should be used, as shown in Figure 10. Show.

Figure 10 Data mart

Data carries information, and good data architecture design will make business systems more fluid, easier to understand and maintain. This article just summarizes some of the experience in practical engineering, for everyone to share. If there are any deficiencies, please add and enlighten me.

The full text is over! If you find this article useful, please collect and reprint it!

Thanks everyone for reading!