Nanny level tutorial! Golang microservice simple architecture practice

Introduction | This article starts from the theory of concise architecture, and relies on the trpc-go directory specification, briefly explains how to divide the overall code architecture, the specific implementation details of trpc-go service code, and the implementation steps, and discusses the difference from DDD. The article originated from the first part of the best practice of go microservices initiated by our group. We hope to summarize a set of go microservice development methodology from development and reading learning, and share with each other the thinking and thinking in the process of seeking the best practice. The process of trade-offs. This time we mainly discuss how to organize the directory. The organization of the directory is actually the design of the architecture. The design of a set of general architecture allows developers to focus on logic design and code design for specific scenarios. The standard operation is simple, and it can be implemented step by step according to the situation, with strong operability.

introduction

Now with the efficient go language, the mature trpc-go framework and a series of middle-end SDKs and publishing platforms, a novice can also quickly write microservices with simple functions through tutorials, so as to get started and start microservice development in go , and handle most development needs.

But once we start, we will find that as the demand increases, we often have to spend a lot of time maintaining the code, changing the existing logic, constantly abstracting, and improving the scalability of some commonly used capabilities. Collaboration and maintenance in the same microservice code is becoming more and more difficult, not only because of different abstract styles, but also for abstract standards, module division, data flow, and layered logic. , Seeing that each service is like a new life, in various poses and with different expressions.

A code base with various forms is not what we want. We want to maintain legibility , scalability , and maintainability in the code architecture . In this way, in addition to the consistency of code details (code standards), we also hope to have architectural standards. , so that development can focus on logic design and code design of specific scenarios, and do a good job of service-related content with the knowledge of the vast amount of knowledge, instead of wasting time and energy on how to reorganize and solve messy, refactoring, etc., if The structure of each service is concise and clear enough, and each warehouse within the team seems to be written by itself, and it will be quick to get started, and the efficiency of the team will increase geometrically.

1. Development status

Different business scenarios are different. In value-added business scenarios, most of the demand boundaries or all service functions cannot be determined at the beginning. Generally, it is a microservice that starts with a small demand, and may gradually change as the business grows. It is very complicated, as if it is gradually growing from a small sapling to a large tree with luxuriant branches. At the beginning, the responsibility of this service may be very simple, very simple, a service with a logic is ok, but later added various If there is such a dependency, the logic will start to become complicated. What's more frightening is that because one requirement is one requirement (assuming that the product demand cannot be predicted in the worst case), for backward development models, or without architectural concepts, one more Requirements are nothing more than adding a function, adding a branch, and importing what is needed. Gradually, most services become:

• There is no reasonable subcontracting, or only logical responsibility subcontracting (sub-category)

• For procedure-oriented programming, the function call chain is very long, interspersed between various packages.

• Without dependency injection, dependencies exist in the form of global introduction, with a large scope and potential concurrency problems.

which eventually results in :

• Not universal, everything is not universal, each modification of a logic part may involve multiple function calls to modify the parameter relationship.

• It is difficult to write test cases. Is the imported function a mock or not? Write mock functions one by one. Is monkey mock reasonable?

• Each module is not independent, and seems to be logically divided into modules, such as order_hanlder, conf, XXX_helper, database, etc., but there is no clear upper-lower relationship. Each module may have configuration reading, external service calls, protocol conversion, etc. .

Let’s first look at the current microservice code status, and cut out a few common microservice directory organization styles:

Four currently common microservice directory organization methods, from left to right are 1, 2, 3, 4, you can see:

• Service 1 is all placed in logic except main, and logic has actually lost its responsibilities.

• Service 2 is all flat, why did the author do this, because he wrote a lot of monkey func mocks, because there is no abstraction, calls between different functions cause many function mocks to be reused, but the content in the test file does not support import , so in order to avoid the underlying logic function to repeatedly write mocks in different packages, it is simply tiled.

• The common organization method of service 3 is to subcontract and decouple modules with logic as a unit, which basically conforms to the principle of single responsibility, but this kind of microservice will cause the problem of network call as the demand grows.

• Service 4 has a certain abstract directory design for external calls, but the organization method is not clear at a glance, there is no reasonable subcontracting, and the logic code is written in the access layer.

(1) No structure

As in the above example, most services have no concept of architecture, and most services are divided into packages (sub-categories) in the form of logical units. The relationship between each package is a horizontal relationship, and the logic of each package is independent. In theory When using the package function, just import it, as the service grows:

• The calls between different package functions of the service slowly evolved into a network structure.

• The flow direction of data flow and the combing of logic become more and more complicated.

• It's hard to figure out where the data is going without looking at the code calls.

This is a common practical problem at present. In the process of business growth, microservices can easily grow into a mountain of garbage, and the development is exhausted and cannot be changed.

The so-called code corruption means that after the code increment reaches a certain level, the function call organization inside the service is a network structure without a hierarchical structure. Even though it may be decoupled on a micro level, it is a mess on a macro level. Designs such as DDD Thoughts are all about solving such problems.

(2) No stratification

Common microservices only have the concept of subcontracting without layering, and the data flow is not layered. Because there is no reasonable layering, naturally there is no relationship between up and down calls. A system without layers is a mess, a mess of interfaces, calling each other at will, and the relationship is chaotic. This is a nightmare for future maintenance and debugging.

2. Explore best architectural practices

(1) Simple structure

From "The Way of Clean Architecture", this architectural model is an abstract architecture in a broad sense that does not distinguish between front-end and back-end. We hope that the code of each microservice also conforms to the concise architecture at the micro level.

In the background service scenario, a pyramid structure can be abstracted from the trpc-go directory specification:

The advantages of this structure are reflected in:

• Standard structure: layers + modules

1. The structure is layered, with modules divided between each layer.

2. The data flow direction is fixed, and there is a single direction from top to bottom.

3. The structure is clear, the demand code growth is structured, and the organizational relationship is not meshed.

• consistency

1. The architecture is common and can be unified and standardized.

2. The architecture of different services is the same during collaborative development, and there is no cost for understanding.

• easy to operate

1. The related concepts are simple, easy to operate, in line with development intuition, and facilitate the correct classification of code.

2. No additional issues such as domain modeling are involved.

• Slow down code bloat

1. Layer the code up or down, and the three-layer structure can reduce the code expansion speed of each layer to a certain extent.

(2) Catalog specification

Layering is divided into interface layer (gateway layer), logic layer, and external dependency layer according to the data flow direction. The division method and the cost of understanding are not very high. The details are as follows:

• gateway

• The place where the interface is implemented, at the entrance of the service interface, corresponds to the service of trpc-go.

• It only performs protocol analysis and conversion, and protocol arrangement, without involving business logic.

• logic

• The place where the core business logic of the service is implemented.

• Implement sub-module sub-contracting internally.

• repo

• External dependency layer, including external databases, RPC calls, etc.

• Each package provides an abstract interface, which calls and organizes external data and provides it to logic in the form of an interface.

• It only does external calls and data collation, and does not contain business logic.

• entity

• The data structure throughout the service, similar to constants and error codes.

• An anemic model, i.e. an object that contains only data structure read and write methods.

• Anti-corrosion layer

• Each layer is exposed to the outside world in the form of an abstract interface, and the anti-corrosion between each layer is realized by means of dependency inversion.

• The abstract interface can naturally generate stub code with gomock, and the upper layer only needs to use the stub code corresponding to the lower layer to mock the lower layer dependencies when the upper layer is tested.

3. Implement the specification

In practice, it is only the first step to classify code directories according to standards. The important thing is the isolation between layers and the decoupling between modules. Therefore, dependency inversion, dependency injection, encapsulation, and test specifications are needed. To implement the specific code, the test specification is a ruler to check whether the code design is qualified in reverse. If each interface cannot use gomock piling, then the dependency inversion is not done well.

(1) Dependency inversion and interface isolation

• Dependency Inversion

• Upper-level modules should not depend on lower-level modules, they should all depend on abstractions.

• Abstractions should not depend on details, details should depend on abstractions.

• interface isolation

• A client should not depend on interfaces it does not need.

• Dependencies between modules should be based on minimal interfaces.

Implementation requirements : The external interfaces between different layers are all provided in the form of interface, and the single-responsibility design, the interface is as simple and clear as possible, the interface file is stored separately, not in the specific implementation file, and the dependent parameter definition and interface declaration are placed together.

For example, api.go under the msg package defines the message interface:

(2) Dependency injection

Dependency Injection (DI, Dependency Injection) refers to a way to achieve Inversion of Control (IOC, Inversion of Control). In fact, it is well understood that what is internally dependent, do not create it internally, but inject it from the outside through parameters. example:

• Internal package

• High cohesion and low coupling.

• Reasonable abstract functions, molecular functions, clustering, etc.

example:

(3) Do not introduce mock packages other than gomock

If you must use monkey mock to pile functions, it means that the code does not conform to the interface principle. And the mock function of the Monkey mock cannot be exported, and the mock needs to be rewritten for each single test in the package of this function that is called.

The Gomock stub code can be automatically generated. When the upper layer needs to mock the lower layer dependencies, it only needs to inject the mock stub as a dependency.

(4) Configuration (remote configuration)

In addition to framework configuration, almost every service now accesses remote configuration (colorful stone configuration), and the logic of reading remote configuration almost every service has to be re-implemented, because the final output of configuration must be a personalized structure (The configuration of each service must be different), so it is difficult to solve it with a set of codes. Here, a package replacement method is used to introduce different config entity definitions to the exported structure to achieve common code (only It is universal, and it cannot achieve zero copy)

• One remote configuration per service.

• The remote configuration is in json format (same as yaml, internally unified)

• The remote configuration is defined in the entity/config package, and the structure is Config.

In this way, the following remote configuration implementations can be reused:

Here if the service has multiple configurations:

Example: This service has been refactored, and there was no specification before, so I made three different remote configurations (in fact, one is enough):

Because the structure returned by Get is different, different configurations are implemented using different interface instances. The configuration of each different structure is a fixed structure when parsing, and the return of get is also a fixed structure. In the case that the go template feature is not supported The configuration of each different file is parsed with different impl implementations. It seems that there are some repetitions in the code, but this expression can ensure clear and easy to understand. Generally, a service business configuration is placed in one file.

One configuration per service is of great help in reducing codes such as configuration initialization.

(5) Use of configuration

Interface-based configuration is very convenient to implement dependency injection, abandoning the previous method of introducing configuration packages and reading global configuration, and reducing the scope of configuration through dependency injection, avoiding many concurrency problems:

4. Landing method

The ideal is very full, the reality is very skinny, the contradiction between the progress of the demand and the quality of the code, if we want to achieve it in one step, in practice, it means that it cannot be implemented in one step.

The actual situation is often that the demand is very urgent, and there is not much time for development to design and optimize the code, so we hope that the first step will not take up too much development time, and the best time allocation can be from 1:9 Way to start, and at any stage can give priority to the rapid completion of requirements (that is, tolerate a certain degree of non-compliance without destroying the whole), that is, you can maintain your own old style with 90% freedom at the beginning, extracting 10% of the time is designed, so that implementing the specification is not very painful.

The overall landing steps can be divided into three stages (it is not necessary to go through, if the time is not tight, it can be realized directly according to the standard)

Practice in phases according to the urgency of your current needs and your personal schedule.

V. Summary

Consistency in microservice code architecture and consistency in implementation specifications can bring many benefits:

(1) Why not DDD

In fact, the reason why DDD is mentioned is because it is an unavoidable problem, but the answer already exists. DDD is the trump card for controlling medium and large-scale projects, but using DDD does not make the development of new projects faster and more convenient. It is for future consideration, so that a huge system can be iterated and updated faster, that is to say, new projects do not need to pay too much attention to domain-driven design, and even new projects can start without domain-driven design.

Advantages and disadvantages of DDD :

Different businesses may face different problems. Many practical requirements are often not big requirements and large projects with top-level design from the very beginning, and even many microservices have not yet determined the elements in their own field, and they die with the business. The domain model and boundaries are not clear at the beginning of service creation. It is impractical to design event storms, divide elements, subdomains, etc. from the beginning of a new service with an interface. Therefore, the smaller the service, the more DDD is not required. Many times we have to consider the rapid growth of new members of the team. It is difficult for a new classmate or intern classmate to quickly get started with DDD and implement DDD in each service, but not all of them. In this way, there will be gaps between services with different needs. When taking over the services of colleagues, there will still be a mental burden of understanding the structure.

postscript

The overall rules describe roughly, but in the process of practice, for the internal details, the abstraction of functions, clustering, and the division of sub-modules, it is the accumulation of experience and practice, which still tests a person's code skills. This architecture Specifications don't help.

A good architecture or catalog design is like a garbage can for garbage classification. With the classification rules set in advance, the garbage can be easily classified, and the sorted garbage can be turned into treasure and available resources. Therefore, in the face of Code like a mountain of garbage, when refactoring, we must first follow the correct architecture for garbage classification.

Although effective layering has been carried out, the module splitting in the logic layer is not strictly required. That is, after the abstract interface is provided, the specific implementation is a matter of details. As the demand grows, it actually faces the complexity brought by the growth. Relationship, but because the external call is split in repo and the data instance is in entity, the final logic code of microservices will not expand very quickly. The three-tier structure can slow down the complexity expansion to a certain extent. If one day the expansion is large , then refactoring using DDD may be another solution.

This article is to record the process of thinking and trade-offs in the process of seeking the best practice. After all, there is no silver bullet for the practice of microservice code architecture, and there is no better situation. There are only relatively easy to implement and simple and effective solutions It is more general .