Demonstration case of DDD, CQRS and microservice architecture based on .NET implementation

Recently, out of work needs, I learned about the Microservice Architecture (MSA). After two weeks of hard work in my spare time, with my understanding of the microservice architecture, I created an application case based on .NET to demonstrate the microservice architecture, combined with domain-driven design (DDD) and command query The Separation of Responsibilities (CQRS) architectural pattern has carried out some practical explorations on the event-driven microservice system architecture. Now I will organize my thoughts and gains into a document and share it with everyone.

Microservice Architecture

Before introducing the source code, I still want to talk about the microservice architecture. Although there are many discussions on the microservice architecture on the Internet, I think it is necessary to say more here. The guru Martin Fowler mentioned on his personal home page about microservices that microservices don't have a very clear definition. In fact, there are many implementations of distributed systems that can be considered (or barely considered) microservice-oriented architectures. Personally, I think a microservice architecture should meet the following characteristics:

• The entire system is divided into multiple applications (the so-called "microservices") with relatively independent business functions (Monolithic Architecture, or monolithic architecture), and each microservice usually follows a standard layered architecture Style or event-driven architectural style, it can process its own related domain logic, use the local database for data storage, and provide a relatively independent API interface or user interface to the upper layer. Each microservice can also use infrastructure layer facilities such as caching, logging, etc., but if these facilities are shared with other microservices, the infrastructure layer facilities need to meet the third characteristic below

• Communication between various microservices can be done in the following ways (see: http://howtocookmicroservices.com/communication/)

• Synchronization method: the most common is based on RESTful style API, it can also be cross-platform, cross-language Apache Thrift IDL

• Asynchronous mode: use a lightweight message communication mechanism, such as RabbitMQ, Redis, etc.

• The entire system is "cloud-friendly". The so-called "cloud friendly" means:

• For every microservice, the possibility of a single point of failure should be avoided. For example, for a microservice A in a system, there needs to be at least two (or more) running instances, and the API Gateway or Load Balancer needs to have at least two (or more) running instances according to certain rules (such as the running of each A). The health of the instance, etc.) Allocate the service request from the client to any running instance of A to complete the processing

• For each microservice, administrators can adjust the deployment of these applications based on some specific real-time technical indicators. For example, the query service load of a shopping website is significantly higher than that of the order management service, so the administrator can increase the deployment volume of the query service (such as deploying three instances of the query service) according to the actual situation, while reducing the deployment volume of the order management service. Compared with the single integrated architecture of the whole system, the advantages of doing so are obvious. It can make full use of cloud server resources, so that each microservice can run in a reasonable resource configuration state, reducing resource waste.

• The public infrastructure layer service facilities should also meet the conditions to avoid a single point of failure. For example, the database service needs to be configured with Replication/Clustering, and the message queue needs to use a similar fault tolerance strategy.

• Based on the requirement of "cloud friendliness", a lot of deployment and operation and maintenance problems are derived, such as the configuration mode of the microservice itself (multiple instances per machine? Or one instance per machine? Or per virtual machine? An instance? Or provisioning the instance into a container like Docker)? How can the message queue be configured so that multiple instances of the same microservice will not process the same message repeatedly? How does RESTful-based communication allow clients to find dynamically changing API addresses? etc. I think there is no specific answer to these questions, and it still needs to be judged according to the actual situation

Compared with the traditional integrated architecture system, the microservice architecture system has the following advantages:

• Each microservice is relatively small, making it easier to develop and debug

• Each microservice is relatively independent, which not only allows developers to focus only on a certain business processing part, but also implements different technologies for each microservice's own characteristics (for example, some microservices are implemented using C#, some using Java or Python, etc.)

• This independence enables microservices to perform well in fault-tolerant isolation: for example, if a microservice has a crash and other problems, it will not cause the entire system to become unavailable, which is in line with BASE (Basically Available, Soft-state, Eventually consistency) idea

• Due to the relative independence, the design of the microservice architecture can be more easily deployed in the cloud environment

• The independence of microservices also provides good support for agile development. For example, each service can be developed and deployed separately, and the project team can also balance development and testing resources according to the technical expertise of the members themselves

Of course, it also has some shortcomings:

• Developers need to deal with the complexity that comes with a distributed architecture. For example, if the asynchronous message communication mechanism is used for communication between microservices, then the development mode introduced by this message mechanism needs to be followed (creating a message handler, forwarding messages, etc.). In addition, this architecture also brings a lot of inconvenience to the test work. For example, when some test cases (Test Cases) need to cover the business of multiple microservices, it is necessary to pay attention to the execution results of weakly consistent distributed transactions. , which is often more complicated. Further, this kind of test work also requires the coordination of multiple teams to proceed smoothly. When each team is distributed in various countries and regions around the world, the coordination work is even more complicated or even difficult to carry out.

• Deploying, installing, and managing a system based on a microservices architecture in a production environment is also not an easy task. This requires the client side to have a strong professional technical background and problem-solving ability. Of course, a better way is to provide the service directly to consumers in a SaaS way

• More resource consumption. For isolation and fault tolerance, microservices may be deployed as N instances, each running in an independent virtual system. Assuming that there is a certain waste of system resources due to improper deployment strategies, this waste may also be expanded by N times.

I will write so much about the microservice architecture for the time being. The microservice architecture is relatively popular now. You can also check the relevant information directly on the Internet. Friends who are better in English suggest to go directly to the English website to search and learn. There are many excellent articles and wonderful discussions. . The architecture itself is that the benevolent sees the benevolent and the wise sees the wisdom. Different people have different understandings and have different viewpoints. Some viewpoints may be more appropriate in some scenarios, but another scenario reflects its weakness. But in any case, what I want to say is that no matter what architecture you choose, it always has advantages and disadvantages. The difficulty of architecture design lies in how to choose the most suitable mode, method, and technology to complete a complete set of system development solutions. In more cases, the entire application system is more likely to be an "ecosystem" that integrates multiple technologies and multiple architectural styles. For the project you are developing now, perhaps the classic three-tier architecture is the most appropriate.

WeText Project

There is theory and practice. To this end, I spent two weeks in my spare time, using Visual Studio 2015 to develop a case project: WeText. The business of this case project is still very simple: users can register and log in, and after logging in, they can modify their personal information, and then they can create some of their own Text (that is, small notes with title and text content), and can also send friend requests to other Users, after the other party accepts the invitation, can share their Text to the other party. As of this writing, the text sharing part has not been completed, but the other business parts have basically gone through, and there may be many bugs.

Seeing this, you will definitely want to complain, it will take two weeks for such a simple system to make such a big move, and there are so many bugs, and it has not been finished yet! Yes, it's not perfect at the moment, why? Because of the complexity of the architecture, I was thinking and designing while coding. Maybe the microservice architecture using CQRS is not suitable for such an application system, and even DDD may not be useful. Adopting such an architectural style on this project, I honestly just want to practice. So far, this project still has the following shortcomings, please bear with your readers. Of course, it is open source (Apache 2.0 License), and you are welcome to participate in discussions and contributions where you feel unsatisfied, just submit a Pull Request to me.

• The query part of CQRS uses a relational database, the database access layer does not use ORM, only the Table Data Gateway mode is implemented, but the implementation of the Table Data Gateway is a single-table structure, and cross-table query cannot complete the JOIN operation: interested friends can Implement another set of ORM-based query mechanism based on the existing WeText project

• Although the homepage of the web program claims to have adopted Event Sourcing, in fact, I did not record any events in the Event Store, but just saved the final state of the aggregation in the Event Store (for time reasons, it may not be finished in another month). It's successful, time and energy can't be wasted). CQRS has no Event Sourcing, Oh my god! But don't be surprised, CQRS doesn't have to use Event Sourcing: Interested friends can implement the Event Sourcing function based on the existing WeText project, but don't forget to get the Snapshot together, this is very important! You can also use the mature Event Store framework on WeText to complete this part of the function. Don't forget to share when you have a conclusion

• The command part of CQRS is encapsulated by RESTful API. Since command execution is asynchronous (only eventual consistency is guaranteed), while RESTful API is synchronous, RESTful API cannot return the final result of command execution. I'm thinking about introducing an Actor model-based solution like Akka to solve such a problem, but it doesn't necessarily work. Still looking for a solution. Interested friends can continue to consider this issue in depth

• The exception handling part is relatively weak: I will continue to strengthen this part

• The front-end interface (WeText.Web project) is relatively ugly and has some flaws. It is simply done using ASP.NET MVC 5 combined with Bootstrap, without using TypeScript+AngularJS, React or even jQuery to create some high-level user experience, basically satisfying Support for back-end business. Interested friends can throw away the WeText.Web project and only use the services provided by WeText to develop their own front-end interface

• It has not been fully verified whether the deployment in the cloud is feasible. It is theoretically feasible, but it has not been fully verified.

Overall structure

First, let's take a look at the overall structure of the WeText project, and the various components it contains, from an overall architectural perspective.

In the above figure, the blue part represents domain-related concepts, such as aggregation, protocol, event, Saga, warehousing, etc.; the yellow part represents microservices, currently there are three microservices, Accounts, Texting, and Social; the gray part represents the infrastructure layer Facilities, including Owin-based Web API host programs, message queues, Event Stores, and databases; the light pink color block represents a service host process (Service Host).

• The client program sends the command request to the server through the RESTful API (Web API)

• The server forwards the request to the corresponding microservice instance through API Gateway or Load Balancer (API Gateway and Load Balancer are not reflected in the above figure, that is another thing, I will discuss it in the future)

• The Web API Controller converts the request into a CQRS Command and dispatches it to the Command Queue

• The Command Handler obtains the Command message, accesses the Domain through the Repository (this process will involve Snapshot), and executes the command operation

• When the Repository saves the aggregation, it will store the events generated by the operation in the Event Store (this process will involve Snapshot), and at the same time dispatch the domain events to the message queue Event Queue

• After the Event Handler obtains the message, it performs message-related operations, and the Saga state transition is triggered in the Event Handler. After the Saga state changes, a state change domain event will be generated, and the Event Handler of this domain event will trigger the occurrence of another Command. (In theory, Command should be triggered directly in Saga, but Saga itself should also be an aggregate root, so it is obviously unreasonable for Saga to directly operate Command dispatch, which will be discussed later)

• The Event Handler will update the Query Database at the same time as needed (that is, the normalize step in the above figure)

• The client's query request will be completed directly through the RESTful API (Web API), and the Query Database is accessed through the Table Data Gateway.

For Service Host, in the above diagram it provides a hosting environment for three service instances at the same time. In fact, the design of WeText allows Service Host to host only one or a few instances, and multiple Service Hosts can be deployed on multiple different physical machines, for example:

So, in the whole environment, we have one Accounts service instance, two Texting service instances and two Social service instances. At least it provides a solution in terms of single point of failure and server resource balancing, but of course it also brings a lot of problems. for example:

• How to configure the routing of API Gateway so that client requests can be dispatched to the corresponding microservice instance according to the specified policy for processing?

• For microservices with multiple instances, how does the Pub/Sub-based message subscription mechanism avoid repeated processing of events or commands?

Also, you would think that there is a single point of failure for infrastructure layer facilities, such as RabbitMQ or databases. In fact, these mature products have their own solutions, such as database clustering. Or simply use the PaaS services (message queues, storage, etc.) provided by AWS or Azure. Therefore, solving this problem is not difficult.

If you want to learn Java engineering, high performance and distributed, explain the profound things in simple language. Friends of microservices, Spring, MyBatis, Netty source code analysis can add my Java advanced group: 694549689, there are Ali Daniel live broadcast technology in the group, and the video of Java large-scale Internet technology is shared with you for free

Start

In order to better understand the architecture of WeText's entire project and the technologies used, it is recommended to have some understanding of the following in advance:

• Domain Driven Design (DDD)

• Command Query Duty Separation (CQRS)

• Microservice Architecture (MSA)

• Messaging Patterns: You can refer to the official RabbitMQ learning materials: http://www.rabbitmq.com/getstarted.html

Next, the important thing, forget it, just say it again, please use git to clone the project code locally:

git clone https://github.com/daxnet/we-text.git

Then use Visual Studio to open the WeText.sln file directly. After opening the code, don't rush to run it, let's first understand the project structure.

• Services directory: A project containing three microservices: Accouts (user account microservice), Social (social microservice), and Texting (small note microservice)

• WeText.Common project: contains the base library for the entire solution

• WeText.Domain project: domain model and command, event definition

• WeText.DomainRepositories project: specific implementation of domain repository (MongoDB implementation)

• WeText.Messaging.RabbitMq Project: Implementation of RabbitMQ-based Messaging System

• WeText.Querying.MySqlClient project: MySQL-based Table Data Gateway implementation, used to provide CRU operations on MySQL databases (Delete is not supported yet)

• WeText.Querying.PostgreSQL project: PostgreSQL-based Table Data Gateway implementation, used to provide CRU operations on the PostgreSQL database (Delete is not supported yet)

• WeText.Service project: the host program of the microservice, which is a console server program after startup (start this project first at runtime)

• WeText.Tests project: NUnit-based unit test project, please ignore it directly

• WeText.Web project: front-end user interface project, start this project after the WeText.Service project is started

• In addition, under the we-text root directory, there is a subdirectory of scripts, which contains initialization scripts for MySQL and PostgreSQL databases (at the time of writing this article, the PostgreSQL scripts were not added, and will be added later). will be used in the steps

External Dependencies

First, WeText only depends on some relevant libraries required by infrastructure layer facilities, including:

• MySql.Data

• RabbitMQ.Client

• MongoDB

• Npgsql

• Autofac related

• Owin related

• Newtonsoft Json

• log4net

• Libraries related to ASP.NET MVC

Besides, no application layer development framework and code base are used, all code is original and included in the whole WeText solution.

Secondly, the server infrastructure layer completely selects cross-platform projects and products such as Owin, MySQL, RabbitMQ, PostgreSQL, etc., so that the entire WeText server can be fully deployed in the Linux environment (in fact, this is also one that I want to practice. section to verify that Mono-based .NET server programs perform well on Linux systems). There is no use of SQL Server, Entity Framework, which are currently more suitable for products running on the Windows platform.

Install and run

First, for convenience, it is strongly recommended to install all services and programs on the same machine. Please follow the steps below to prepare the system environment:

1. Download the source code (refer to the git command above)

2. Download and install RabbitMQ. The installation process uses the default configuration (including service ports, etc.). For the installation of RabbitMQ, see: https://www.rabbitmq.com/download.html

3. Download and install MongoDB. The installation process uses the default configuration (including service ports, etc.). For the installation of MongoDB, please refer to: https://docs.mongodb.org/manual/installation/. If you need to change the MongoDB configuration of WeText after installation , please move to the MongoSetting.cs file under the WeText.DomainRepositories project (the hard code is still in the code at the time of writing this article, and will be moved to App.config in the future)

4. Download and install MySQL Community Edition (including server and client). The installation process uses the default configuration. Please use P@ssw0rd for the root password. For MySQL installation, please refer to: http://dev.mysql.com/doc/refman/5.7/en/installing.html. If you need to change the MySQL configuration of WeText after installation, please directly modify the App of the WeText.Service project. config file

5. If you plan to use PostgreSQL as the query database, then you only need to install PostgreSQL, not MySQL. Please also use the default configuration during installation

6. Use the SQL script in the scripts directory of the we-text project folder to initialize the corresponding database. The PostgreSQL script has not been added yet, and will be added later

Once the environment is ready, you can try to start the project.

Start and debug the project in Windows

1. Open the WeText.sln file with Visual Studio 2015

2. Start the WeText.Service project and you should see the following screen:

Start the WeText.Web project and you should see the following screen:

4. Try to click Register to register your account and log in

Compile and start the server program in Linux

Note: I have not yet had time to test using the WeText.Web site to access the server deployed on Linux, just trying to compile and start the server program in the Linux environment. The Web site program (WeText.Web) itself is not intended to run in the Linux environment, but you can try it later.

1. To install Mono, follow this page to complete the installation of Mono: http://www.mono-project.com/docs/compiling-mono/linux/. It is recommended to install directly from the Release Package, you can go to http://download.mono-project.com/sources/mono/ to download the latest version of Mono. WeText requires .NET Framework 4.6.1 and C# 6.0 support

2. It is also necessary to prepare the system environment according to the above steps, including the installation and initialization of RabbitMQ, MongoDB and the query database

3. Download the source code using the above git command

4. Due to the current support of nuget under Mono, there are still some problems, and some packages cannot be downloaded. It is recommended to compile WeText under Windows first, and then upload the downloaded packages directory to the we-textsrc directory in Linux.

5. Enter the we-textsrc directory and use this command to complete the compilation: xbuild /p:TargetFrameworkVersion=v4.6.1 /p:Configuration=ServerDebug WeText.sln. Some warnings may appear during compilation, please ignore them for the time being

6. Enter the we-textin directory, execute the ./WeText.Service.exe command, and you should see the following screen:

7. If you want to try to access the service on Linux from the WeText.Web site, temporarily search for the string http://localhost:9023/ in the WeText.Web project, and replace localhost with the service URL of the Linux host.

Summarize

This article first briefly introduces the microservice architecture, and introduces WeText, a demonstration case of WeText, a .NET-based DDD, CQRS, and microservice architecture from the aspects of overall architecture, use of code bases, environment preparation, and compilation and deployment. Friends who are interested in microservices are welcome to try the source code of this case, and welcome to participate in a more in-depth discussion. WeText is still immature at present. I will gradually improve this case, and I will also share my own experience in the process. Welcome everyone to pay attention.