So far, the basic content of server-side development has been covered. We have spent a relatively long time introducing the basic software on the server side, including load balancing and various storage middleware. Then our previous lecture introduced some common issues on the business architecture of the server.
Today we get into actual combat.
Comparing the contents of server and desktop, we can see that server-side development and desktop-side development each have their own complexity. Server-side development is difficult because there is a lot of basic software, which requires programmers and architects to have high knowledge and depth of understanding. But in terms of business complexity, the business logic on the server side is relatively simple. On the contrary, desktop development is difficult because of the complex user interaction logic, large amount of code, and high complexity of the business architecture.
Many people reported that the practical part of the previous chapter was a bit difficult. This is to some extent related to the planning of our course content design. In the last chapter, from an architectural perspective, we focused on introducing the outline design, that is, the system architecture. Therefore, we did not analyze the implementation details too much, but focused on the interface coupling between modules. This is to hope that you focus on the overall situation instead of getting into local details right away. However, due to the lack of analysis of the complete process, we are unable to string together the entire process, and our understanding will be compromised.
In this chapter, we will focus on detailed design in terms of architecture. This will also be reflected in the actual combat chapter.
In the previous chapter, we implemented a mock version of the server. The code is as follows:
https://github.com/qiniu/qpaint/tree/v31/paintdom
Next, we will turn it into a production-level server program step by step.
RPC framework
In the first step, we introduce the RPC framework.
In order to facilitate understanding, in the actual combat in the previous chapter, our mock server program did not introduce any non-standard library content. code show as below:
https://github.com/qiniu/qpaint/blob/v31/paintdom/service.go
The entire Service is about 280 lines of code.
We implemented it based on Qiniu Cloud’s open source restrpc framework. The code is as follows:
https://github.com/qiniu/qpaint/blob/v41/paintdom/service.go
In this way, the entire Service only has about 163 lines of code, less than 60% of the original one.
What code was written less? Let's take a look at creating a new graphic. Originally we wrote this:
func (p *Service) PostShapes(w http.ResponseWriter, req *http.Request, args []string) {
id := args[0]
drawing, err := p.doc.Get(id)
if err != nil {
ReplyError(w, err)
return
}
var aShape serviceShape
err = json.NewDecoder(req.Body).Decode(&aShape)
if err != nil {
ReplyError(w, err)
return
}
err = drawing.Add(aShape.Get())
if err != nil {
ReplyError(w, err)
return
}
ReplyCode(w, 200)
}Now write this:
func (p *Service) PostShapes(aShape *serviceShape, env *restrpc.Env) (err error) {
id := env.Args[0]
drawing, err := p.doc.Get(id)
if err != nil {
return
}
return drawing.Add(aShape.Get())
}The return package in this example is relatively simple, without the body of the HTTP package.
Let's look at another example of a more complex return package, taking the content of the graphic. Originally we wrote this:
func (p *Service) GetShape(w http.ResponseWriter, req *http.Request, args []string) {
id := args[0]
drawing, err := p.doc.Get(id)
if err != nil {
ReplyError(w, err)
return
}
shapeID := args[1]
shape, err := drawing.Get(shapeID)
if err != nil {
ReplyError(w, err)
return
}
Reply(w, 200, shape)
}Now write this:
func (p *Service) GetShape(env *restrpc.Env) (shape Shape, err error) {
id := env.Args[0]
drawing, err := p.doc.Get(id)
if err != nil {
return
}
shapeID := env.Args[1]
return drawing.Get(shapeID)
}Comparing these two examples, we can see:
Originally, the values of URL parameters such as DrawingID and ShapeID in these two requests POST /drawings/<DrawingID>/shapes and GET /drawings/<DrawingID>/shapes/ were passed through the parameters args[0] and args[1] Pass in, now pass in env.Args[0], env.Args[1].
Originally, our PostShapes needed to define the Shape instance ourselves and parse the contents of the HTTP request package req.Body. Now we only need to specify the Shape type in the parameters, and the restrpc framework will automatically complete the parsing of the parameters.
Originally, our GetShape needed to reply to the error itself or return a normal HTTP protocol packet. Now we only need to return the data to be replied in the return value list, and the restrpc framework automatically completes the serialization of the return value and replies to the HTTP request.
By comparing the code differences between the two versions, we can roughly guess what is done behind the HTTP processing function of restrpc. Its core code is as follows:
https://github.com/qiniu/http/blob/v2.0.2/rpcutil/rpc_util.go#L96
What is worth paying attention to is the support of Env. The RPC framework does not limit what the Env class looks like, but only stipulates that it needs to meet the following interfaces:
type itfEnv interface {
OpenEnv(rcvr interface{}, w *http.ResponseWriter, req *http.Request) error
CloseEnv()
}In the OpenEnv method, we generally initialize Env. The CloseEnv method does the opposite. Why is the ResponseWriter interface passed in as a pointer in the OpenEnv method? Because there may be customers who want to rewrite the implementation of ResponseWriter.
For example, suppose we want to extend the API audit log function to the RPC framework. Then we need to take over and record the HTTP packets returned by the user. At this time, we need to rewrite the ResponseWriter to achieve the purpose of taking over and recording.
It is also worth noting that the restrpc version of the HTTP request processing function no longer looks like an HTTP processing function, but like an ordinary function.
This means we can test the Service class in two ways. In addition to testing HTTP Service using the normal method, we can also test the Service class as a normal class, which greatly reduces the cost of unit testing. Because we no longer need to wrap the Client SDK of the service, and then do unit testing based on the Client SDK.
Of course, we have such a low-cost testing method, but we are still worried that this testing method may not cover some small accidents in coding. After all, we do not use the HTTP protocol, and we feel a little uneasy.
After understanding the HTTP processing function of restrpc, the rest is the routing function of restrpc. It is done by the Register function of the restrpc.Router class. code show as below:
https://github.com/qiniu/http/blob/v2.0.1/restrpc/restroute.go#L39
It supports two routing methods, one is automatic routing based on the method name. For example, a request such as POST /drawings/<DrawingID>/shapes requires the method name to be "PostDrawings_Shapes". A request like GET /drawings/<DrawingID>/shapes/ requires the method name "GetDrawings_Shapes_".
The rules are relatively simple. "/" in the path are separated by capital letters, and URL parameters such as DrawingID and ShapeID are replaced with "_".
Of course, some people will think that the name of this method looks ugly. Then you can choose manual routing and pass in routeTable. It looks like this:
var routeTable = [][2]string{
{"POST /drawings", "PostDrawings"},
{"GET /drawings/*", "GetDrawing"},
{"DELETE /drawings/*", "DeleteDrawing"},
{"POST /drawings/*/sync", "PostDrawingSync"},
{"POST /drawings/*/shapes", "PostShapes"},
{"GET /drawings/*/shapes/*", "GetShape"},
{"POST /drawings/*/shapes/*", "PostShape"},
{"DELETE /drawings/*/shapes/*", "DeleteShape"},
}Although it is manual routing, there are still restrictions on method names, which must start with Get, Put, Post, or Delete.
Layering of business logic
After understanding the restrpc framework, let's take a look at the business itself of the QPaint server. It can be seen that our server-side business logic is divided into two layers: one is the implementation layer of the business logic, which we usually organize consciously into a DOM tree. code show as below:
https://github.com/qiniu/qpaint/blob/v41/paintdom/drawing.go
https://github.com/qiniu/qpaint/blob/v41/paintdom/shape.go
The other layer is the RESTful API layer, which is responsible for receiving user network requests and converting them into method calls on the underlying DOM tree. With the restrpc framework we introduced above, each method in this layer is often relatively simple, and some are even just a simple function call. for example:
func (p *Service) DeleteDrawing(env *restrpc.Env) (err error) {
id := env.Args[0]
return p.doc.Delete(id)
}The complete RESTful API layer code is as follows:
https://github.com/qiniu/qpaint/blob/v41/paintdom/service.go
The reason for this layering is because when we implement core business logic, we do not assume that it must be exposed through RESTful API. We consider the following possibilities:
For one, it's possible that we don't need the network call at all.
To make an analogy, we all know that mysql provides service interfaces through the TCP protocol, while sqlite is an embedded database and provides service interfaces through local function calls. The layering here is similar to when I implemented mysql. I first implemented an embedded database similar to sqlite at the bottom layer, and then provided a network interface based on the TCP protocol.
Second, it is possible that we need to support many network protocols.
RESTful API is popular today, so our interface is RESTful style. If one day we want to switch to GraphQL like Github, then at least the underlying business logic implementation layer does not need to be changed, we only need to implement a relatively thin GraphQL layer.
Moreover, often in this case RESTful API and GraphQL need to be supported at the same time. After all, we cannot abandon old users just to follow the trend.
When multiple sets of network interfaces need to be supported at the same time, the value of this layering is reflected. The modules of different network interfaces share instances of the same DOM tree. The entire system not only realizes the coexistence of multiple protocols, but also realizes Perfectly decoupled and completely independent of each other.
unit test
After talking about business, let’s take a look at unit testing.
Before, our unit testing basically didn’t do much:
https://github.com/qiniu/qpaint/blob/v31/paintdom/service_test.go#L62
code show as below:
type idRet struct {
ID string `json:"id"`
}
func TestNewDrawing(t *testing.T) {
...
var ret idRet
err := Post(&ret, ts.URL + "/drawings", "")
if err != nil {
t.Fatal("Post /drawings failed:", err)
}
if ret.ID != "10001" {
t.Log("new drawing id:", ret.ID)
}
}As can be seen from the test code here, we just created a drawing and requested that the returned drawingID be "10001".
From the perspective of unit testing, such testing intensity is of course very insufficient. The same test case is implemented using the httptest test framework we introduced in the previous lecture:
func TestNewDrawing(t *testing.T) {
...
ctx := httptest.New(t)
ctx.Exec(
`
post http://qpaint.com/drawings
ret 200
json '{"id": "10001"}'
`)
}Of course, in reality we should test more situations, such as:
func TestService(t *testing.T) {
...
ctx := httptest.New(t)
ctx.Exec(
`
post http://qpaint.com/drawings
ret 200
json '{
"id": $(id1)
}'
match $(line1) '{
"id": "1",
"line": {
"pt1": {"x": 2.0, "y": 3.0},
"pt2": {"x": 15.0, "y": 30.0},
"style": {
"lineWidth": 3,
"lineColor": "red"
}
}
}'
post http://qpaint.com/drawings/$(id1)/shapes
json $(line1)
ret 200
get http://qpaint.com/drawings/$(id1)/shapes/1
ret 200
json $(line1)
`)
if !ctx.GetVar("id1").Equal("10001") {
t.Fatal(`$(id1) != "10001"`)
}
}What do we want to demonstrate in this case? This is a relatively complex case. First we create a drawing and put the drawingID into the variable $(id1). We then add a line $(line1) to the drawing. In order to confirm that the addition is successful, we take out the graphic object and determine whether the obtained graphic is consistent with the added $(line1).
In addition, it also demonstrates the interoperability of qiniutest DSL scripts and Go language code. We use Go code to get the variable $(id1) and determine whether it is equal to "10001".
For more information about qiniutest, please check the following information:
https://github.com/qiniu/httptest
https://github.com/qiniu/qiniutest
Speech transcript: http://open.qiniudn.com/qiniutest.pdf
In our test code, we also used an open source mockhttp component from Qiniu Cloud, which is also very interesting:
https://github.com/qiniu/x/blob/v8.0.1/mockhttp/mockhttp.go
This mockhttp does not really monitor the port. Interested students can study it.
Conclusion
Let’s summarize today’s content. Starting today, we will step by step transform the mock server we wrote before into a real server program.
Our first step to transform is the RPC framework and unit testing. In this way, we started to rely on third-party code libraries for the first time, as follows:
http://github.com/qiniu/http (restrpc used)
http://github.com/qiniu/qiniutest
http://github.com/qiniu/x (use mockhttp)
Once there are external dependencies, we need to consider version management of the dependent libraries. The good thing is that most modern languages have good version management specifications. For the Go language, we use go mod for version management.