When pointer, when passed by value, the structure, in particular, be in sync with the pointer passed packet; lightweight and some data may be transferred using the value does not change its value;
avoid reflection, high-performance service to eliminate reflections using
multi-multiplexed using sync.Pool
Online optimization problem:
batch reading and writing, the main problem: CPU occupancy is too high, mainly due to release a large amount of resources caused by the application;
Solution: Use "sync.Pool" implements a pool of objects,
P: = {sync.Pool
New: new new BUFF (),
}
. P.Get () (* BUFF)
p.Put (buff.clean ())
Further use goroutine concurrent read and write;
Peak PCT99 services reduced from 100ms to 15ms.
Micro-services, each service has its own database, micro-services division: Membership authentication module, comment module, and other modules thumbs up;
gRPC developed by Google RPC leading frame, using HTTP / 2 protocol and used as the sequence Protobuf tool;
Important:
centralized configuration
service discovery
journal
distributed to explore the
fuse
load balancing
edge
monitoring
safety
Service registration and discovery, API GATEWAY, REST API; message center, RPC API; that is, the use of external service RESTFUL json, internal use protobuf
Center Configuration: etcd
Service discovery: the request to reach the gateway, to know the IP address of the API service is the number; in the production practice, there are two main service discovery mechanism: The client discovery and server discovery.
Client found:
e.g. three examples to the client service registered set of measured (self-registration heartbeat mode), the service may access the service registration form, which is a database of available services, and the client uses a load balancing algorithm ( consistent hashing), select an available service instances and then initiate a request.
Also use etcd
Go kit package, which can be used http, grpc various transmission; Distributed explore: parallel computation partitioning rules; log: GRPC or aggregated kafka; statistics: the number of connections of each method, GC quartile and the like;
Fuse: go-kit of hystrix, mainly includes: 1. Request timeout; 2. the maximum allowed number of concurrent requests; 3. How often blown open attempt to launch a request;
Load balancing: distributed processing with a service code to the three machines, all requests advanced server load balancing,
Network transmission three-tier structure:
HTTP packets: the WWW ... GET
TCP packets: Source Address: 3345 (client), Destination: 80 (SLB)
IP packet: Source Address: ip- client, purpose to: ip- server load balancing
In server load balancing, TCP port, IP address will change the address of the destination city RS1 services;
Seven-story structure is the basis of the three, plus URL, browser and language;
but there is a problem: a short request, but the response returns a html file; let Load Balancer only deal with the request, so that each server the response directly to the client;
Solution:
First of all servers have the same IP, we called him VIP bar (as shown in 115.39.19.22).
Network transmission four-layer structure:
Ethernet frame: Source Address: 11: 27: F5: .. Destinations: ???
the HTTP packets: the GET WWW ...
the TCP packet: source address: 3345 (client), the purpose of to: VIP 80
the IP packet: source address: IP- client, the destination: 115.39.19.22 vip
Use ARP protocol will 115.39.19.22 vip broadcast to go out and have this IP machine will revert its own MAC address.
Such a request can be handed out, the VIP will be able to respond directly to the client;
tcp explain:
now typical model, Non-Block + I / O multiplexing.
Go and developers need not be concerned whether the socket is non-block, and there is no need to personally file descriptor callbacks registered, just treat the socket can be treated in each goroutine to "block I / O" way to connect corresponding;
l, err: = net.Listen ( "tcp", ": 8888") // server listens to this port
{for
C, ERR: l.Accept = () // read here corresponds blocked waiting In Flag;
}
After the processing go handleConn (c) // Get connected
TCP Socket connection to establish a three-way handshake to go through the process of client and server. Connection establishment, the server is a standard Listen + Accept structure;
// client attempts to connect:
conn, ERR: = net.DialTimeout ( "tcp", ": 8080", 2 * time.Second)
During the handshake will experience the following:
1. Network unreachable or other service is not started; immediately return an error;
2. server side listen backlog queue is full;
3. Network delay is large, Dial and blocking timeout
After conn connection, can be used to read and write, conn.Read (), conn.Write ()
the TCP is a full duplex communication, so in each direction independent data buffer. When the sender receives the other side of the buffer and its own transmit buffer is full, Write will be blocked.
Reading and writing are lock safe, and full use of locks to read, write complete;
rpc explain:
RPC (Remote ProcedureCall, Remote Procedure Call), constructed in the UDP or TCP, or on the HTTP, allowing the caller directly on another computer, without additional network traffic associated write the code for the calling procedure.
Use:
In the RPC server, an object can be registered as a service accessible after the disclosed method of the object will be able to provide access to remote way. A RPC server can register multiple objects of different types, but does not allow registration of multiple objects of the same type.
func (t * T) MethodName ( argType T1, replyType * T2) error
The first parameter passed by the client RPC parameter, the second parameter to return the result to the RPC client, the method returns the last error type a value.
RPC call to the client's Call (), synchronous; RPC client calls Go (), asynchronous;
If not specified what codec used during the RPC transport, Go will use the default standard library encoding / gob packet data transmission; Gob encoded binary data stream only go for the language;
arith: clients can be very simple, such as type int or interface {}, it is important that the method of output.
服务端：
rpc.Register(arith)
rpc.HandleHTTP()
l, e := net.Listen("tcp", ":1234")
go http.Serve(l, nil)
客户端：
client, err := rpc.DialHTTP("tcp", serverAddress + ":1234")
args := &server.Args{7,8}
var reply int
err = client.Call("Arith.Multiply", args, &reply) // 使用同步
quotient := new(Quotient)
divCall := client.Go("Arith.Divide", args, "ient, nil) // 使用异步
replyCall := <-divCall.Done
Rpc program load balancing:
Server:
var (
// ETCD service address
etcdServer = "127.0.0.1:2379"
// a directory service
prefix = "Arith.Multiply"
// start the service instance currently address
instance = "127.0.0.1:50052"
/ / registration service instance path
Key = prefix + instance
// service instance registered Val
value = instance
ctx = context.Background ()
// listens service address
serviceAddress = ": 50052"
)
// the connection information, written to the leader etcd;
// Create Registrar
Registrar: = etcdv3.NewRegistrar (Client, etcdv3.Service {
Key: Key,
the Value: value,
}, log.NewNopLogger ())
// Register the start registration
registrar.Register ()
// The method rpc, added to tcp port;
arith: = new new (Arith)
rpc.Register (arith)
rpc.HandleHTTP ()
L, E: = net.Listen ( "tcp", serviceAddress) // use the service registration address
! IF = E {nil
fmt.Print ( "the listen error:", E)
}
http.Serve (L, nil)
type Arith int
func (t *Arith) Multiply(args *Args, reply *int) error {
*reply = args.A * args.B
return nil
}
Client:
// add etcd address, and the name of the service required;
var (
// Registration Center Address
etcdServer = "127.0.0.1:2379"
// listening service prefix
prefix = "Arith.Multiply"
ctx = context.Background ()
)
// create an instance manager, this manager will Watch monitor changes etc directory in the prefix of updating the cache service instance data
instancer, err: = etcdv3.NewInstancer (client , prefix, logger)
// Create the endpoint manager, this manager instance to create endPoint Factory and listen and subscribe to changes according to instancer buzz endPoint Factory created
endpointer: = sd.NewEndpointer (instancer, reqFactory, Logger)
// create a load balancer
balancer : = lb.NewRoundRobin (endpointer)
// func reqFactory (instanceAddr string) in the client, err: = rpc.DialHTTP ( "tcp", instanceAddr), client.Call ( "Arith.Multiply", args, & reply) method which specific call;
MQTT agreement:
MQTT (the Message Queuing Telemetry Transport, Message Queuing Telemetry Transport Protocol), is based publish / subscribe (publish / subscribe) mode "lightweight" communication protocol built on TCP / IP protocol;
Quality of service Qos:
"at most once", the news release completely dependent on the underlying TCP / IP network. Message loss occurs.
"At least once" to ensure that the message reaches, but the message is repeated may occur.
"Only once" to ensure that the message reaches once. In some of the more stringent requirements of the billing system.
Features: Small transmission, only 2 bytes; Last Will: Notify other equipment under the same theme, the device has been disconnected;
Components: publisher (Publish), subscribers (Subscribe), agent (Broker)
publisher and the subscriber is a client, the publisher can be simultaneously subscribers;
proxy server is;
MQTT transmitted into message: topic (Topic) and a load (payload) in two parts;
Theme for subscribers to subscribe, the load can be understood as the content of the message.
MQTT client: 1. Publish information about other clients may subscribe; 2. Subscribe to news published by other clients;
MQTT server (broker): 2. It is located between news publishers and subscribers, from 1 to accept customer network connections; 2. Accept the application of information released by the customer;
Process connection:
Each client and server to establish a session after the connection is, subscriptions will be associated with a session (Session). A session can contain multiple subscriptions.
MQTT protocol packet structure:
fixed header (Fixed header) is connected to the main information, the first variable (Variable header), message body (payload) of three parts.
MQTT agreement is not reciprocal trust, it does not provide the identity of the client-side validation service mechanism.
Installation broker server:
NGINX: Nginx-1.11.0.tar.gz
EMQ: emqx-centos6.8-V3.0-beta.4.x86_64.rpm single node may receive a connection 500,000 to 1,000,000;
emr:
Cluster mode automatically based static cluster node list:
cluster.static.seeds emqx1 @ = 192.168.1.2, 192.168.1.3 emqx2 @
LDAP authentication: Install certificate, service emqx start;
nginx:
http {
server 192.168.1.2:18083;
server 192.168.1.3:18083;
}
Install the client:
连接broker: clinetOptions := mqtt.NewClientOptions().AddBroker("tcp://xxxxx:1883").SetUsername("admin").SetPassword("public")
订阅消息: token := client.Subscribe("go-test-topic", 1, messageSubHandler) token.Wait()
发布消息: token := client.Publish("go-test-topic", 1, false, text) token.Wait()
## Some thought practice:
Machine data -> broker appoll (ssl two-way encryption) -> raw;
at least one of the strategies to prevent data loss, a simple etl on the server;
Limiting and disaster recovery services, limiting memory to prevent the collapse of the blocked queue, disaster recovery: involving state machine, heartbeat, reconnection strategy; maximum load from a state began to run; e-mail downtime status of distribution;
Fuse -> limiting -> appoll -> Services to restore broken state -> nigix -> vue
The concept of
service avalanche effect
reason: Since the time or the load is too high as a result of the backlog of requests, take up a lot of system resources, server to achieve performance bottleneck, the service provider is unavailable
phenomenon: the upstream service failures lead to paralysis of downstream services, cascading failure occurs
coping strategies:
expansion
control traffic
blown
service degradation
hystrix.ConfigureCommand ( "AAA", {hystrix.CommandConfig
the Timeout: 5000,
MaxConcurrentRequests:. 5,
})
. 1, the Timeout] [timeout request
2, ErrorPercentThreshold [allow] error ratio occur
3, SleepWindow [blown open how long to try to launch a request]
4, MaxConcurrentRequests [maximum allowable number of concurrent requests]
5, RequestVolumeThreshold [requests minimum period fluctuation, the fluctuation of the default 10S]