Technical background:
Business system: management of robots, robot task execution, etc.
The robot uses ros1, and the business system interacts with the robot using rosbridge, Rosbridge is a websocket link, so there are some architectural ideas as follows
Architecture diagram
client
The client is mainly divided into app client, pc client, h5 client, etc. Each end will maintain a stable long connection with the gateway. Of course, the multiple windows on the PC side are actually multiple websocket connections.
ingress
Ingress is responsible for the seven-layer domain name, which can be the nginx component or the ingress controller component of k8s.
And at this layer, you can mount the certificate, perform TLS encryption and decryption, and then proxy the long connection to the gateway.
What should be noted here is that the client and ingress layers are wws protocols, and ingress and ws-gateway are ws protocols.
ws-gateway
ws-gateway maintains the ws long connection after all robots log in. Has the following responsibilities:
-
As much as possible, bear the long WS connection of the robot.
-
Perform robot authentication and parse the robot ID and some business IDs.
-
Responsible for distributing robot messages to downstream and notifying server messages to online robots.
Generate socketId and maintain the relationship between socketId and ws connection. The sid here is particularly important in subsequent design.
The design of the gateway should be stable and traffic-free.
ws-api
ws-api is the intermediary between business systems and gateways. In other words, it is a bridge between business systems and online robots.
For the information interaction between ws-api and the gateway, our initial design was ws channel, and then we switched to gRPC-Stream. For thoughts here, you can read some articles about gRPC-Stream websocket.
The responsibilities of ws-api are as follows:
API responsible for notifying online bots of real-time messages.
Responsible for maintaining the mapping relationships of multiple dimensions.
Responsible for message broadcasting, broadcasting a message to each gateway node.
Responsible for maintaining the heartbeat of the connection (implemented through redis key ttl).
Responsible for statistics on indicators such as messages/robot connections.
Responsible for delivering online machine messages to MQ Topic for consumption by downstream business systems.
ws-api and ws-gateway are packaged together, and all services are downstream.
MQ
The message middleware can be RocketMQ or Kafka. It is mainly used to decouple the gateway and business systems.
As for what message middleware to use, it depends on your own needs. Basically, all commonly used ones can be satisfied. It depends on your specific business needs.
Let’s take a look at some business process scenarios:
Business system push frame flow chart
Online robot message push
