The Internet of Vehicles is a major application direction of the Internet of Things. Vehicles can interact with messages in real time by connecting to the Internet of Vehicles platform. The platform can provide various functions such as vehicle remote control, fault detection, and vehicle-road collaboration.
I have been engaged in technical work in the Internet of Vehicles industry for a long time and participated in the construction of the entire Internet of Vehicles platform and the development of many different Internet of Vehicles applications. Here I intend to use the construction of an Internet of Vehicles platform as an example to summarize the architectural design involved. aspects of things.
System functions
An Internet of Vehicles platform needs to implement some of the following functions:
1. Message interaction with vehicles
In the Internet of Things, the main communication protocol used is MQTT, which is an Internet of Things communication protocol based on the publish/subscribe model. It has the characteristics of supporting QoS, being simple and easy to implement, and having compact messages. In the Internet of Vehicles, most car companies also use the MQTT protocol for communication. Therefore, in the architecture of the Internet of Vehicles, we need to consider setting up an MQTT Broker cluster to connect and communicate with a large number of vehicles. There are currently many open source Brokers, such as ActiveMQ, EMQ, RocketMQ, etc. Among them, EMQ and RocketMQ are domestic products and have detailed Chinese information. Here I choose EMQ as the MQTT Broker.
2. Consumption and storage of vehicle messages
After MQTT Broker receives the vehicle's message, it needs to pass the message to the upper-layer application for processing. We can save these messages to the database or forward them to a message queue for caching. Here I choose Kafka. The upper-layer application subscribes to the Kafka topic to obtain the relevant vehicle information it needs for processing. The upper-layer application can also send the message to be sent to the vehicle to the Kafka topic, and then let the MQTT Broker forward it to the vehicle. It can also directly send the message to the vehicle by publishing the message on the MQTT topic.
3. V2X applications
Including V2V, V2I, V2P and other application scenarios, vehicles need to be able to interact with different data sources to provide decision-making information for driving. I will show the development and design of some V2X applications based on this platform and implement some V2X scenarios specified in the 3GPP specifications.
4. Vehicle data analysis and reports
The Internet of Vehicles platform collects and generates a large amount of data every day. By exploring and analyzing this data, we can better understand the business operation and provide a better reference for business decisions. We can process the data in real time based on currently popular big data processing platforms, such as Spark/Beam/Flink, etc., save it to the data warehouse, and then perform various data analysis and report presentation.
In this article, I will first introduce the first function mentioned above and build an MQTT messaging platform.
MQTT messaging platform
I chose EMQX to build this platform. EMQX is an excellent MQTT broker software in China. It has an enterprise version and an open source version. Here I choose the open source version. The installation method is introduced on the official website. On Kubernetes, it is installed using the Operator method, but I use the kustomization method to install it here because it makes it easier for me to make some settings changes. I started a kubernetes cluster locally using minikube.
Install EMQX cluster
Define a new namespace
apiVersion: v1
kind: Namespace
metadata:
name: emqxCreate a service account for this namespace and grant relevant permissions
apiVersion: v1
kind: ServiceAccount
metadata:
namespace: emqx
name: emqx
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: emqx
name: emqx
rules:
- apiGroups:
- ""
resources:
- endpoints
verbs:
- get
- watch
- list
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
namespace: emqx
name: emqx
subjects:
- kind: ServiceAccount
name: emqx
namespace: emqx
roleRef:
kind: Role
name: emqx
apiGroup: rbac.authorization.k8s.ioDefine a configmap because we want to create multiple emqx nodes of a statefulset and implement the function of auto cluster to automatically form these multiple nodes into a cluster, so we need to define relevant configurations:
apiVersion: v1
kind: ConfigMap
metadata:
name: emqx-config
namespace: emqx
data:
EMQX_NAME: "emqx"
EMQX_CLUSTER__DISCOVERY_STRATEGY: "k8s"
EMQX_CLUSTER__K8S__SERVICE_NAME: "emqx-headless"
EMQX_CLUSTER__K8S__NAMESPACE: "emqx"
EMQX_CLUSTER__K8S__ADDRESS_TYPE: "hostname"
EMQX_CLUSTER__K8S__APISERVER: "https://kubernetes.default.svc:443"
EMQX_CLUSTER__K8S__SUFFIX: "svc.cluster.local"Define a headless service for statefulset service exposure and communication.
apiVersion: v1
kind: Service
metadata:
name: emqx-headless
namespace: emqx
spec:
type: ClusterIP
clusterIP: None
selector:
app: emqx
ports:
- name: mqtt
port: 1883
protocol: TCP
targetPort: 1883
- name: mqttssl
port: 8883
protocol: TCP
targetPort: 8883
- name: mgmt
port: 8081
protocol: TCP
targetPort: 8081
- name: websocket
port: 8083
protocol: TCP
targetPort: 8083
- name: wss
port: 8084
protocol: TCP
targetPort: 8084
- name: dashboard
port: 18083
protocol: TCP
targetPort: 18083Finally, a statefulset is defined, which contains 2 nodes.
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: emqx-statefulset
labels:
app: emqx
namespace: emqx
spec:
serviceName: emqx-headless
updateStrategy:
type: RollingUpdate
replicas: 2
selector:
matchLabels:
app: emqx
template:
metadata:
labels:
app: emqx
spec:
serviceAccountName: emqx
containers:
- name: emqx
image: emqx/emqx:5.1.6
resources:
requests:
memory: "1Gi"
cpu: "250m"
limits:
memory: "1Gi"
cpu: "250m"
ports:
- name: mqtt
containerPort: 1883
- name: mqttssl
containerPort: 8883
- name: mgmt
containerPort: 8081
- name: ws
containerPort: 8083
- name: wss
containerPort: 8084
- name: dashboard
containerPort: 18083
envFrom:
- configMapRef:
name: emqx-configDefine a kustomization.yaml file and include the manifest defined above:
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- namespace.yaml
- rbac.yaml
- configmap.yaml
- headless.yaml
- statefulset.yamlFinally, run kubectl apply -k to deploy. We can run the following command to view the status of emqx cluster:
kubectl exec emqx-statefulset-0 -n emqx -- emqx_ctl cluster statusIf it runs successfully, the following information will be displayed:
Cluster status: #{running_nodes =>
['[email protected]',
'[email protected]'],
stopped_nodes => []}It can be seen that the current EMQX cluster includes two nodes and has been running successfully.
Configure HAProxy
Next I will configure a HAProxy as a load balancer to connect to the EMQX cluster. This approach can provide the following benefits:
- As a reverse proxy, HAProxy can hide the information of emqx nodes and provide a unified address for the outside world to connect to.
- It can be used as the end of MQTT over TLS, reducing the computational load of emqx nodes processing SSL encryption, and simplifying certificate deployment and management.
- Provides built-in MQTT support and supports parsing MQTT messages to implement features such as sticky attachment and intelligent load distribution.
- Provide high reliability through active and backup mode
Similarly, I also deploy HAProxy using kustomization.
Define a namespace
apiVersion: v1
kind: Namespace
metadata:
name: haproxyDefine a configmap, because haproxy needs to read the information of the haproxy.cfg configuration file when starting it, and load this file through configmap.
apiVersion: v1
kind: ConfigMap
metadata:
name: haproxy-config
namespace: haproxy
data:
haproxy.cfg: |
global
log 127.0.0.1 local3 info
daemon
maxconn 10240
defaults
log global
mode tcp
option tcplog
#option dontlognull
timeout connect 10000
# timeout > mqtt's keepalive * 1.2
timeout client 240s
timeout server 240s
maxconn 20000
backend mqtt_backend
mode tcp
# 粘性会话负载均衡
stick-table type string len 32 size 1000k expire 30m
stick on req.payload(0,0),mqtt_field_value(connect,client_identifier)
server emqx0 emqx-statefulset-0.emqx-headless.emqx.svc.cluster.local:1883
server emqx1 emqx-statefulset-1.emqx-headless.emqx.svc.cluster.local:1883
frontend mqtt_servers
bind *:1883
mode tcp
# 拒绝非 MQTT 连接
# tcp-request content reject unless { req.payload(0,0),mqtt_is_valid }
default_backend mqtt_backend
Define a deployment
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: haproxy
name: haproxy
namespace: haproxy
spec:
replicas: 1
selector:
matchLabels:
app: haproxy
template:
metadata:
labels:
app: haproxy
spec:
containers:
- name: haproxy
image: haproxy:2.8
ports:
- name: http
containerPort: 80
- name: https
containerPort: 443
- name: haproxy-mgmt
containerPort: 1024
- name: mqtt
containerPort: 1883
- name: mqttssl
containerPort: 8883
- name: mgmt
containerPort: 8081
- name: ws
containerPort: 8083
- name: wss
containerPort: 8084
- name: dashboard
containerPort: 18083
volumeMounts:
- name: haproxy-config
mountPath: /usr/local/etc/haproxy/haproxy.cfg
subPath: haproxy.cfg
volumes:
- name: haproxy-config
configMap:
name: haproxy-config
items:
- key: haproxy.cfg
path: haproxy.cfgDefine a service to expose the port of harpoxy
apiVersion: v1
kind: Service
metadata:
name: haproxy-service
namespace: haproxy
spec:
selector:
app: haproxy
ports:
- name: mqtt
port: 1883
protocol: TCP
targetPort: mqtt
- name: mqtts
port: 8883
protocol: TCP
targetPort: 8883
- name: ws
port: 8083
protocol: TCP
targetPort: 8083
- name: wss
port: 8084
protocol: TCP
targetPort: 8084
- name: dashboard
port: 18083
protocol: TCP
targetPort: 18083Finally define a kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- namespace.yaml
- configmap.yaml
- haproxy_deployment.yaml
- service.yamlRun kubectl apply -k to deploy
Configure Ingress
Expose the HAProxy port on my minikube k8s cluster so that MQTT can be accessed externally. Because I want to still expose port 1883 to external access, I need to set it when minikube starts.
minikube start --extra-config=apiserver.service-node-port-range=1-65535Then install HAProxy ingress and install it through helm
helm repo add haproxytech https://haproxytech.github.io/helm-charts
helm repo update
helm install haproxy-kubernetes-ingress haproxytech/kubernetes-ingress \
--create-namespace \
--namespace haproxy-controllerAfter the installation is complete, we need to create a configmap, configure the TCP ports to be exposed, and deploy it through kubectl apply -f.
apiVersion: v1
kind: ConfigMap
metadata:
name: tcp
namespace: haproxy
data:
1883:
haproxy/haproxy-service:1883
8883:
haproxy/haproxy-service:8883
8083:
haproxy/haproxy-service:8083
8084:
haproxy/haproxy-service:8084
18083:
haproxy/haproxy-service:18083Read the configuration information of HAProxy ingress and save it in the values.yaml file
helm show values haproxytech/kubernetes-ingress > values.yamlThen find the following corresponding location in values.yaml and modify it:
tcpPorts:
- name: mqtt
port: 1883
targetPort: 1883
nodePort: 1883
- name: mqtts
port: 8883
targetPort: 8883
nodePort: 8883
- name: ws
port: 8083
targetPort: 8083
nodePort: 8083
- name: wss
port: 8084
targetPort: 8084
nodePort: 8084
- name: dashboard
port: 18083
targetPort: 18083
nodePort: 18083
# add extra args in controller section
extraArgs:
- --configmap-tcp-services=haproxy/tcpRun the following command to update the configuration of haproxy-ingress
helm upgrade -f values.yaml haproxy-kubernetes-ingress -n haproxy-controller haproxytech/kubernetes-ingressNow we can connect to EMQX through HAPROXY through an MQTT client. The server address is minikubeip:1883. You can access the EMQX dashboard by accessing minikubeip:18083.
Configure certificate
In practical applications, vehicles and platforms communicate through TLS encryption, with two methods: one-way authentication and two-way authentication. One-way authentication means that the client needs to verify whether the server holds a trusted certificate, while two-way authentication means that both parties need to verify. Here we take two-way authentication as an example for configuration.
1. Create a root CA certificate
To do it with a self-signed certificate, first create a root CA certificate, such as the following command.
openssl req -newkey rsa:2048 -nodes -x509 -days 3650 -keyout root-ca.key -out root-ca.crtUse the following command to view the contents of the created certificate
openssl x509 -noout -text -in root-ca.crt2. Issue a certificate for the client
With the intermediate CA certificate, we can create a certificate for the client, for example, issue a certificate for a vehicle with ID vehicle-1.
openssl req -newkey rsa:2048 -nodes -days 365 -subj "/CN=vehicle-1/O=vehicle" -keyout client.key -out client.csrCreate an extension file client-cert-extensions.cnf and declare it to be client certificate type
basicConstraints = CA:FALSE
keyUsage = digitalSignature
extendedKeyUsage = clientAuth
subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuerThen use the intermediate CA certificate to issue
openssl x509 -req -in client.csr -out client.crt -CA intermediate-ca.crt -CAkey intermediate-ca.key -CAcreateserial -days 365 -extfile client-cert-extensions.cnf3. Issue a certificate for the server
In the same way, a certificate is also issued for the server. The slight difference here is that the IP or DNS address of the server needs to be specified in the certificate.
First create the private key of the server
openssl genrsa -out emqx.key 2048Create an extension file openssl.cnf, in which distinguished_name can be set according to your own needs, and the server address is set in IP.1 and DNS.1. For example, I set Minikube’s IP on IP.1
[req]
default_bits = 2048
distinguished_name = req_distinguished_name
req_extensions = req_ext
x509_extensions = v3_req
prompt = no
[req_distinguished_name]
countryName = CN
stateOrProvinceName = Zhejiang
localityName = Hangzhou
organizationName = EMQX
commonName = Server certificate
[req_ext]
subjectAltName = @alt_names
[v3_req]
subjectAltName = @alt_names
[alt_names]
IP.1 = BROKER_ADDRESS
DNS.1 = BROKER_ADDRESSGenerate a csr requesting the issuance of a certificate
openssl req -new -key ./emqx.key -config openssl.cnf -out emqx.csrSigned with root CA certificate
openssl x509 -req -in ./emqx.csr -CA root-ca.crt -CAkey root-ca.key -CAcreateserial -out emqx.crt -days 3650 -sha256 -extensions v3_req -extfile openssl.cnfThen combine emqx.crt and emqx.key into one file
cat emqx.crt emqx.key > server.pem4. Create a secret
Save the contents of the emqx.pem and root-ca.crt files you just created into secret and define a yaml
apiVersion: v1
kind: Secret
metadata:
name: haproxy-secret
namespace: haproxy
type: Opaque
stringData:
server.pem: |
-----BEGIN CERTIFICATE-----
XXXX
-----END CERTIFICATE-----
-----BEGIN RSA PRIVATE KEY-----
XXXX
-----END RSA PRIVATE KEY-----
cacert.pem: |
-----BEGIN CERTIFICATE-----
XXXX
-----END CERTIFICATE-----5. Modify haproxy deployment
Modify the previously defined haproxy deployment manifest and add a secret type volume to Volumes
- name: haproxy-secret
secret:
secretName: haproxy-secret
items:
- key: server.pem
path: server.pem
- key: cacert.pem
path: cacert.pemAdd mounts in volumeMounts
- name: haproxy-secret
mountPath: /etc/haproxy/certs/cacert.pem
subPath: cacert.pem
- name: haproxy-secret
mountPath: /etc/haproxy/certs/server.pem
subPath: server.pem6. Modify HAproxy.cfg configuration
In the configmap just defined, modify the configuration of HAproxy.cfg and add a mqtt_tls_frontend configuration to enable two-way authentication.
frontend mqtt_tls_frontend
# bind *:8883 ssl crt /etc/haproxy/certs/server.pem
# 双向认证
bind *:8883 ssl crt /etc/haproxy/certs/server.pem ca-file /etc/haproxy/certs/cacert.pem verify required
mode tcp
default_backend mqtt_backendFinally, redeploy HAproxy to take effect.
Connection check
We can use mqttx to test connecting to the MQTT messaging platform
Run the following command to subscribe to messages on the hello topic. You can see that the connection is successful.
mqttx sub -t "hello" -h "minikubeip" -p 8883 -l "mqtts" --key "client.key" --cert "client.crt" --ca "root-ca.crt"To send a message, you can test it with the following command
mqttx pub -t "hello" -m "test message" -h "minikubeip" -p 8883 -l "mqtts" --key "client.key" --cert "client.crt" --ca "root-ca.crt"We have successfully built an EMQX messaging platform that can implement MQTT message communication with vehicles. In the next blog, I will introduce how to bridge the messages received by EMQX to the Kafka message queue.