Introduction to Mainflux, an open source IoT platform

Mainflux Study Notes

  Mainflux is an open source IoT cloud platform developed and maintained by a French start-up company. Mainflux uses the Go language and microservice framework. Mainflux supports a variety of access devices, including devices, users, and APPs; supports multiple protocols, including HTTP, MQTT, WebSocket, and CoAP, and supports protocol conversion between them.
  The southbound interface of Mainflux is connected to the device, and the northbound interface is connected to the application, providing a complete solution from the equipment end to the platform to the application.

overview

Mainflux provides the following services:

• Provide message switching between devices and applications, including HTTP, MQTT, WS, CoAP;
• System Management;
• Device management - southbound interface;
• Application management - northbound interface;
• User Management;
• Data storage - time series database;
• Complicated message processing, based on the system event bus NATS, triggers events according to the time data flow through the rule engine;
  

Message reversal and protocol conversion

Mainflux can realize the communication between devices and applications, complete the upload and delivery of messages; or the communication between devices and devices, applications and applications.

System Management

• User
  Users refer to people who use Mainflux, including multiple roles (administrators, application creators, developers), and if these roles have permissions, they will manage devices, applications, and channels.
• Client
  Clients are mainly divided into two categories, one is equipment; the other is application;
• Devices
  are mainly devices with limited power consumption, limited memory, and limited computing power. These devices tend to use MQTT and CoAP protocols, and use DTLS instead of TLS for encryption. For such devices, Mainflux mainly provides MQTT and CoAP to connect devices. For some devices with rich computing resources, HTTP and WebSocket protocols are provided.
• Application
  The application is mainly run on the business server, and the connection of the application is mainly through HTTP REST or WebSockets, which can communicate in two directions. Of course, applications can also choose protocols such as MQTT or CoAP.
• Channel (connection)
  Channel is an important communication model in Mainflux. It is an expression of two-way message flow. It can be understood as a connection. The specific performance is MQTT topics. Devices and applications can subscribe or publish on a message channel. topic. All data streams flowing through the channel are persisted in the database.
• Device Management
  Mainflux provides a model for describing the devices in the system, and the model can represent the status of the devices. Every device connected to Mainflux has its own model in the database, which is updated by the device itself. If a device needs to update its status, just send an UPDATE request to Mainflux directly, and this update will also be updated for the application or other devices that need to obtain its information, and vice versa.
  Mainflux maintains all these internal device models, keeps connected, and manages these devices, getting their information:
  • How many devices are connected;
  • where they are distributed;
  • What is the firmware version;
  • How is the state of the battery;
  • What is the serial number of the device;
    Mainflux can also send various commands to the device:
  • turn the device on or off;
  • OTA upgrade;
  • Device grouping or changing access rights;
  • …
• User and Application Management
  Mainflux is a multi-user and multi-organization application management platform. Developers of Mainflux can create multi-user applications based on Mainflux without having to manage these users themselves, which means that Mainflux can help applications manage users. (The article says that it is similar to Twitter's application creation process)

time series database

Mainflux uses the NoSQL distributed database Cassandra to store event data, and industrial IoT is mostly instrument data, which can be very helpful for data storage and fast complex queries.

time processing engine

Complex Event Processing (CEP) is a method used to track and analyze data flow and draw some conclusions. Mainflux uses Spark for data transformation and Cassandra for storage.

Event processing and system Dashboard

Mainflux's messaging subsystem is an asynchronous event system that handles messages from sensors and applications. The message mechanism uses NATS and uses the pub/sub mechanism.
Mainflux forms a message conversion center through NATS, which means that a topic subscribed to an MQTT can receive messages from HTTP POST on the same topic.
At the same time, NATS will receive these events through Writer's microservice, and store them in the Cassandra database in a suitable format.
Mainflux's message format uses SenML syntax, which is a standard message model.

SenML

SenML is an IETF standard that provides a simple model for receiving sensor data and controlling some devices. SenML can represent data and metadata (metadata) through minimal sentences and some extensions and links.
example

[
  {
    "bn" : "urn:dev:ow:10e2073a0108006;",
    "bt" : 1.276020076001e+09,
    "bu":"A",
    "bver" : 5,
    "n" : "voltage",
    "u" : "V",
    "v" : 120.1
  },
  {"n" : "current", "t" : −5, "v" : 1.2},
  {"n" : "current", "t" : −4, "v" : 1.3},
  {"n" : "current", "t" : −3, "v" : 1.4},
  {"n" : "current", "t" : −2, "v" : 1.5},
  {"n" : "current", "t" : −1, "v" : 1.6},
  {"n" : "current", "v" : 1.7}
]

Mainflux Message

Mainflux wraps SenML data from sensors in the following format:

type RawMessage struct {
   Channel string `json:"channel"`
   Publisher string `json:"publisher"`
   Protocol string `json:"protocol"`
   ContentType string `json:"content_type"`
   Payload []byte `json:"payload"`
}

Normalization

Mainflux's Writer microservice subscribes to all events from the protocol conversion center in NATS. The message received by the Writer microservice uses the internal message format of Mainflux, parses the RAW data, and obtains a useful SenML payload.
The aggregation of these SenML can be standardized into time series messages, and the Writer microservice is responsible for processing the standardization of SenML and writing to the time series database.
What is finally written to the database

type Message struct {
	Channel string
	Publisher string
	Protocol string
	Version int `json:"bver,omitempty"`
	Name string `json:"n,omitempty"`
	Unit string `json:"u,omitempty"`
	Value float64 `json:"v,omitempty"`
	StringValue string `json:"vs,omitempty"`
	BoolValue bool `json:"vb,omitempty"`
	DataValue string `json:"vd,omitempty"`
	ValueSum float64 `json:"s,omitempty"`
	Time float64 `json:"t,omitempty"`
	UpdateTime float64 `json:"ut,omitempty"`
	Link string `json:"l,omitempty"`
}

Security

Mainflux is based on ACL (Access Control List) and customer role security mechanism. Also provide encrypted communication mechanisms, such as TLS1.3; use DTLS for UDP-based CoAP protocols.

The management service (Manager service) and reverse proxy (nginx) will maintain the security mechanism. The management service provides authentication mechanism through JWT, and provides authorization service according to the authority list in the database.

nginx serves several important roles in Mainflux:

• reverse proxy
• TLS termination
• load balancing
• High availability (scalability)

Authentication (AuthX)

The Manager service provides the creation of system instances and manages their permissions. During the instance creation process, each user and client (device and application) will be assigned a token in JWT format.
However, the JWT mechanism of Mainflux has some differences in the expiration time and TTL (time-to-live): 1. The JWT for the application is stateless and the TTL is very short; 2. The JWT for the device will never expire;

Generic JWTs need to be stateless, with no recycling mechanism. However, it is difficult for some devices with limited resources to refresh the JWT token, because the computing power required to decode and obtain the validity period may not be satisfied; so for such devices, JWT is a simple bearer token, a key string and pre-burned Recorded in the firmware, and there are necessary physical means to protect the key string, such as anti-tampering measures.
The token of this JWT is stateless and recyclable.

In addition, in addition to JWT, the public certificate of the server should also be burned into the firmware of the device, which is mainly used to create a TLS connection.

Authorization (AuthZ)

Management service (Manager service) provides authorization and authentication services.
Authorization service, 1. Confirm the permission to operate the device and application; 2. Execute the right, assign the device or application to the communication group corresponding to the same channel, and those without permission will not be able to operate the channel.

Building an End-to-End Example

Freeflex

system name

Hardware Sensor

ESP32 microprocessor

• PIR (presence) sensor
• Temperature and humidity sensor
• Noise sensor (microphone)
• Light sensor
• Air quality sensor
• NFC and RFID transceiver

Connect to Mainflux

EACH DEVICE
protocol: MQTT
message format: SenML
Channel:

• message channel
• Command channel: the application sends commands to the device
• "bootstrap" channel: device management and configuration, shared by all such devices, used to get initial configuration.
  Mass production and delivery steps:

1. The factory uses the same firmware to burn;
2. The device is connected to Mainflux for the first time, and the initial configuration for each device is obtained through the "bootstrap" channel;
3. Mainflux sends the device model and channel name (message channel and command channel);
4. Mainflux sends the device UUID, authentication certificate, and channel UUID, and the device is stored in its own flash;

Apps and Monitors

Backend: NodeJS+MQTT+MongoDB
Frontend: AngularJS
Function: System Management + Data Visualization

• Device List
• connection list
• channel list
• Versatile visualization of sensors
• Visual display, intuitively display the sensor installation location and occupancy of the floor

LoRa Water Meter

Mainflux inherits a LoRaWAN server, the main components: gateway, base station, LoRaWAN cloud service. The device is first connected to the LoRa gateway through the LoRa physical protocol, and the LoRa gateway forwards the information to the LoRaWAN cloud service through the UDP protocol.

Hardware

MatchX LoRa gateway

Software

Mainflux integrates LoRa Server written in Go language , and connects Mainflux and LoRa Server through an Adapter through the MQTT protocol, that is to say, this Adapter is an MQTT client connected to Mainflux and LoRa Server, and the main work is a link from LoRa Server to Mainflux SenML conversion.

Summary

• Apache-2.0 licensed
• microservice architecture
• Multi-protocol support (HTTP, MQTT, WebSocket, CoAP)
• Device management and provisioning
• Storage space grows linearly
• Platform logging and instrumentation support
• container deployment

Project introduction and directory structure

architecture design

data model

Microservice Model

Microservice interface

== API by micro service：

• UsersService

// RPC
service UsersService {
    rpc Identify(Token) returns (UserID) {}
}
//--------
// HTTP
type Service interface {
	Register(User) error
	Login(User) (string, error)
	Identify(string) (string, error)
}

• ThingsService

type Thing struct {
	ID       string
	Owner    string
	Type     string
	Name     string
	Key      string
	Metadata string
}

type Channel struct {
	ID       string
	Owner    string
	Name     string
	Metadata string
}
// RPC
service ThingsService {
    rpc CanAccess(AccessReq) returns (ThingID) {}
    rpc Identify(Token) returns (ThingID) {}
}
//--------
// HTTP
type Service interface {
	AddThing(string, Thing) (Thing, error)
	UpdateThing(string, Thing) error
	ViewThing(string, string) (Thing, error)
	ListThings(string, uint64, uint64) (ThingsPage, error)
	ListThingsByChannel(string, string, uint64, uint64) (ThingsPage, error)
	RemoveThing(string, string) error

	CreateChannel(string, Channel) (Channel, error)
	UpdateChannel(string, Channel) error
	ViewChannel(string, string) (Channel, error)
	ListChannels(string, uint64, uint64) (ChannelsPage, error)
	ListChannelsByThing(string, string, uint64, uint64) (ChannelsPage, error)
	RemoveChannel(string, string) error

	Connect(string, string, string) error
	Disconnect(string, string, string) error

	CanAccess(string, string) (string, error)
	Identify(string) (string, error)
}

Installation example

reference documents

• scalable-architecture-for-the-internet-of-things.pdf