In today's digitally driven world, real-time processing of data streams is a necessary condition for business success.
The introduction of 5G networks has increased requirements for data volume and speed, and these requirements have put pressure on traditional data architectures. The need to absorb data traffic has grown unprecedentedly, while at the same time driving execution by making intelligent and dynamic decisions across multiple data streams.
Current data stream processing systems are usually sufficient to structure processing pipelines, but they cannot meet the needs of application-critical task programs, and low-latency and multi-step response decision-making highlight the needs of these task-based applications.
In addition, it is contrary to the traditional few central hub data centers. With the expected increase in the density of 1 million interconnected things per square kilometer and the specified low latency of a single millisecond, data and processing will be decentralized through several edge data centers.
At the confluence of incomplete information, both traditional and modern choices for processing streaming data will fail. In order for interactive low-latency applications and pipelines to coexist, they must use the same data to drive consistency across functions.
The first four parts with incomplete information are:
1. The microservice architecture requires the separation of state and logic
What is missing is an understanding of the logic of the business type and where it should exist. Although the application flow control logic can remain in the application layer to make the computing container truly stateless, the data business logic must be driven together with the existing data.
2. Network bandwidth usage efficiency
When you store state in a NoSQL data store, and the instance container must move 10 to 25 KB of data payload for each interaction (for example, read an object from the store, modify it and send it back to the data Storage), applications will soon start to consume a lot of network bandwidth. In a virtualized or containerized world, network resources are like gold. People should not waste it for trivial data movement.
3. The basic premise of stream processing
Today's stream processing is based on the concept of time windowing: one of event time windows or processing time windows. This does not represent the real situation. Organizations need to continuously process events, regardless of whether they arrive individually or in context. This approach will avoid problems such as missed events, because they will only be late instead of inflating the database to wait for the last known event that is late.
4. Cross-polling multiple data streams to build complex events that drive decision-making
The event-driven architecture is a message flow, and each message flow is associated with events to drive certain operations. The challenge faced by the architecture is to build complex events from multiple data streams, or to drive changes to multiple states from a single data stream based on complex business logic.
- The intelligent stream processing architecture can operate:
- Absorb incoming event data into the state machine
- Build contextual entity state from multiple ingestion streams
- Apply a set of business rules to drive decisions
- Enhance and enrich these rules by iteratively integrating new knowledge gained from machine learning plans
- Let decisions spread to drive execution
-
Once the contextually completed/processed data is not needed in real-time processing, it is migrated to archive storage
The intelligent stream processing architecture is composed of a unified environment for ingestion, processing and storage.
This integrated method with built-in intelligence can be analyzed where the data is located. It utilizes the fast in-memory relational data processing platform (IMRDPP) not only to "smart" the stream, but also provides linear scaling, predictable low latency, strict ACID, and a much lower value that can be easily deployed in the following locations Edge of the hardware space.
With built-in analysis functions such as aggregation, filtering, sampling and correlation, as well as stored procedures/embedded supervised and unsupervised machine learning, all the elements of stream processing for real-time decision-making can be obtained on an integrated platform.
If you are interested in VoltDB's industrial IoT big data low-latency solution, full life cycle real-time data platform management, etc., welcome to chat privately and discuss with more friends.