concept
Real-time streaming computing:
In the big data environment, streaming data will be a new type of data, which is continuous, infinite and instantaneous. It is the data type for real-time data processing, and the real-time calculation of this kind of streaming data is real-time streaming calculation.
feature
Different from traditional data processing technology, real-time streaming computing has the following characteristics:
Low latency: From the perspective of processed data, each piece of data can be successfully processed by the system within a limited time, that is, the response time is very short.
High throughput: From the perspective of the processing process, the system nodes can successfully process a large amount of data per unit time, that is, high throughput. For the purpose of data processing, high throughput and low latency are essentially the same.
High fault tolerance: due to network or other reasons, there will be errors or incomplete data, the system has a certain fault tolerance for these data, and will not
There is missing data causing the system to crash.
The algorithm complexity is relatively low: the inevitable requirement of high throughput requires the use of more efficient and simple algorithms.
scenes to be used
Real-time calculation and processing of website background logs
Real-time monitoring of operator traffic
Real-time processing of traffic data
Other Streaming Computing Frameworks
Queue+Worker
Apache S4
Spark Streaming
(Storm's calculation is based on events, and one data stream is processed one; spark streaming processes data based on time periods, which can be smaller and smaller, but cannot be processed based on events. This determines the delay of Storm. smaller than spark streaming)