What is real-time streaming computing:
Real-time streaming computing is the superposition of real-time computing and streaming computing.
Real-time calculation: The response time is subject to real-time constraints, the time constraints are very short, in seconds, milliseconds
Streaming Computing: Computation on a continuously generated stream of data
A data stream consists of basic data units
The data flow is constantly generated, without end, and the calculation results are constantly generated/updated
Real-time streaming computing: real-time computing on the continuously generated data stream, the input data stream is continuously generated without end, and the generation time of the calculation result is real-time relative to the time of the data inflow.
Real-time streaming computing features:
1. Streaming processing, data flows in and out continuously, as opposed to batch processing, such as: dynamically refreshed PV lines, daily PV statistical reports
2. High timeliness, results are obtained quickly, the speed here is relative, it is fast enough for business needs, and fast enough for other technologies such as batch processing
Suitable scene:
1. Real-time streaming data analysis, such as real-time statistics, real-time early warning
2. Real-time ETL, such as real-time log filtering and normalization
3. Continuous calculation, such as news auto-discovery analysis
Key technical points:
System architecture (single-machine architecture, distributed architecture, master-slave), latency throughput, programming model (provide developers with simple programming methods and interfaces, focus on business, and improve development speed), data transmission, high availability ( Ensure the stable operation of the system), maintainability (when a lot of work is done automatically, it means that the system has high maintainability)
Common live streaming systems:
MQ plus Worker process, S4, SparkStreaming, Strom
Storm's computational model:
DAG模型：spout/bolt/tuple/stream/grouping
Storm's system architecture:
nimbus/supervisor/woker/executor/zk