Spout side:
In the figure, the solid line part represents the original trunk tuple emitted from the spout, and the sub-tuple represented by the dotted line part are all derived from the original tuple. The graph generated in this way is called a tuple tree. In the process of guaranteed data processing, each time the bolt receives a tuple, it needs to acknowledge the acknowledgement (ack) or report an error to the upstream. For a tuple in the trunk tuple, if each bolt in the tuple tree acknowledges the response, the spout will call the ack method to indicate that the message has been fully processed. If any bolt in the tree fails to process the tuple, or if the processing times out, the spout will call the fail method.
Storm's ISpout interface defines three reliability-related APIs: nextTuple, ack, and fail.
As mentioned earlier, Storm sends a tuple by calling the Spout's nextTuple(). To achieve reliable message processing, first give each tuple a unique ID, and pass the ID as a parameter to the SpoutOutputCollector's emit() method: collector.emit(new Values("value1", "value2"), msgId);
Specify an ID for the tuple to tell the Storm system that whether the execution succeeds or fails, the spout must receive notifications from all nodes on the tuple tree. If the processing is successful, the ack() method of the spout will acknowledge the message with the ID number. If the execution fails or times out, the fail() method will be called.
The Fail method resends the failed tuple.
Bolt reliability: There
are two steps for a bolt to implement a reliable message processing mechanism:
1. When emitting a derived tuple, it needs to anchor the read tuple.
2. When the processing of the message succeeds or fails, confirm the response or report an error respectively.
Anchoring a tuple means establishing the correspondence between the read-in tuple and the derived tuple, so that the downstream bolt can be added to the tuple tree structure by responding to confirmation, error or timeout.
Unanchored tuples do not contribute to the reliability of the data stream. If a non-anchored tuple fails downstream processing, the original root tuple will not be resent.