spark notes related to the operation of DStream

And similar operations on the RDD DSTREAM, divided Transformations (conversion) and Output Operations (output), with the switching operation in addition there are a number of special operations, such as: updateStateByKey (), transform () and all the associated Window operation.

4.1 Transformations on DStreams
spark notes related to the operation of DStream
special Transformations

(1)UpdateStateByKey Operation

UpdateStateByKey for recording history, saved the last state

(2) Window Operations (windowing function)

Sliding window switching operation:

Sliding window calculation conversion operation as shown below, we can pre-set a sliding window length (i.e. duration of the window), and the sliding window set time interval (how often to perform a calculation), then, you can let the window slide on the source DSTREAM specified time interval, each window parked position, there will be a portion DSTREAM into the window frame, the formation of a small pieces DSTREAM, time, you can start this subparagraph DSTREAM calculations.
spark notes related to the operation of DStream
(1) is a red rectangle window, the window is framed data flow within a period of time.

(2) every time there is a time unit, in the example of officially every window size is 3 time unit, and every 2 units of time, a window slide.

Therefore, based on the operation window to specify two parameters:

· window length - The duration of the window (3 in the figure)

· slide interval - The interval at which the window-based operation is performed (2 in the figure).  

a. the window size, data of the container over time.

Computing a b. Slide interval, how often.

4.2 Output Operations on DStreams

Output Operations DStream can output data to an external database or file system, when a Output Operations to be called (the same as the RDD of Action), spark streaming program will start the real calculation.
----------------
spark notes related to the operation of DStream

Guess you like

Origin blog.51cto.com/14473726/2439140