The technologies that need to be mastered in the real-time computing stage of big data are: Mahout, Spark, and storm.
1. Spark
Spark is a fast and universal computing engine designed for large-scale data processing. It provides a comprehensive and unified framework for managing various data sets and data sources of different types of data. Data development requires knowledge of Spark basics, SparkJob, Spark RDD, spark job deployment and resource allocation, Spark shuffle, Spark memory management, Spark broadcast variables, Spark SQL, Spark Streaming, and Spark ML.
2.
Storm Storm provides a set of common primitives for distributed real-time computing, which can be used in "stream processing" to process messages and update databases in real time. This is another way to manage queues and worker clusters. Storm can easily write and expand complex real-time calculations in a computer cluster. Storm is used for real-time processing, just like Hadoop is used for batch processing. Storm guarantees that every message will be processed, and it's fast-in a small cluster, it can process millions of messages per second.
