OpenSOC------ a powerful tool for big data collection/storage/analysis/display

OpenSOC is an open source framework used by Cisco in the security field, or it can be said to be the open source framework of SOC (Security Operations Center). Its components include:

quote

!  Telemetry Capture Layer:Apache Flume
!  Data Bus:Apache Kafka
!  Stream Processor: Apache Storm
!  Real-Time Index and Search: Elastic Search
!  Long-Term Data Store:Apache Hive
!  Long-Term Packet Store: Apache Hbase
!  Visualization Platform:Kibana

  

I think no matter what system, it can evolve, or tailor, or expand from the above components.
Flume collects data. The data source can be text, syslog, or http request, which is quite flexible.
The role of Kafka is the message queue in the early system, now called the data bus, and its functions are also leveraged.
The spout and bolt in storm are also extremely powerful. OpenSOC provides general data processing processes, such as data parsing, formatting, filtering, enhancement (that is, supplementary information), business processing, which can be flexibly added on top of the framework. any business.
Elastic Search is also a good retrieval tool.
Kibana, a web presentation tool, is not too customary.
This framework integrates most big data tools and is an essential tool for study and work.