background:
With the rapid development and expansion of the company's business, servers and various applications will increase, while the application system, server stability, sustainability put forward higher requirements, the company hopes to build a comprehensive set of analysis and monitoring system to provide decision support for the various parts.
issues that need resolving:
(1) data island issues, data dispersed among different business systems;
(2) server log information about how to effectively collect and store;
(3) business analysis needs is not fixed, it is often put forward new demands, how to meet the needs of various departments;
(4) How quickly erected visual analysis system based on multi-source above problems;
Technology selection and overall architecture:
Program: traffic data ETL, based on the number of bins architecture pattern
Option II: using direct mode data
(1) acquired by the log collection primarily logstash, and then stored in the search engine;
(2) business system data can use two solutions architecture The first is the construction of several positions, will bring together data to several different bins; second: by way of direct direct docking platform for big data analysis;
(3) visualization of large data were analyzed using analysis of NBI large data visualization platform, multi-source data analysis can be achieved, is well supported elasticsearch, relational database, data access large data storage platform; drag type highly flexible and layout mode, you can quickly build all kinds of visual analysis applications; through access control, editing and set access rights, protect the security of corporate data assets effective use;
(4) final by NBI large data visualization analysis platform to build various types of data analysis applications, the output to the corresponding department personnel.
Precautions:
(1) Elasticsearch need to install a plug-sql elasticsearch-sql, according es corresponding version downloaded widget, do facilitate interactive analysis;
(2) NBI big data platform, currently modeling Elasticsearch support for SQL modeling
Steps:
(1) in the data platform NBI large by the data management module is added Elasticsearch data source;
Data source connection
(2) test the connection, whether the test can be connected to normal Elasticsearch
Connection Test
(3) successful connection, through the establishment of SQL worksheet model
Create a data worksheet model
(4) build data reports:
Visualization Report Builder
(5) set access rights:
Permission settings
The above case by NBI large data visualization tools (nbi.easydatavis.com:8033) completed