DBus experience rapid deployment of real-time data flow computation

First, the environmental description

Description: all in one environment is a stand-alone version of dbus, dbus experience is to give users fast function, simply a trial version, or the environment can not be used for other purposes, including the following:

1) form the basis of:

• zookeeper 3.4.6
• kafka 0.10.0.0
• storm 1.0.1
• granfana 4.2.0
• logstash 5.6.1
• influxdb (installed separately, with reference to the following step 3)
• MySQL (installed separately, with reference to the step 2)

2) dbus related packages:

• dbus-keeper 0.5.0
• dbus-stream-main 0.5.0
• dbus-router 0.5.0
• dbus-heartbeat 0.5.0
• dbus-log-processor 0.5.0

3) mysql data source required:

• canal

1.1 Environmental dependent

Dbus-allinone proposed installation environment configured as follows

• JDK 1.8.181 or above
• CPU 2 or more nuclei
• Memory 16GB or more
• Disk 20GB or more

Note: Linux centos a server, which is preferably empty the machine, do not depend mounted thereon dbus zk, kafka, storm etc.

1.2 modify the domain name

Note: ip your specific ip, used here as an example 192.168.0.1

Modify server / etc / hosts file corresponding to the domain information is provided as follows:

192.168.0.1 dbus-n1

Modify the server hostname command as follows:

hostname dbus-n1

Once configured, the server IP and domain information is as follows:

1.3 app users create and configure SSH-free dense Login

Since dbus start ssh topology using call storm command, all in one package in the default calling ssh use app users and 22 ports, and therefore the normal experience of all in one need to create app accounts and configure ssh-free dense Log-free dense login configuration from dbus-n1 to dbus-n1-free dense Login

After configuration, execute the following command to see when the configuration is successful

[app@dbus-n1 ~]$ ssh -p 22 app@dbus-n1
Last login: Fri Aug 10 15:54:45 2018 from 10.10.169.53
[app@dbus-n1 ~]$

Second, the preparation

2.1 Installation Mysql

2.1.1 download

Recommended download Mysql Version: 5.7.19 Address: https://dev.mysql.com/downloads/mysql/

2.1.2 Installation

After extracting mysql-5.7.19-1.el6.x86_64.rpm-bundle.tar package, run the following command installation:

rpm -ivh mysql-community-server-5.7.19-1.el6.x86_64.rpm --nodeps
rpm -ivh mysql-community-client-5.7.19-1.el6.x86_64.rpm --nodeps
rpm -ivh mysql-community-libs-5.7.19-1.el6.x86_64.rpm --nodeps
rpm -ivh mysql-community-common-5.7.19-1.el6.x86_64.rpm --nodeps
rpm -ivh mysql-community-libs-compat-5.7.19-1.el6.x86_64.rpm --nodeps

2.1.3 Configuration

In /etc/my.cnf configuration file, just add bin-log configuration, without modifying the other, following the comments section of Chinese characters

[mysqld]
# dbus相关配置开始
log-bin=mysql-bin
binlog-format=ROW
server_id=1
# dbus相关配置结束

2.1.4 Start

Execute the following command to start mysql:

service mysqld start

2.2. Installation InfluxDB

2.2.1 download

Recommended download InfluxDB version: influxdb-1.1.0.x86_64 Address: https://portal.influxdata.com/downloads

2.2.2 Installation

Switching on dbus-n1 to the root user, execute the following commands stored in the directory influxdb-1.1.0.x86_64.rpm:

rpm -ivh influxdb-1.1.0.x86_64.rpm

2.2.3 Start

Run the following commands on dbus-n1:

service influxdb start

2.2.4 Initial Configuration

Run the following commands on dbus-n1:

#登录influx
influx

#执行初始化脚本
create database dbus_stat_db
use dbus_stat_db
CREATE USER "dbus" WITH PASSWORD 'dbus!@#123'
ALTER RETENTION POLICY autogen ON dbus_stat_db DURATION 15d

Third, the installation package Dbus-allinone

3.1 Download

Providing dbus-allinone.tar.gz package in Baidu network disk, access the page to download the latest release package: https://github.com/BriData/DBus/releases

3.2 Installation

Download dbus-allinone package uploaded to the server / app directory, and must be in this directory

# 如果没有app目录，要先创建app目录
mkdir /app
cd /app
tar -zxvf dbus-allinone.tar.gz

3.3 initialize the database

Log in as root mysql client, execute the following command to initialize the database, and user-created dbmgr library, canal users, dbus libraries, and user, testschema libraries, and user:

source /app/dbus-allinone/sql/init.sql

3.4 start

Start.sh perform a key to start dbus all the services, startup items more.

cd /app/dbus-allinone
./start.sh

Please wait (probably takes about five minutes), the correct boot log is as follows:

Start grafana...
Grafana started. pid: 23760
=============================================================================================
Start zookeeper...
zookeeper pid 23818
Zookeeper started.
=============================================================================================
Start kafka...
No kafka server to stop
kafka pid 24055
kafka started.
=============================================================================================
Start Canal ... 
canal started.
=============================================================================================
Start logstash...
No logstash to stop
nohup: appending output to `nohup.out'
logstash pid 24151
logstash started.
=============================================================================================
Start storm nimbus...
No storm nimbus to stop
Storm nimbus pid 24215
Storm nimbus started.
=============================================================================================
Start storm supervisor...
No storm supervisor to stop
Storm supervisor pid 24674
Storm supervisor started.
=============================================================================================
Start storm ui...
No storm ui to stop
Storm ui pid 24939
Storm ui started. ui port: 6672
=============================================================================================
Stop storm topology.
Storm topology stoped.
=============================================================================================
Start storm topology...
Storm topology started.
=============================================================================================
Start Dbus Heartbeat...
No Dbus Heartbeat to stop
Dbus Heartbeat pid 26854
Dbus Heartbeat started.
=============================================================================================
Start Dbus keeper...
=========================stop===========================
keeper-proxy process not exist
gateway process not exist
keeper-mgr process not exist
keeper-service process not exist
register-server process not exist

=========================start===========================
register-server started. pid: 27077
keeper-proxy  started. pid: 27172
gateway started. pid: 27267
keeper-mgr started. pid: 27504
keeper-service started. pid: 27645
Dbus keeper prot: 6090
Dbus keeper started.
=============================================================================================

3.5 generate inspection reports to see whether the normal start

Enter the directory /app/dbus-allinone/allinone-auto-check-0.5.0, automatic detection script auto-check.sh, wait a little while

cd /app/dbus-allinone/allinone-auto-check-0.5.0
./auto-check.sh

It will produce the corresponding inspection report at the time /app/dbus-allinone/allinone-auto-check-0.5.0/reports directory, as shown

[app@dbus-n1 reports]$ tree
.
└── 20180824111905
    └── check_report.txt

Open file check_report.txt view the inspection report, the following

(Note that beginning with # is the interpretation information, the report is not generated)

# 出现以下信息说明dbusmgr库正常
check db&user dbusmgr start:
============================================
table t_avro_schema data count: 0
table t_data_schema data count: 4
table t_data_tables data count: 4
table t_dbus_datasource data count: 2
table t_ddl_event data count: 0
table t_encode_columns data count: 0
table t_encode_plugins data count: 1
table t_fullpull_history data count: 0
table t_meta_version data count: 5
table t_plain_log_rule_group data count: 1
table t_plain_log_rule_group_version data count: 1
table t_plain_log_rule_type data count: 0
table t_plain_log_rules data count: 5
table t_plain_log_rules_version data count: 5
table t_project data count: 1
table t_project_encode_hint data count: 1
table t_project_resource data count: 1
table t_project_sink data count: 1
table t_project_topo data count: 1
table t_project_topo_table data count: 1
table t_project_topo_table_encode_output_columns data count: 1
table t_project_topo_table_meta_version data count: 0
table t_project_user data count: 1
table t_query_rule_group data count: 0
table t_sink data count: 1
table t_storm_topology data count: 0
table t_table_action data count: 0
table t_table_meta data count: 7
table t_user data count: 2

# 出现以下信息说明dbus库正常
check db&user dbus start:
============================================
table db_heartbeat_monitor data count: 15
table test_table data count: 0
table db_full_pull_requests data count: 0

# 出现以下信息说明canal用户正常
check db&user canal start: 
============================================
master status File:mysql-bin.000002, Position:12047338
table db_heartbeat_monitor data count: 15
table test_table data count: 0
table db_full_pull_requests data count: 0

# 出现以下信息说明testschema库正常
check db&user testschema start: 
============================================
table test_table data count: 0

# 出现以下信息说明zk启动正常
check base component zookeeper start: 
============================================
23818 org.apache.zookeeper.server.quorum.QuorumPeerMain

# 出现以下信息说明kafka启动正常
check base component kafka start: 
============================================
24055 kafka.Kafka

# 出现以下信息说明storm nimbus、supervisor、ui 启动正常
check base component storm start: 
============================================
26500 org.apache.storm.daemon.worker
25929 org.apache.storm.daemon.worker
27596 org.apache.storm.LogWriter
26258 org.apache.storm.LogWriter
24215 org.apache.storm.daemon.nimbus
27035 org.apache.storm.LogWriter
27611 org.apache.storm.daemon.worker
26272 org.apache.storm.daemon.worker
24674 org.apache.storm.daemon.supervisor
24939 org.apache.storm.ui.core
26486 org.apache.storm.LogWriter
27064 org.apache.storm.daemon.worker
25915 org.apache.storm.LogWriter

# 出现以下信息说明influxdb 启动正常
check base component influxdb start: 
============================================
influxdb 10265     1  0 Aug08 ?        02:28:06 /usr/bin/influxd -pidfile /var/run/influxdb/influxd.pid -config /etc/influxdb/influxdb.conf
app      28823 28746  0 11:19 pts/3    00:00:00 /bin/sh -c ps -ef | grep influxdb
app      28827 28823  0 11:19 pts/3    00:00:00 grep influxdb

# 出现以下信息说明grafana 启动正常
check base component grafana start: 
============================================
app      23760     1  0 11:09 pts/3    00:00:00 ./grafana-server
app      28828 28746  0 11:19 pts/3    00:00:00 /bin/sh -c ps -ef | grep grafana
app      28832 28828  0 11:19 pts/3    00:00:00 grep grafana

# 出现以下信息说明心跳heartbeat 启动正常
check base component heartbeat start: 
============================================
26854 com.creditease.dbus.heartbeat.start.Start

# 出现以下信息说明logstash 启动正常
check base component logstash start: 
============================================
24151 org.jruby.Main

# 出现以下信息说明canal 启动正常
check canal start: 
============================================
zk path [/DBus/Canal/otter-testdb] exists.
24105 com.alibaba.otter.canal.deployer.CanalLauncher

# 出现以下信息说明dispatcher-appender、mysql-extractor、splitter-puller、router 启动正常
check topology start: 
============================================
api: http://dbus-n1:6672/api/v1/topology/summary
topology testlog-log-processor status is ACTIVE
topology testdb-mysql-extractor status is ACTIVE
topology testdb-splitter-puller status is ACTIVE
topology testdb-dispatcher-appender status is ACTIVE
topology tr-router status is ACTIVE

# 出现以下信息说明从数据库->extractor-dispatcher->appender线路正常
check flow line start: 
============================================
first step insert heart beat success.
data arrive at topic: testdb
data arrive at topic: testdb.testschema
data arrive at topic: testdb.testschema.result

Fourth, all in one package to verify whether the installation is successful

4.1 Login grafana premise

Need to configure the host file in the browser to log grafana machine, if already configured, skip this premise

If the verification system is windows machine, modify the C: \ Windows \ System32 \ drivers \ etc \ hosts file to set the appropriate domain information is as follows:

# 192.168.0.1 替换成部署allinone包的服务器ip地址
192.168.0.1 dbus-n1

If the machine is a linux verification system, modify / etc / hosts file corresponding to the domain information is provided as follows:

# 192.168.0.1 替换成部署allinone包的服务器ip地址
192.168.0.1 dbus-n1

4.2 Login grafana

Login grafana url address: HTTP: // dbus-N1: 3000 / the Login

4.3 mysql insert data validation

#登录测试用户
mysql -utestschema -p     #testschema账户密码：j0<C6cqcqr:TestSchema
#执行测试脚本
use testschema;
INSERT INTO test_table (NAME, BIRTHDAY) VALUES ('testdataname', '2018-08-10 18:00:00');
INSERT INTO test_table (NAME, BIRTHDAY) VALUES ('testdataname', '2018-08-10 18:00:00');
INSERT INTO test_table (NAME, BIRTHDAY) VALUES ('testdataname', '2018-08-10 18:00:00');
INSERT INTO test_table (NAME, BIRTHDAY) VALUES ('testdataname', '2018-08-10 18:00:00');
INSERT INTO test_table (NAME, BIRTHDAY) VALUES ('testdataname', '2018-08-10 18:00:00');

Check whether DBus 4.4 real-time access to data

V. logstash verify whether the installation was successful extraction heartbeat log

dbus-heartbeat heartbeat children have a log module in position: /app/dbus-allinone/dbus-heartbeat-0.5.0/logs/heartbeat/heartbeat.log, logstash extracted using the log file, the unstructured data, extraction into structured data

5.1 log raw unstructured data as follows:

By command to check the unstructured data to be extracted, as shown below, have a log every minute with "heartbeat packets inserted success" 3

cd /app/dbus-allinone/dbus-heartbeat-0.5.0/logs/heartbeat/
tail -f heartbeat.log | grep "插入心跳包成功"

5.2 Results configure the structure in view rules and rule execution Dbus Keeper

FIG extraction rules are as follows:

Rule Execution Results

5.3 extract real-time traffic monitoring in grafana View

Select log table deatil board

Select the table: testlog.testlog_schema.t_heartbeat_data

Six login DBus Keeper examination UI operations

6.1 Login dbus keeper premise

Need to configure the host file dbus keeper login through the browser machine, if already configured, skip this premise

If the verification system is windows machine, modify the C: \ Windows \ System32 \ drivers \ etc \ hosts file to set the appropriate domain information is as follows:

# 192.168.0.1 替换成部署allinone包的服务器ip地址
192.168.0.1 dbus-n1

If the machine is a linux verification system, modify / etc / hosts file corresponding to the domain information is provided as follows:

# 192.168.0.1 替换成部署allinone包的服务器ip地址
192.168.0.1 dbus-n1

6.2 Login dbus keeper

Login dbus keeper url address: HTTP: // dbus-N1: 6090 / the Login

Experience administrator functions, use user: admin Password: 12345678

Administrator interface is as follows:

Experience tenant function, use the user: [email protected] Password: 12345678

Tenant interface is as follows:

Git Address: https://github.com/BriData