Software version zookeeper
3.4.10
kafka 2.11-0.10.2.0 elaticsearch
2.10.16 logstash
2.10.2 kibana
4.3.1
jdk 1.8.0_171 All are installed in the /usr/local/share/applications/ directory, and the log output is all in /tmp/package name —Software installation and configuration 1.jdk installation Detect the Java version of the server Java -version
The native Java has met the requirements. I will not go into details
. 2. Zk is installed
to the official website to download related software packages
sudo tar -xvf zookeeper-3.4.10.tar.gz -C /usr/local/share/applications/
Modify the configuration file
and enter cp in the conf directory zoo_sample.cfg zoo.cfg
tickTime A time unit in ZooKeeper. All time in ZooKeeper is based on
this time unit and is configured in integer multiples. The default is 2s
initLimit Follower During the startup process, it will synchronize all the latest data from the Leader to determine the starting state of
its ability to serve externally.
When the Follower has not completed data synchronization within initLimt tickTime , the Leader considers that the Follower connection fails
syncLimit The length of time for the communication request and response between the Leader and the Follower. If the leader
has not received the follower response within syncLimit tickTime, it is considered that the
leader has been offline
in the directory where dataDir stores snapshot files. By default, transaction logs are also stored in this
directory . Since the write performance of the transaction log directly affects the performance of ZooKeeper, it is
recommended to configure the parameter dataLogDir
clientPort ZooKeeper external port
dataDir transaction log output directory at the same time
Configuration of cluster information (first need to configure parsing under /etc/hosts)
server.1=10.253.2.50:2888:3888
server.2=10.253.2.51:2888:3888
server.3=10.253.2.41:2888:3888
( 2888 is the port for communication between the server and the leader. 3888 is the port for mutual communication during elections)
Kafka relies on ZooKeeper, and uses ZooKeeper to manage brokers, consumer online and offline management, cluster, partition
metadata management, etc., so ZooKeeper is also the basis for Kafka to run one of the environments.
For the specific content of zk to kafka, please refer to the zk official website
in the zk bin directory./zkServer.sh start/status
3.kafka install
sudo tar -xvf kafka_2.11-0.10.2.0.tgz -C /usr/local/share/applications/
Modify the configuration file
to enter the config file
vi server.properties
broker.id=1
lod.dirs=/tmp/ kafak
zookeeper.connect=10.253.2.50:2181,10.253.2.51:2181,10.253.2.41:2181
hostname=ip
start
cd ../bin
kafka-server-start .sh - daemon . ./config/server .properties
Note: 2 and 3 operations respectively operate on 51 and 41 and
enter the zk bin
below./zkCli.sh -server 10.253.2.51:2181
4.elk installation and configuration
of elasticsearch
1. You can choose tar package installation or rpm package installation to achieve the same goal here, choose rpm to install
rpm -ivh elsticsearch.6.1.2.rpm
2. Modify configuration files and change permissions
chown -R patrol:patrol /usr/share/ elasticsearch chown -R
patrol:patrol /etc/elasticsearch/elasticsearch.yml
chown -R patrol:patrol /etc/sysconfig/elasticsearch
The configuration files mainly include elasticsearch.yml and sysctl.conf (need to execute sysctl -p to take effect)
vi /etc/elasticsearch/elasticsearch.yml
cluster.name: ELK_node_0
node.name: ELK_node_0_4
path.data: /data/data_sdb/elasticsearch
path. logs: /data/data_sdb/log
bootstrap.memory_lock: false
bootstrap.system_call_filter : false
network.host: 10.253.2.50
transport.tcp.port: 9301
vi /etc/sysctl.conf
add a line
vm.max_map_count = 524288
Behavior / usr / share / elsticsearch
/ bin / elsticsearch -d
Test
curl 'http://10.253.2.50:9201/?pretty'
Introduction to logstach
:
Logstash is a completely open source tool that can collect, filter, and store your logs for later use (eg, search).
Install
using rpm package to install
rpm -ivh logstash-6.1.2.rpm
Configuration:
mkdir -p /data/data_sdb/logstash
mkdir -p /data/data_sdb/log/logstash
Modify the configuration file
vi /etc/logstash/conf.d/elasticsearch.conf
input {
stdin { }
}
output {
elasticsearch {
hosts = > ["10.253.2.50:9201"]
}
stdout {
codec => rubydebug
}
}
vi /etc/logstash/logstash.yml
Modify a few lines:
path.data: /data/data_sdb/logstash
path.logs: /data/data_sdb /log/logstash
chown -R patrol:patrol /usr/share/logstash
chown -R patrol:patrol /etc/logstash
start:
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/elasticsearch.conf Introduction
to kibana
:
Kibana is also an open source and free tool, Kibana can provide Logstash and ElasticSearch with a friendly web interface for log analysis, which can help you summarize, analyze and search important data logs
New folder package
mkdir /data/data_sdb/log/kibana
rpm -ivh kibana-6.1.2-x86_64.rpm
chown -R patrol:patrol /etc/kibana
chown -R patrol:patrol /usr/share/kibana
Modify the configuration file
vi /etc/kibana/kibana.yml
server.port: 5601
server.host: "10.253.2.50"
elasticsearch.url: "http://10.253.2.50:9201"
pid.file: /var/run/kibana/kibana.pid
mkdir /var/run/kibana/
chown -R patorl:patrol /var/run/kibana
Behavior
/ usr / share / kibana / bin / kibana
Key-value optimization and cluster construction
restart elaticsearch
curl 10.253.2.50:9201/_cat/health?v
View node health status
Test:
Modify the configuration file /etc/logstash/conf.d/elasticsearch.conf
input {
file {
path => "/data/data_sdb/elasticsearch/*.log"
}
}
output {
elasticsearch { hosts => ["10.253.2.50:9201"] }
stdout {
codec => rubydebug
启动 logstash
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/elasticsearch.conf
当出现
WARNING: Could not find logstash.yml which is typically located in $LS_HOME/config or /etc/logstash. You can specify the path using --path.settings. Continuing using the defaults
Could not find log4j2 configuration at path /usr/share/logstash/config/log4j2.properties. Using default config which logs errors to the console
时 编辑
data/data_sdb/elasticsearch/*.log 文件
{
"@timestamp" => 2018-02-28T08:34:36.276Z,
"message" => "adsf",
"@version" => "1",
"path" => "/data/data_sdb/elasticsearch/1.log",
"host" => "centos-wz3hgym2"
}
{
"@timestamp" => 2018-02-28T08:34:36.274Z,
"message" => "fifdsfds",
"@version" => "1",
"path" => "/data/data_sdb/elasticsearch/1.log",
"host" => "centos-wz3hgym2"
}
The content you edit will appear in this form. The relevant content will appear on the
start kibana page.
Elasticsearch configuration file parameter modification
# Set the number of shards for the index, the default is 5
index.number_of_shards: 15
# Path to temporary file
path.work: /path/to/work
# Plugin installation path
# path.plugins: /path/to/plugins
# When the JVM starts to write to the swap space (swapping) ElasticSearch performance will be slow, you should ensure that it does not write to the swap space
bootstrap.mlockall: true
# Probe timeout, the default is 3 seconds, increase it a little to prevent split-brain when the network is not good
discovery.zen.ping.timeout: 60s
Two related tests
for the combination of kafka and elk
First create a topic into the kafka bin directory
kafka-topics.sh --zookeeper 10.253.2.50:2181,10.253.2.51:2181,10.253.2.41:2181 --create --topic kafka- elk-log --partitions 1 --replication-factor 1
topic name is kafka-elk-log
1.logstash transmits data to kafka and
enters logstash directory mkdir etc
vi logstash-input-kafka
input{ #Receive input
from console
stdin{}
output{
kafka{ #messages
, written topics
top worker c_id => "kafka-elk-log " #connect
to Kafka cluster configuration
bootstrap_servers => "10.253.2.50:2181,10.253.2.51:2181,10.253.2.41:2181"
# Batch write configuration
batch size => 5
# logstash import data encoding method
codec => ” plain ”
stdout{
# Set console print data representation
codec => rubydebug
}
}
cd bin/
logstash -f ../etc/logstash*
screen output
验证：
kafka-run-class.sh kafka.tools.DumpLogSegments --files /tmp/kafka-logs/kafkaelk-log-0/00000000000000000000 . log --print-data-log
2. kafka data is sent to es
vi logstash-output-es
input{
kafka {
# logstash export data decoding method
codec => ” plain ”
# consumption group
group_id => ” kafka_elk_group”
# consumer identification
cl worker ent_id => logstash
# consumption Topic
topics =>”kafka_elk_log” #Connect
Kafka cluster configuration
bootstrap_servers =>”10.253.2.50:2181,10.253.2.51:2181,10.253.2.41:2181” #Consumption
start position
auto offset reset =>” earliest”
#Consumption By: the number of
consumer threads => 5
output{
# import elasticsearch elasticsearch
{
# elasticsearch cluster address, multiple addresses are separated by commas
hosts => [ ” 10.253.2.50:9201”]
# Specify the data import elasticsearch format
codec = > ”plain” #Create index
number｜
index => ” kaf ka_elk_log - ”{+YYYY.MM.dd} ”
}
}
Verification:
Browser: http://10.253.2.50:9201/_plugin/head/
kibana 誌证:
You need to add the index index name under setting now and write it in the logstash configuration file