Environment installation of linux ELK+kafka to build log collection system

background

ELK consists of three components: Elasticsearch, Logstash and Kibana;

• Elasticsearch is an open source distributed search engine. Its characteristics are: distributed, zero configuration, automatic discovery, automatic index sharding, index copy mechanism, restful style interface, multiple data sources, automatic search load, etc.

• Logstash is a completely open source tool, it can collect and analyze your logs, and store them for later use

• Kibana is an open source and free tool that provides Logstash and ElasticSearch with a friendly web interface for log analysis, which can help you summarize, analyze, and search for important data logs.

Set up the environment

ELK+Kafka execution principle diagram

Build steps

• Installation of Kafka and Zookpeer
• elk installation
• Use springboot to send messages to kafka and use kibana to view logs

Kafka+zookeeper stand-alone installation and configuration

http://kafka.apache.org/quickstart

The Kafka+zookeeper cluster can be built separately, or you can use the zookeeper that comes with Kafka. This installation uses the zookeeper that comes with Kafka

Unzip

# 解压 kafka_2.13-2.5.1.tgz
 tar -xzvf kafka_2.13-2.5.1.tgz
# 为 kafka 创建软链接
 ln -sv kafka_2.13-2.5.1 ../kafka
# 进入kafka
cd kafka

Start zookeeper

Start zookeeper

bin/zookeeper-server-start.sh config/zookeeper.properties &

Start kafka

Start kafka

bin/kafka-server-start.sh config/server.properties &

Create topic quickstart-events

bin/kafka-topics.sh --create --topic logger-channel --bootstrap-server 20.26.99.21:8092

View topics that have been created

 bin/kafka-topics.sh --list --zookeeper localhost:2181

Producer sends message

$ bin/kafka-console-producer.sh --topic quickstart-events --bootstrap-server localhost:9092
>This is my first event
>This is my second event

Consumer accepts the message

$ bin/kafka-console-consumer.sh --topic quickstart-events --from-beginning --bootstrap-server localhost:9092
This is my first event
This is my second event

logstash

Unzip

tar -xzvf logstash-7.9.1.tar.gz
# 进入文件夹
cd logstash-7.9.1

Modify the configuration file

Copy the configuration file of logstash-sample.conf to the root directory and rename it to core.conf

cp config/logstash-sample.conf ./core.conf

Edit file

# Sample Logstash configuration for creating a simple
# Beats -> Logstash -> Elasticsearch pipeline.
 # Sample Logstash configuration for creating a simple
    # Beats -> Logstash -> Elasticsearch pipeline.
    
    input {
    
    
      kafka {
    
    
        id => "my_plugin_id"
        bootstrap_servers => "20.26.99.21:8092"
        topics => ["logger-channel"]
        auto_offset_reset => "latest" 
        decorate_events => true    #在输出消息的时候回输出自身的信息，包括：消费消息的大小、topic来源以及consumer的group信息。
        type => "statistic"
      }
    }
    #filter {
    
    
    
    #    grok {
    
    
    #      patterns_dir => ["./patterns"]
    #        match => {
    
     "message" => "%{WORD:module} \| %{LOGBACKTIME:timestamp} \| %{LOGLEVEL:level} \| %{JAVACLASS:class} - %{JAVALOGMESSAGE:logmessage}" }
    #      }
        
        
    #}
    output {
    
    
      #stdout {
    
     codec => rubydebug }
      elasticsearch {
    
    
           hosts =>["localhost:8200"]
           index => "kafka"
      }
}

start up

bin/logstash -f core.conf --config.reload.automatic &

Install ES

Unzip

tar -xzvf  elasticsearch-7.9.1-linux-x86_64.tar.gz
# 进入文件夹
cd  elasticsearch-7.9.1

Modify the configuration file

Modify the port number to 8200

vi config/elasticsearch.yml
http.port: 8200

Since es cannot be activated by root account, you need to create a new account cloud

New account

root添加用户名
useradd cloud
passwd cloud
输入两次密码  字母+数字

Empower account

chown -R cloud /app/cloud/

Switch to new account

su cloud

start up

 ./bin/elasticsearch &

View all index numbers
http://20.26.99.21:8200/_cat/indices?v

Kibana

Unzip

tar -xzvf kibana-7.9.1-linux-x86_64.tar.gz
# 进入文件夹
cd kibana-7.9.1

Modify port number

vi config/kibana.yml


server.port: 8601
server.host: "0.0.0.0"
elasticsearch.hosts: ["http://localhost:8200/"]
#elasticsearch.url: "http://20.26.99.21:8200"
kibana.index: ".kibana"

start up

./bin/kibana --allow-root &

Create index kafka

Create index

address

http://20.26.99.21:8601/app/discover#/?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-15m,to:now))&_a=(columns:!(_source),filters:!(),index:fe9b9250-f95b-11ea-9b5c-2bc75f19f1b3,interval:auto,query:(language:kuery,query:’’),sort:!())