[Big Data] ELK real-time log analysis platform (the simplest entry-level case, taking you into the ELK world)

This article introduces you to the ELK world through the simplest and purest case.

1 Introduction

ELK is the abbreviation of Elasticsearch, Logstash, and Kibana. If you don't know much about Elasticsearch, Logstash, and Kibana, you can refer to the introductory case on the official website. The author also wrote an article introducing the official introductory case.

Take Elasticsearch6.6.2 as an example:

• Official entry

https://www.elastic.co/guide/en/elasticsearch/reference/6.6/getting-started.html

https://www.elastic.co/guide/en/kibana/6.6/getting-started.html

https://www.elastic.co/guide/en/logstash/6.6/getting-started-with-logstash.html

• author translation

to be written

to be written

to be written

2. Installation

Stepping pit guide:

• The author used Docker or Docker-compose to install the ELK environment on a Mac with various problems. For one thing, the host host has insufficient permissions, the host host has insufficient disk space, and so on.
• In the end, it still didn't work out, and the author gave up on Docker. So I went to try it with Docker on Linux, and it was successful, but it was annoying to start a Linux virtual machine.
• The purpose of using Docker is to bring convenience. Since it does not bring convenience, I choose to use the installation package or compressed package on the Mac .

The download URL of ELK is: https://www.elastic.co/cn/downloads/past-releases

Take Elasticsearch6.6.2 as an example:

• download

If you are a Windows user please use other download links.

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.6.2.tar.gz
wget https://artifacts.elastic.co/downloads/kibana/kibana-6.6.2-darwin-x86_64.tar.gz
wget https://artifacts.elastic.co/downloads/logstash/logstash-6.6.2.tar.gz

3. Start ELK

Start Elasticsearch

• Start Elasticsearch

curl -L -O https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.6.2.tar.gz
tar -xvf elasticsearch-6.6.2.tar.gz
cd elasticsearch-6.6.2/bin
./elasticsearch

• Test ES startup

curl http://localhost:9200

Start Kibana

• Start Kibana

curl -O https://artifacts.elastic.co/downloads/kibana/kibana-6.6.2-darwin-x86_64.tar.gz
# 校验shasum
shasum -a 512 kibana-6.6.2-darwin-x86_64.tar.gz
tar -xzf kibana-6.6.2-darwin-x86_64.tar.gz
cd kibana-6.6.2-darwin-x86_64/
./bin/kibana

• Visit http://localhost:5601 to see if the startup is successful

Start Logstash

• New configurationfirst-pipeline.conf

input {
    
    
    tcp {
    
    
        mode => "server"
        host => "0.0.0.0"
        port => 4560
    	codec => json_lines
    }
}
output {
    
    
    elasticsearch {
    
    
        hosts => ["http://localhost:9200"]
        index => "springboot-logstash-%{+YYYY.MM.dd}"
    }
}

Accept input data from any source without any processing and send it to the local ES.

• start up

bin/logstash -f first-pipeline.conf --config.reload.automatic

Configured --config.reload.automaticto not need to restart after modifying the first-pipeline.conf file

4. Test the ELK environment

• Send a string of json data to logstash using tcp

echo '{"logstash": "hello world"}' | nc localhost 4560shell

• View the log information just sent hello world in kibana

I saw the hello world json message I just sent.