Log center single node and cluster construction manual

1. Introduction to ELK Log Center

The log center, through the standard log center architecture of collection, buffering, storage, and display, collects the local log data generated by each application, summarizes them for final retrieval.

The working principle of ELK is as follows (understanding the principle, you can deploy a highly available ELK platform): Logstash collects logs generated by APPServer (application server) and stores them in the Elasticsearch cluster, while Kibana queries data from the Elasticsearch cluster to generate charts , and then returned to the browser (browser). To put it simply, log processing and analysis generally requires the following steps:
1) Logstash centralizes the management of logs.
2) Format the log (Logstash) and output it to Elasticsearch.
3) Elasticsearch indexes and stores the formatted data.
4) Kibana’s display of front-end data.

Tip:
Use filebeat 6.2.4 for collection, Kafka 2.11 for buffering, logstash 5.6.8 for offloading, and Elasticsearch 5.6.8 for storage.

https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-oss-7.5.0-linux-x86_64.tar.gz
Logstash download address:
https://www.elastic.co/cn/downloads/logstash
Kibana download Address:
https://www.elastic.co/cn/downloads/kibana

2. Installation and configuration

This section contains the following parts.
1. Install Elasticsearch
2. Install LogStash
3. Install Kibana
4. Install Filebeat

Note:
This chapter has introduced single-instance deployment. If users need to deploy cluster mode, please refer to the official documentation.
The content of the installation package configuration file below only lists key configuration items, and the rest are omitted (all are default configuration values). For details, please refer to the official documentation.
The premise requires a java environment: Elasticsearch requires at least Java 8

2.1 Install elasticSearch

The application directory under $HOME will be created, elasticsearch-7.5.0-linux-x86_64.tar.gz will be placed in this directory, and decompressed.

2.1.1 In the "elasticsearch-7.5.0-linux-x86_64" directory, execute the following command, and edit the elasticsearch.yml file according to the Elasticsearch installation parameter instructions in the Linux environment.

$ cd elasticsearch-7.5.0-linux-x86_64/
$ vim config/elasticsearch.yml

Elasticsearch.yml  5.6版本
#(略....)
#
#---------------------------------- Cluster -----------------------------------
#
#Use a descriptive name for your cluster:
#
cluster.name: my-application   #集群名
#
#------------------------------------ Node ------------------------------------
#
#Use a descriptive name for the node:
#
node.name: node-1  #节点名
#
#Add custom attributes to the node:
#
node.attr.rack: r1
#
#----------------------------------- Paths ------------------------------------
#Path to directory where to store the data (separate multiple locations by comma):
path.data: /path/to/data   #数据文件路径
#Path to log files:
#
path.logs: /path/to/logs    #日志文件路径
#----------------------------------- Memory -----------------------------------
#Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
#(略....)
#---------------------------------- Network -----------------------------------
#Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: 192.168.0.1  #绑定的IP地址
#
#Set a custom port for HTTP:
#
http.port: 9200    #服务端口
#For more information, consult the network module documentation.
#
#--------------------------------- Discovery ----------------------------------
#
#Pass an initial list of hosts to perform discovery when new node is started:
#The default list of hosts is ["127.0.0.1", "[::1]"]   #  允许访问的地址列表
#
discovery.zen.ping.unicast.hosts: ["host1", "host2"]
#
#Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
#
node.master: true
discovery.zen.minimum_master_nodes:2       #集群中至少nodes/2 + 1为主节点，避免脑裂现象    
#
#For more information, consult the zen discovery module documentation.
#---------------------------------- Gateway -----------------------------------
#
#Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#For more information, consult the gateway module documentation.
#---------------------------------- Various -----------------------------------
#Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
#以下是为了避免X-PACK插件与head冲突导致elasticsearch-head无法正常连接elasticsearch而配置的。
http.cors.enabled: true
http.cors.allow-origin: "*"
http.cors.allow-headers: Authorization,X-Requested-With,Content-Length,Content-Type

7.5版本
cluster.name: log-collect
node.name: yelksvc9
network.host: 0.0.0.0
http.port: 9200
discovery.zen.ping.unicast.hosts: [“11.104.34.155”,“11.104.34.156”,“11.104.34.157”]
cluster.initial_master_nodes: [“yelksvc7”,“yelksvc8”]
node.master: true
discovery.zen.minimum_master_nodes: 2

2.1.2 Execute the following command to start the service.

$ cd elasticsearch-7.5.0-linux-x86_64/
$ bin/elasticsearch -d

Note:
If the startup error: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536], please execute the following command.

ulimit -n 65536

遇到如下错误解决方案：
ERROR: [2] bootstrap checks failed
[1]: max file descriptors [65535] for elasticsearch process is too low, increase to at least [65536]
[2]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

ulimit -n 65536
 
vi /etc/sysct.conf
最后一行添加：
vm.map_map_count=262144
sysct -p

If after execution, an error is still reported. If the error is reported at startup: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536], please ulimit -Sn;ulimit -Hn;ulimit -Hu to check the current value, if not To set the value, you need to check whether it is configured in the configuration file /etc/security/limits.conf

• soft nofile 65536

• hard nofile 65536

• soft nproc 65536

• hard nproc 65536
  and then execute the environment variable to make it effective, and then use ulimit -Sn to check whether the value is 65536

2.1.3 Execute the following command, and the startup will be successful as shown in the figure below.

$ curl http://localhost:9200/?pretty

[The difference between ElasticSearch's 9300 and 9200 ports]
9200 is the port used by the ES node to communicate with the outside. It is a RESTful interface of the http protocol (various CRUD operations use this port, such as query: http://localhost:9200/user/_search).
9300 is the port used for communication between ES nodes. It is the tcp communication port, which is used by both clusters and TCPclient. (When using ES in a java program, this port must be configured in the configuration file)

[Word segmenter plug-in elasticsearch-analysis-ik]
https://github.com/medcl/elasticsearch-analysis-ik/releases
Elasticsearch already contains a word segmentation method standard by default. The default word segmenter will divide Chinese into single words for full-text search. , not the result we want!
If not installed, test

curl -H 'Content-Type: application/json'  -XGET 'localhost:9200/_analyze?pretty' -d '{"text":"测试1"}'

After installation, the test is as follows:

curl -H 'Content-Type: application/json'  -XGET 'localhost:9200/_analyze?pretty' -d '{"analyzer":"ik_max_word","text":"测试2"}'

[View el cluster status]

curl --user elastic:changeme localhost:9200/_cluster/stats?Pretty

[View el node status]

curl --user elastic:changeme http://11.104.34.149:9200/_nodes/process?pretty

[View el cluster health]

curl --user elastic:changeme localhost:9200/_cluster/health?Pretty

[Check what el attributes are]
curl --user elastic:changeme localhost:9200/_cat/, the attributes that can be viewed will be returned

How to delete data files in el

datetime=date +%Y.%m.%d -d "3 days ago"

curl -XDELETE "http://127.0.0.1:9200/logstash-2017.08.19" 

2.2 Install LogStash

LogStash has multiple installation forms. This section mainly uses compressed package decompression and installation as an example to introduce.
The reason why kafka is used for data storage is to reduce the pressure on the front end of the Es cluster, so the message queue Kafka is added as a transition.
The operation of LogStash needs to specify a configuration file to specify the flow direction of the data. Create a "log-collect.conf" file in the "logstash-7.5.0.tar.gz/config" directory, and its content is as follows.

log-collect.conf

#日志导入
input {
    
    
    kafka {
    
    
        bootstrap_servers => ["192.168.16.167:9092"]
        #或
   zk_connect => "192.168.4.1:2181,192.168.4.2:2181,192.168.4.3:2181"
   client_id => "eoslog"
        group_id => "eoslog"
        auto_offset_reset => "latest"
        topics => ["192.168.16.169-60000","192.168.16.169-60001","192.168.16.169-60002"]
        type => "eos"
        codec => "json"
        consumer_threads => 5
        decorate_events => true
   compression_type => "snappy"  # 压缩方式
    }
}
 
 
input {
    
    
   tcp {
    
    
       port => 8888
       mode => "server"
       ssl_enable => false
   }
 
   tcp {
    
    
       port => 9999
       mode => "server"
       ssl_enable => false
   }
}
 
 
input {
    
    
       redis {
    
    
               data_type => "list"
               type => "redis-input"
               key => "logstash:redis"
               host => "192.168.212.37"
               port => 6379
               threads => 5
               codec => "json"
       }
}
 
 
 
 
# 日志筛选匹配处理
filter{
    
    
    grok {
    
    
 match => {
    
     "source" => "%{GREEDYDATA}/%{GREEDYDATA:app}-%{GREEDYDATA:logtype}.log" }
        add_field => {
    
     "appID" => "%{app}" }
        add_field => {
    
     "logType" => "%{logtype}" }
    }
}
# 日志匹配输出,输出到ES中
output {
    
    
 elasticsearch {
    
    
   hosts => ["11.104.34.149:9200"]
   action => "index"
   index => "eos=%{+YYYY.MM.dd}"
   user => elastic     #安装x-pack后，需要配置认证
   password => changeme  #安装x-pack后，需要配置认证
}
}
 
 
output {
    
    
       elasticsearch {
    
    
               hosts => "192.168.212.37:9201"
               index => "logstash-test"
       }
       stdout {
    
    
               codec => rubydebug {
    
    }
       }
}
 

Description:
Configure LogStash to output logs to ElasticSearch.

1. In the "PFPJ_2.1_Log_Center\logstash-5.6.8\bin" directory, execute the following command to install LogStash.

Unix

#./logstash -f ../config/log-collect.conf --config.reload.automatic

2.3 Install Kibana

1. Unzip kibana-7.5.0-linux-x86_64.tar.gz
2. Use a text editor to open the "kibana.yml" file in the "kibana-7.5.0-linux-x86_64/config" directory, and configure the following parameters.

kibana.yml

# Kibana is served by a back end server. This setting specifies the port to use.
server.port: 5601
 
# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
server.host: 0.0.0.0
 
# Enables you to specify a path to mount Kibana at if you are running behind a proxy. This only affects
# the URLs generated by Kibana, your proxy is expected to remove the basePath value before forwarding requests
# to Kibana. This setting cannot end in a slash.
#server.basePath: ""
 
# The maximum payload size in bytes for incoming server requests.
#server.maxPayloadBytes: 1048576
 
# The Kibana server's name.  This is used for display purposes.
#server.name: "your-hostname"
 
# The URL of the Elasticsearch instance to use for all your queries.
elasticsearch.url: "http://192.168.16.85:9200"
 

1. In the "bin" directory, execute the following command to install Kibana.

# ./kibana

2.4 Install Kafka

2.4.1 Configure zookeeper

1. Open the "zookeeper.properties" file in the "PFPJ_2.1_Log_Center\kafka_2.11-0.10.1.0\config" directory with a text editor, and configure the following parameters.

zookeeper.properties

(略……)
# the directory where the snapshot is stored.
dataDir=/opt/zookeeper
# the port at which the clients will connect
clientPort=2181
# disable the per-ip limit on the number of connections since this is a non-production config
maxClientCnxns=0
initLimit=5
syncLimit=2
server.1=192.168.16.85:2888:3888
#server.2=192.168.16.86:2888:3888
#server.2=192.168.16.87:2888:3888

2. In the "bin" directory, execute the following command to start the zookeeper server instance.

# nohup ./zookeeper-server-start.sh ../config/zookeeper.properties &

2.4.2 Deploying Kafka

1. Open the "server.properties" file in the "PFPJ_2.1_Log_Center\kafka_2.11-0.10.1.0\config" directory with a text editor, and configure the following parameters.
server.properties

(略……)
 
############################# Server Basics #############################
 
# The id of the broker. This must be set to a unique integer for each broker.
broker.id=1
 host.name=11.104.34.149
listeners=PLAINTEXT://11.104.34.149:9092
advertised.listeners=PLAINTEXT://11.104.34.149:9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
(略……)
 
############################# Log Basics #############################
 
# A comma seperated list of directories under which to store log files
log.dirs=opt/kafka_2.11-0.10.1.0/logs
 
# The default number of log partitions per topic. More partitions allow greater
# parallelism for consumption, but this will also result in more files across
# the brokers.
num.partitions=1
 num.recovery.threads.per.data.dir=1
(略……)
############################# Zookeeper #############################
 
# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
 
 
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
zookeeper.connect=192.168.16.85:2181
 
# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=6000

3. In the "bin" directory, execute the following command to start the Kafka server instance.

# kafka-server-start.sh ../config/server.properties

The following steps 4 and 5 are optional operations.
4. In the "PFPJ_2.1_Log_Center\kafka_2.11-0.10.1.0\bin" directory, execute the following command to create a kafka topic. Used to test whether kafka starts successfully.

Unix
# kafka-topics.sh --list --zookeeper 192.168.16.85:2181
 
Windows
> windows\kafka-topics.bat --list --zookeeper 192.168.16.85:2181

5. In the "PFPJ_2.1_Log_Center\kafka_2.11-0.10.1.0\bin" directory, execute the following commands to open a Producer and a Consumer. Producer is used to send messages, and Consumer receives messages. Used to test whether kafka starts successfully.

Unix
# kafka-console-producer.sh --broker-list 192.168.16.85:9092 --topic 192-168-16-50.node
# kafka-console-consumer.sh --zookeeper 192.168.16.85:2181 --topic 192-168-16-50.node --from-beginning
 
Windows
> windows\kafka-console-producer.bat --broker-list 192.168.16.85:9092 --topic 192-168-16-50.node
> windows\kafka-console-consumer.bat --zookeeper 192.168.16.85:2181 --topic 192-168-16-50.node --from-beginning

2.4.3 Verification

1. After kibana is started, you can access http://192.168.16.85:5601/, as shown in the figure below.
   illustrate:

Kibana is connected to port 9200 of the local machine by default, and the bound port is 5601, where "192.168.16.85" is the server host IP address, configured in "kibana.yml".

2.5 Install Filebeat

Filebeat is used to collect logs in a specified single directory or multiple directories. This chapter takes configuring multiple directories as an example to explain.

Scenario: Three microservice instances are deployed on a machine: ordermgr, productmgr, and stockmgr. The instance ports are 60000, 60001, and 60002 respectively. Each microservice instance has its own log storage directory.

2.5.1 Installation

Use a text editor to open the "filebeat.yml" file under "PFPJ_2.1_Log_Center\filebeat-6.2.4-windows-x86_64" and configure the relevant parameters according to the following table.
filebeat.yml

(略……)
 
#=========================== Filebeat prospectors =============================
 
filebeat.prospectors:
 
# Each - is a prospector. Most options can be set at the prospector level, so
# you can use different prospectors for various configurations.
# Below are the prospector specific configurations.
 
- type: log
 
  # Change to true to enable this prospector configuration.
  enabled: true
 
  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /home/ordermgr/logs/*.log
  fields:
    log_topics: 192.168.16.169-60000
 
- type: log
  enabled: true
  paths:
    - /home/productmgr/logs/*.log
  fields:
    log_topics: 192.168.16.169-60001
 
- type: log
  enabled: true
  paths:
    - /home/stockmgr/logs/*.log
  fields:
    log_topics: 192.168.16.169-60002
 
(略……)
 
#========================== Filebeat modules ===============================
 
filebeat.config.modules:
 path: ${path.config}/modules.d/*.yml
 reload.enabled: false
#==================== Elasticsearch template setting ==========================
setup.template.settings:
 index.number_of_shards: 3
#=============================== General ===================================
 
 
#=========================== Dashboards ===================================
 
#============================== Kibana =====================================
setup.kibana:
 
 host: "11.104.34.149:5601"
#=============================Elastic Cloud ==================================
 
#===============================Outputs====================================
#-------------------------- Kafka output ------------------------------
output.kafka:
  hosts: ['192.168.16.85:9092'] #Kafka所在的ip地址及端口
  topic: '%{[fields][log_topics]}'
  partition.round_robin:
    reachable_only: false
  required_acks: 1
  compression: gzip
  max_message_bytes: 1000000
 
#-------------------------- Elasticsearch output ------------------------------
#output.elasticsearch:
 # Array of hosts to connect to.
# hosts: ["20.200.26.86:9200"]
 
 # Optional protocol and basic auth credentials.
 #protocol: "https"
 #username: "elastic"
 #password: "changeme"
 
#----------------------------- Logstash output --------------------------------
#output.logstash:
 # The Logstash hosts
 #hosts: ["localhost:5044"]
 
 # Optional SSL. By default is off.
 # List of root certificates for HTTPS server verifications
 #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]
 
 # Certificate for SSL client authentication
 #ssl.certificate: "/etc/pki/client/cert.pem"
 
 # Client Certificate Key
 #ssl.key: "/etc/pki/client/cert.key"
 
#================================ Logging =====================================
 
# Sets log level. The default log level is info.
# Available log levels are: error, warning, info, debug
#logging.level: debug
 
# At debug level, you can selectively enable logging only for some components.
# To enable all selectors use ["*"]. Examples of other selectors are "beat",
# "publish", "service".
#logging.selectors: ["*"]
 
#============================== Xpack Monitoring ===============================
# filebeat can export internal metrics to a central Elasticsearch monitoring
# cluster.  This requires xpack monitoring to be enabled in Elasticsearch.  The
# reporting is disabled by default.
 
# Set to true to enable the monitoring reporter.
#xpack.monitoring.enabled: false
 
# Uncomment to send the metrics to Elasticsearch. Most settings from the
# Elasticsearch output are accepted here as well. Any setting that is not set is
# automatically inherited from the Elasticsearch output configuration, so if you
# have the Elasticsearch output configured, you can simply uncomment the
# following line.
#xpack.monitoring.elasticsearch:

2. In the "filebeat-6.2.4-windows-x86_64" directory, install FileBeats.

Unix
# sudo chown root filebeat.yml
# sudo ./filebeat -e -c filebeat.yml -d "publish"
 
Windows
> filebeat.exe -e -c filebeat.yml

2.5.2 Uninstall

You can directly delete the "PFPJ_2.1_Log_Center" directory to uninstall.

3. ELK deployment process problem set

3.1 Redis is also available, why choose kafka?

We all know that Redis uses key hashing to store data in distributed columns, and when Redis is used as a cluster, the corresponding application corresponds to a Redis, which to some extent will cause data skew, leading to data loss.
Judging from the previous deployment of Kafka cluster, a Kafka topic can have multiple partitions (copies), and they are evenly distributed on the Kafka cluster, so there will be no data skew like redis. Kafka also has the redundancy mechanism of Redis. If a machine in a Redis cluster goes down, it is very likely to cause data loss. However, because Kafka is evenly distributed on the cluster hosts, even if a machine goes down, it will not be affected. use. At the same time, Kafka, as a subscription messaging system, also has high throughput of millions per second, persistence, and distributed features.

3.2 El always initializes kibana

【ElasticSearch is still initializing the kibana index】

solution:

3.3 plugin:[email protected] Service Unavailable

日志现象：
[2019-12-14T14:35:15,840][INFO ][o.e.d.z.ZenDiscovery ] [yelksvc1] failed to send join request to master [{yelksvc2}{9rVke5zYRL-qq8EXynIAvw}{M4KyckULTfehOLKBJCOahQ}{11.104.34.150}{11.104.34.150:9300}], reason [RemoteTransportException[[yelksvc2][11.104.34.150:9300][internal:discovery/zen/join]]; nested: NotMasterException[Node [{yelksvc2}{9rVke5zYRL-qq8EXynIAvw}{M4KyckULTfehOLKBJCOahQ}{11.104.34.150}{11.104.34.150:9300}] not master for join request]; ], tried [3] times

Reason: The data is inconsistent. Is the directory where data is stored in the elasticsearch directory inconsistent with the data of this node?

Solution: https://blog.csdn.net/HuoqilinHeiqiji/article/details/88402637

The data directory data is inconsistent. Because it is copying someone else's elasticsearch, the original node has data. During the copying process, the data also stores the original information. Because it is a local test environment, after the data directory is deleted, everything returns to normal! In fact, the error indicated by the log is obvious. Each node is started independently, but the read data is consistent, which makes it impossible for multiple nodes to form a cluster.

3.4 Kafka出现：Error while fetching metadata with correlation id : {LEADER_NOT_AVAILABLE}

Reason: kafka appears:
Error while fetching metadata with correlation id: {LEADER_NOT_AVAILABLE}
means that kafka hostname cannot be recognized

solution:

4. Page index configuration

https://www.cnblogs.com/martin-wang/p/10656673.html

curl -XPUT http://11.104.34.150:9200/eos=*

To view the status of the elasticsearch cluster, run the command in the Kibana console:
GET /_cat/health?V

[View zoopeer cluster status]
./kafka-topics.sh --describe --zookeeper 11.104.34.150:2181

[View el cluster status]
curl http://11.104.34.152:9200/_nodes/process?pretty

5. x-pack installation

查询所有用户
curl -XGET -u elastic 'localhost:9200/_xpack/security/user?pretty' 
查询所有Roles
curl -XGET -u elastic 'localhost:9200/_xpack/security/role'

[Installing X-pack in elk]
Reference address:
https://segmentfault.com/a/1190000010981283
https://www.cnblogs.com/xbq8080/p/6547765.html

bin/logstash-plugin install /app/elk/soft/x-pack-5.5.2.zip

Unable to download data from https://rubygems.org - SocketError: initialize: name or service not known (https://rubygems.org/latest_specs.4.8.gz)
ERROR: Installation aborted, verification failed for /app/elk/soft/x-pack-5.5.2.zip

参考地址：https://blog.csdn.net/weixin_34392906/article/details/91938797

bin/logstash-plugin install --no-verify /app/elk/soft/x-pack-5.5.2.zip

Error Bundler::HTTPError, retrying 10/10
Could not fetch specs from https://rubygems.org/
Too many retries, aborting, caused by Bundler::HTTPError
ERROR: Installation Aborted, message: Could not fetch specs from https://rubygems.org/

elasticsearch安装x-pack过程

Check whether the x-pack package size is correct:

Kibana installation x-pack process

logstash installation x-pack process

x-pack default user password
elastic/changeme

X-PACK change password
https://www.cnblogs.com/xbq8080/p/6547765.html