A, ELK ready platform to build
1.2 ELK Download: https://www.elastic.co/downloads/
cd /data/package/ wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.3.2.tar.gz wget https://artifacts.elastic.co/downloads/logstash/logstash-6.3.2.tar.gz wget https://artifacts.elastic.co/downloads/kibana/kibana-6.3.2-linux-x86_64.tar.gz
1.3 Installation Preparation
# Configure iptables, to ensure that the network can communicate between [the root @ Elk_Server] # iptables -F; -I the INPUT iptables -s 192.168.0.0/16 -j ACCEPT [@ Elk_Server the root] # iptables Save-Service; iptables the restart-Service # Close SELinux [@ Elk_Server the root] # the setenforce 0 [@ Elk_Server the root] # Vim / etc / sysconfig / SELinux the SELINUX = Disabled
1.4 Time Synchronization
cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime yum install ntpdate -y;ntpdate time.windows.com echo '01 00 * * * ntpdate time.windows.com' >>/etc/crontab
Source 1.5 yum Configuration
#导入公钥 [root@Elk_Server ~]# rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch [root@Elk_Server ~]# vim /etc/yum.repos.d/elasticsearch.repo [elasticsearch-6.x] name=Elasticsearch repository for 6.x packages baseurl=https://artifacts.elastic.co/packages/6.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=1 autorefresh=1 type=rpm-md
Second, the stand-alone cluster installation Elasticsearch
In fact, this is also the core of ELK start when we must pay attention, because es root account can not be started, so you need to open a elsearch account. **
2.1.1 yum installed
[root@Elk_Server ~]# sudo yum install elasticsearch -y
2.1.2 Source Installation (herein installation configuration for compilation)
# The file unpacked three copies of the tar-6.3.2.tar.gz -C zxvf elasticsearch / usr / local / Music Videos /usr/local/elasticsearch-6.3.2 / usr / local / ES-Master CP -rf / usr / local / ES-Master / usr / local / ES-DATAl CP -rf / usr / local / ES-Master / usr / local / ES-DATA2 the groupadd elasticsearch the useradd -g elasticsearch elasticsearch
2.2 Creating a data directory elasticsearch
mkdir -p /data{,1,2}/elasticsearch/data
mkdir -p /data{,1,2}/elasticsearch/logs
chown -R elasticsearch:elasticsearch /data{,1,2}/elasticsearch
2.3 Editing Elasticsearch cluster configuration file
/usr/local/es-master/config/elasticsearch.yml /usr/local/es-master/config/elasticsearch.yml-bak cp # Create a soft link mkdir / etc / elasticsearch / LN -s / usr / local / es -master / config / elasticsearch.yml /etc/elasticsearch/es-master.yml LN -s /usr/local/es-data1/config/elasticsearch.yml /etc/elasticsearch/es-data1.yml LN -s / usr /local/es-data2/config/elasticsearch.yml /etc/elasticsearch/es-data2.yml
The master node configuration:
/etc/elasticsearch/es-master.yml vim # ======================== elasticsearch the Configuration ============ ============= # cluster.name: Elk-Cluster node.name: Master Node- node.attr.rack: R1 # -------------- --------------------- Paths ---------------------------- -------- path # is stored in data set / data / elasticsearch / data Path.Data: / data / elasticsearch / data # set log logs path / data / elasticsearch / logs path.logs: / data / elasticsearch / logs # ---------------------------------- ---------- Network ------------------------- # the local address network.host: 192.168.6.108 # open listening port 9200 http.port: 9200 #tcp communication port transport.tcp.port: 9330 # whether as a host node.master: to true # Whether as a data node node.data: false node.max_local_storage_nodes: 3 # when you delete the index, the need to explicitly specify the name of the value is false, you can delete by regular or _all:. Action.destructive_requires_name: to true # - ------------------------------- Discovery ------------------ ---------------- # cluster information, the default communication port is 9300 discovery.zen.ping.unicast.hosts: [ "192.168.6.108:9330", "192.168.6.108: 9331 "," 192.168.6.108:9332 "] # set up early in the new cluster, there will always be a few nodes to communicate with other nodes frequently cause abnormal join and leave the cluster. This process is performed automatically. To control the response time of a cluster node join or start elections (default 3s) by configuring discovery.zen.ping_timeout. discovery.zen.ping_timeout: 60s # The closed automatically create the index. As also safety considerations, or even within the network, there are many scanners, once opened, the scanner will automatically give you a lot to create the index. action.auto_create_index: false # ---------------------------------- Gateway --------- -------------------------- # # Block initial recovery after a full cluster restart until N nodes are started: data recovery cluster # Set the start node N, the default is one. gateway.recover_after_nodes: 2 # Set the initialization data recovery process timeout, the default is 5 minutes. gateway.recover_after_time: 3m Number # Set the nodes in the cluster, the default is 2, N nodes once it starts, it will immediately perform data recovery. gateway.expected_nodes: 3 # ---------------------------------- Various --------- -------------------------- # Enable cross-domain access support, default is false http.cors.enabled: to true # allow cross-domain access domain name address, (allow all domain names) than using regular http.cors.allow-origin: /.*/
1 data node configuration:
/etc/elasticsearch/es-data1.yml vim # ======================== elasticsearch the Configuration ============ ============= # cluster.name: Elk-Cluster node.name: Node-DATAl node.attr.rack: R1 # -------------- --------------------- Paths ---------------------------- -------- path # set data is stored in / DAT1 / elasticsearch / data Path.Data: / DATAl / elasticsearch / data # set path logs log / DATAl / elasticsearch / logs path.logs: / DATAl / elasticsearch / logs # ---------------------------------- ---------- Network ------------------------- # the local address network.host: 192.168.6.108 # open listening port 9201 http.port: 9201 #tcp communication port transport.tcp.port: 9331 # whether as a host node.master: false # whether as a data node node.data: to true node.max_local_storage_nodes: 3 # when you delete the index, the need to explicitly specify the name of the value is false, you can delete by regular or _all:. action.destructive_requires_name : to true # --------------------------------- ------------- Discovery --------------------- # cluster information, the default communication port is 9300 discovery.zen.ping.unicast.hosts: [ "192.168.6.108:9330", "192.168.6.108:9331","192.168.6.108:9332"] discovery.zen.ping_timeout: 60s action.auto_create_index: false # --------------------- Gateway ----------------------------------- ------------- # # Block initial recovery after a full cluster restart until N nodes are started: data recovery cluster # set the start node N, the default is one. gateway.recover_after_nodes: 2 # Set the timeout time initialization data recovery process, the default is 5 minutes. gateway.recover_after_time: 3M gateway.expected_nodes: 3 # ---------------------------------- Various ---- ------------------------------- # Enable cross-domain access support, default is false http.cors.enabled: to true # cross domain name address to allow access, (allow all domain names) than using regular http.cors.allow-origin: /.*/
2-node configuration data:
/etc/elasticsearch/es-data2.yml vim # ======================== elasticsearch the Configuration ============ ============= # cluster.name: Elk-Cluster node.name: Node-DATA2 node.attr.rack: R1 # -------------- --------------------- Paths ---------------------------- -------- path # set data is stored in / DATA2 / elasticsearch / data Path.Data: / DATA2 / elasticsearch / data # set path logs log / DATA2 / elasticsearch / logs path.logs: / DATA2 / elasticsearch / logs # ---------------------------------- ---------- Network ------------------------- # the local address network.host: 192.168.6.108 # open listening port 9202 http.port: 9202 #tcp communication port transport.tcp.port: 9332 # whether as a host node.master: false # whether as a data node node.data: to true node.max_local_storage_nodes: 3 # when you delete the index, the need to explicitly specify the name of the value is false, you can delete by regular or _all:. action.destructive_requires_name : to true # --------------------------------- ------------- Discovery --------------------- # cluster information, the default communication port is 9300 discovery.zen.ping.unicast.hosts: [ "192.168.6.108:9330", "192.168.6.108:9331","192.168.6.108:9332"] discovery.zen.ping_timeout: 60s # the closed automatically create the index. As also safety considerations, or even within the network, there are many scanners, once opened, the scanner will automatically give you a lot to create the index. action.auto_create_index: false # ---------------------------------- Gateway --------- -------------------------- # # Initial Block A Full Recovery After an until the restart Cluster Nodes are N Started: # Set when data recovery start cluster nodes N, the default is one. gateway.recover_after_nodes: 2 # Set the initialization data recovery process timeout, the default is 5 minutes. gateway.recover_after_time: 3M gateway.expected_nodes: 3 # ---------------------------------- Various ---- ------------------------------- # Enable cross-domain access support, default is false http.cors.enabled: to true # cross domain name address to allow access, (allow all domain names) than using regular http.cors.allow-origin: /.*/
Note the following when using the external network to build clusters:
network.host need to be modified to 0.0.0.0, at the same time to expose your external network address, as follows:
network.host: 0.0.0.0 network.publish_host: xx.xx.xx.xx
discovery.zen.ping.unicast.hosts modify the external network address and communications port to each server.
2.4 Installation head plug
head plugin can generate statistics for the cluster, and provides a browser queries, while the index of elasticsearch structured query.
CD / usr / local / ES-Master /
mkdir && head CD head
Git clone https://github.com/mobz/elasticsearch-head.git
CD-head elasticsearch /
#install Node V7.x
curl --silent --location HTTPS : //rpm.nodesource.com/setup_7.x | bash -
sudo yum install nodejs
npm the SET Registry config = HTTP: //registry.npm.taobao.org
npm install ## runs slowly
npm install -g Grunt-cli
## modify profile
[@ Elk_Server the root-head elasticsearch] # Vim Gruntfile.js
Connect: {
Server: {
Options: {
Port: 9100,
hostname: '192.168.6.108', add this line ##, there is a space after the colon
base: '.',
keepalive: true
}
}
}
2.5 System parameter adjustment
## es modify JVM memory to run three nodes, this can be changed as needed, but -Xms and -Xmx value must be the same, or start error [root @ Elk_Server] # vim / usr / local / es-master / config / jvm.options [root @ Elk_Server] # vim /usr/local/es-data1/config/jvm.options [root @ Elk_Server] # vim /usr/local/es-data2/config/jvm.options ----- -------------------------------------------------- ------------------ -Xms2g -Xmx2g ## to modify the maximum number of open files Linux vim /etc/security/limits.conf added at the end nofile 655350 ## Soft * * Hard 655350 nofile ## this step requires restart linux configuration to take effect reboot [root @ Elk_Server ~] # ulimit -n 655350 ## modify sysctl.conf file [root @ Elk_Server ~] # vim /etc/sysctl.conf vm.max_map_count = 655350 [root @ Elk_Server ~] # sysctl -p
The ES 2.6 plug-in head start
CD / usr / local / ES-Master / head / head-elasticsearch Grunt Server & && [. 3] 5946 ## 9100 port opens.
2.7 start Elasticsearch (ES Use only regular user-initiated)
-R & lt elasticsearch chown: elasticsearch / usr / local / ES- *
chown -R & lt elasticsearch: elasticsearch / Data {,} 1,2 / elasticsearch
## is switched to the normal user starts elasticsearch
SU - elasticsearch
the nohup / usr / local / ES-Master / bin / elasticsearch & # -d to boot elasticsearch way station or & later
[1] 6202
## start 9200,9300 success will open two ports.
## transmission port is port 9300 to accept http request for 9200
## open cases can view the cluster using the plug head and the index elasticsearch in the browser
2.9 Verify start
ps axu | grep elasticsearch
Method a:
curl 'http://192.168.6.108:9200/_search?pretty'
-----
{
"Took":. 9,
"TIMED_OUT": to false,
"_shards": {
"Total" : 0,
"successful": 0,
"Skipped": 0,
"failed": 0
},
"Hits": {
"Total": 0,
"max_score": 0.0,
"Hits": []
}
}
method two:
// or accessed through a browser: http: //192.168.6.108: 9200 /
Checking version information # es
curl -u <User>: <the passwd> http://192.168.6.108:9200
# ES observed at this time the state of the cluster:
curl http://192.168.6.108:9200/_cluster/health?pretty
# Observation each state in the cluster index:
curl http://192.168.6.108:9200/_cat/indices?pretty
# elasticsearch node status query:
curl -XGET http://192.168.6.108:9200/_cat/shards | grep UNASSIGNED
# View node a list of
HTTP: // IP: 9200 / _cat / v Nodes?
curl 'IP:? 9200 / _cat / v Nodes'
# list all indexes and storage size
HTTP: // IP:? 9200 / _cat / v indices
curl' IP ?: 9200 / _cat / indices v '- all index and query data size
# create an index
created index called XX, the default will be five slices, an index
curl -XPUT' IP: 9200 / XX Pretty '?
# Add a type
curl -XPUT 'the IP:? 9200 / XX / External / 2 Pretty' -d '
{
"gwyy": "John"
}'
# Update a type of
curl -XPOST 'IP:? 9200 / XX / External / 1 / _Update Pretty' -d '
{
"DOC": { "name": "Jaf"}
}'
# delete the specified index
curl -XDELETE 'IP : 9200 / _index pretty '?
Third, the installation Logstash:
Logstash the following functions:
In fact, it is just a collector, we need to specify the Input and Output for it (of course, Input and Output may be plural). Since we need to put Java code Log4j logging output to ElasticSearch, so here is Log4j Input, Output and is ElasticSearch.
3.1 Installation Logstash
tar zxvf logstash-6.3.2.tar.gz -C /usr/local/ mv /usr/local/logstash-6.3.2 /usr/local/logstash ln -s /usr/local/logstash/bin/logstash /usr/bin/ mkdir -p /data/logs/logstash
3.2 verify whether the installation is successful, we can see from the figure below, is currently open port 9600
-e logstash 'stdin INPUT {{Output} {} {the hosts elasticsearch => [ "192.168.6.108:9200"]} {CODEC = stdout> rubydebug}}'
### "stdin of The plugin IS Waiting after waiting a while now for input: ", input" test "carriage return following output standards.
-----
[2018-07-31T07: 34 is: 23,968] [the INFO] [logstash.agent] Successfully Started Logstash the API {Endpoint: Port => 9600}
Test
{
"@version" => ". 1",
"Host "=>" Elk_Server ",
" Message "=>" Test ",
" @timestamp "=> 2018-07-31T07: 34 is: 40.422Z
}
3.3 logstash Configuration
## Next is the key, but also the difficulties ELK it, is to log analysis, using regular match content filtering logs want.
[root@Elk_Server logstash]# cat /data/logs/test.log 120.79.189.51 - - [10/Jun/2018:12:58:12 +0800] "POST /wp-cron.php?doing_wp_cron=1528606692.3628709316253662109375 HTTP/1.1" 499 0 "-" "WordPress/4.8; http://120.79.189.51"
## This is a log row, logstash regular match the default match line, a carriage return as a delimiter.
## Of course, if the log is json format then there is no need to deal with direct output can be.
#我们到config目录中新建一个logstash.conf配置。
[root@Elk_Server logstash]# vim /etc/logstash/conf.d/logstash.conf
input {
file {
path => "/data/logs/test.log"
type => "test.log"
start_position => "beginning"
}
}
filter {
grok {
match => {
"message" => "(?<ip_addr>\d+?\.\d+?\.\d+?\.\d+?)\s-\s-\s\[(?<log_date>\d+?/\w+?/\d+?):(?<log_time>\d+?:\d+?:\d+?)\s+"
}
}
}
output {
stdout {codec => rubydebug}
}
## This is logstash log file analysis, a total of three majority, input, flter, output, where the input is the input log, choose here is the file file, path is the path to the file, type is the type of file, this It can be customized, mainly used to distinguish each log, beginning start_position means is set to start reading from the beginning of the file.
## flter is partially processed logs, use grok this powerful component information filtered for positive log of the match most always format output json, so regular match format is (? <field name> regular expression filtering will be matching content) create content filtering in each of the brackets, the unwanted content in brackets, the match could have been done to know the end of the log, here I simply match in front of the log ip address and date and time.
For regular match field Details:
## regular match fields defined by "(<field name> regular expression?)," We see "(<ip_addr> \ d + \ d + \ d + \??.?.?. d +? "matches the IP address of the log, and then will not need to match the contents of the brackets on the outside, followed by the date and time, using the same method, and unwanted content on the outside because I am here only match the previous a little content, if there is need to extract, you can always match, knowing that the end of the line, logstash only match the default log line as the split, the next line is a new one matches.
#output is processing the output section, here I simply output to the terminal, after the first experiment the regular matching no problem, the output point elasticsearch.
[root@Elk_Server logstash]# logstash -r -f /etc/logstash/conf.d/logstash.conf
..............
{
"path" => "/data/logs/test.log",
"host" => "Elk_Server",
"type" => "test.log",
"log_date" => "10/Jun/2018",
"@version" => "1",
"log_time" => "12:58:12",
"@timestamp" => 2018-07-31T09:45:16.247Z,
"ip_addr" => "120.79.189.51",
"message" => "120.79.189.51 - - [10/Jun/2018:12:58:12 +0800] \"POST /wp-cron.php?doing_wp_cron=1528606692.3628709316253662109375 HTTP/1.1\" 499 0 \"-\" \"WordPress/4.8; http://120.79.189.51\""
}
Next, the log output to elasticsearch, modified to output the contents inside.
{Output
elasticsearch {## output to the specified ES
Action => "index" to index ##
hosts => "192.168.6.108:9200" ## ES address port
index => "test_log -% { + yyyy.MM.dd} "name index ##
}
stdout = {CODEC> rubydebug}
}
3.4 verify the configuration, and report errors
logstash -f /etc/logstash/conf.d/logstash.conf --config.test_and_exit \\或者 logstash -t -f /etc/logstash/conf.d/logstash.conf
3.5 service starts
nohup logstash -f /etc/logstash/conf.d/logstash.conf --config.reload.automatic -l /data/logs/logstash & \\或者 nohup logstash -r -f /etc/logstash/conf.d/logstash.conf -l /data/logs/logstash &
--config.reload.automatic option enables automatic configuration reload, so you do not have to stop every time you modify the configuration file and restart Logstash.
3.6 combined filebeat case
log reads as follows:
2018-12-20 17:40:50.665 INFO 1 --- [Thread-422] f.s.j.e.common.iml.AbstractJobWorker : no task in app f.s.job.executor.social.users's job 0
2018-12-22 11:23:55.947 INFO -- [cluster-ClusterId{value='5xxx001145200', description='null'}-dds-xxxc9e41.mongodb.rds.aliyuncs.com:3717] org.mongodb.xxx.cluster : Server xxx:3717 is no longer a member of the replica set. Removing from client view of cluster.
filebeat profile:
#=========================== Filebeat inputs =============================
filebeat.inputs:
- type: log
enabled: true
paths:
- /data/log/*.log
fields:
app: tomcat
type: all
fields_under_root: true
#multiline.pattern: '^\['
multiline.pattern: '^\d+\-\d+\-\d+\s+\d+\:\d+\:\d+'
multiline.negate: true
multiline.match: after
multiline.max_lines: 200
multiline.timeout: 10s
#tags: ["filebeat"]
#exclude_lines: ['^\d+\-\d+\-\d+\s+\d+\:\d+\:\d+\s+DEBUG']
#include_lines: ['^ERR', '^WARN']
#----------------------------- Logstash output --------------------------------
output.logstash:
# enabled: true
hosts: ["127.0.0.1:5044"]
# index => "filebeat-%{+YYYY.MM.dd}"
# worker: 4
# bulk_max_size: 1024
# compression_level: 6
# loadbalance: false
# backoff.max: 120s
logstash profile:
[root@Elk_Server logstash]# vim /etc/logstash/conf.d/logstash.conf
input {
beats {
host => '0.0.0.0'
port => "5044"
}
}
filter {
grok {
match => { "message" =>
["%{TIMESTAMP_ISO8601:date}\s+(?:%{LOGLEVEL:loglevel}|%{LOGLEVEL:loglevel}\s+\d) \-+ \[%{NOTSPACE:servicename}\] %{JAVALOGMESSAGE:message}",
"%{TIMESTAMP_ISO8601:date}\s+(?:%{LOGLEVEL:loglevel}|%{LOGLEVEL:loglevel}\s+\d) \-+ \[%{NOTSPACE:servicename}\{%{DATA:description}\}-%{DATA:db}\] %{JAVALOGMESSAGE:message}"]
}
overwrite => [ "message" ]
}
#geoip {
# source => "remote_addr"
# target => "geoip"
# database => "/opt/GeoLite2-City.mmdb"
# add_field => ["[geoip][coordinates]", "%{[geoip][longitude]}"]
# add_field => ["[geoip][coordinates]", "%{[geoip][latitude]}"]
#}
date {
locale => "en"
match => ["date", "dd/MMM/yyyy:HH:mm:ss Z"]
}
}
output {
elasticsearch {
hosts => [ "192.168.6.108:9200" ]
index => "logstash-%{type}-%{+YYYY.MM.dd}"
}
# stdout {codec => rubydebug}
}
Fourth, the installation Kibana
4.1 AnSo Kibana
tar zxvf kibana-6.3.2-linux-x86_64.tar.gz -C /usr/local/ mv /usr/local/kibana-6.3.2-linux-x86_64 /usr/local/kibana
4.2 arrangement Kibana
CD / usr / local / kibana / Vim config / kibana.yml # released the following notes the function configuration. ----- server.port: 5601 server.host: "192.168.6.108" elasticsearch.url: "http://192.168.6.108:9200" kibana.index: ".kibana"
4.3 Start Kibana and test access can be seen from the log, the current open port 5601
./bin/kibana #后台启动 nohup /usr/local/kibana/bin/kibana & ----- log [03:30:39.833] [info][status][plugin:[email protected]] Status changed from uninitialized to green - Ready log [03:30:39.882] [info][status][plugin:[email protected]] Status changed from uninitialized to yellow - Waiting for Elasticsearch log [03:30:40.026] [info][status][plugin:[email protected]] Status changed from uninitialized to green - Ready log [03:30:40.033] [info][status][plugin:[email protected]] Status changed from uninitialized to green - Ready log [03:30:40.037] [info][status][plugin:[email protected]] Status changed from uninitialized to green - Ready log [03:30:40.061] [info][listening] Server running at http://192.168.30.211:5601 log [03:30:40.117] [info][status][plugin:[email protected]] Status changed from yellow to green - Ready
4.4 View starts without error, the browser can be accessed through the 192.168.6.108:5601.
Add index
Discover ① Click the following interface, logstash log file analysis which has established an index, the index name is test_log - $ {date} this form, it is an index in the beginning of the name Elasticsearch. Kibana automatically detects whether the index name exists in Elasticsearch.
② According to comments configuration and click Next step, create a second page click create select @timestamp
After ③ created, you can see one of the following domain interface, the red box is automatically generated, can be understood as similar with fields in the database, which has a message field, what we want to log information.
④ click Discover again the following interface, you can see the default search is the last 15 minutes of the log, the time range can search by clicking on settings.
⑤ On the right side fields can add domain set to be displayed 
after the addition is completed, the log is shown below:
⑥ can also view and respond to the request body member