一.前提条件
1.操作系统版本:
Centos 6.5
2.安装顺序
a. elasticsearch
b.kibana
c.logstash
d.beats
3.安装jdk-8u131-linux-x64.tar.gz
#vi /etc/profile
export JAVA_HOME=/usr/local/java
export CLASSPATH=$JAVA_HOME/lib/
export PATH=$PATH:$JAVA_HOME/bin
#source /etc/profile
4.ES stack基本概念
database → tables → rows → column indices → type → document → fields
二.安装ES
1.创建es用户启动ES,否则会报错
Caused by: java.lang.RuntimeException: can not run elasticsearch as root
at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:105) ~[elasticsearch-6.2.3.jar:6.2.3]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:172) ~[elasticsearch-6.2.3.jar:6.2.3]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) ~[elasticsearch-6.2.3.jar:6.2.3]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-6.2.3.jar:6.2.3]
#useradd es #passwd es #chown -R es:es /usr/local/elasticsearch-6.2.3 #/usr/local/elasticsearch-6.2.3/bin/elasticsearch -d #后台启动ES
修改es配置文件内容:
[es@elk config]$ cat elasticsearch.yml | grep -v '^#'
path.data: /usr/local/elasticsearch-6.2.3/data
path.logs: /usr/local/elasticsearch-6.2.3/logs
network.host: 172.16.68.150
http.port: 9200
[es@elk config]$
如启动失败，通过日志分析并处理：
[4] bootstrap checks failed
[1]: max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
#vi /etc/security/limits.conf
* soft nofile 65536
* hard nofile 131072
* soft nproc 2048
* hard nproc 4096
[2]: max number of threads [1024] for user [es] is too low, increase to at least [4096]
#vi /etc/security/limits.d/90-nproc.conf
修改为* soft nproc 4096
[3]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
#vi /etc/sysctl.conf 添加
vm.max_map_count = 262144
[4]: system call filters failed to install; check the logs and fix your configuration or disable system call filters at your own risk
在elasticsearch.yml中配置bootstrap.system_call_filter为false，注意要在Memory下面:
bootstrap.memory_lock: false
bootstrap.system_call_filter: false
三.安装Kibana
#tar -zxf kibana-6.2.3-linux-x86_64.tar.gz
#mv /tmp/kibana-6.2.3-linux-x86_64 /usr/local/kibana
# cat kibana.yml | grep -v "^#"
server.port: 5601
server.host: "172.16.68.150"
elasticsearch.url: "http://172.16.68.150:9200"
kibana.index: ".kibana"
默认情况下,kibana进程处于前台运行，需要将其设置为后台运行
#nohup /usr/local/kibana/bin/kibana &
#jobs
[1]+ Running nohup /usr/local/kibana/bin/kibana &
# netstat -antlp | grep :5601
tcp 0 0 172.16.68.150:5601 0.0.0.0:* LISTEN 1977/node
检查Kibaba运行状态:
http://IP:5601/status
四.安装logstash
#tar -zxf /tmp/logstash-6.2.3.tar.gz
#mv /tmp/logstash-6.2.3 /usr/local/logstash
编辑配置文件
#vi /usr/local/logstash.conf
input {
beats {
port => 5044
}
}
output {
elasticsearch {
hosts => "172.16.68.150:9200"
}
}
#测试配置文件语法正确性
#bin/logstash -f logstash.conf --config.test_and_exit
#后台运行Logstash服务
#nohup /usr/local/logstash/bin/logstash -f /usr/local/logstash/logstash.conf --config.reload.automatic & #The "--config.reload.automatic" option enables automatic config reloading so that you don’t have to stop and restart Logstash every time you modify the configuration file.
五.安装filebeat插件
1.在需要收集日志的客户端服务器中执行
#rpm -ivh filebeat-6.2.3-x86_64.rpm
#vi /etc/filebeat/filebeat.yml
修改具体要获取的日志绝对路径以及输出的logstash ip/port
————————————————————————————————————
[root@webtest filebeat]# cat /etc/filebeat/filebeat.yml | grep -v "#" | grep -v "^$"
filebeat.prospectors:
- type: log
enabled: true
paths:
- /data/tomcatwww/logs/b2c/root.log
fields:
service: 68_224_b2c_root
scan.order: desc
- type: log
enabled: true
paths:
- /data/tomcatwww/logs/b2c/order.log
fields:
service: 68_224_b2c_order
scan.order: desc
- type: log
enabled: true
paths:
- /data/tomcatwww/logs/b2c/sale.log
fields:
service: 68_224_b2c_sale
scan.order: desc
- type: log
enabled: true
paths:
- /data/tomcatwww/logs/b2c/srv.log
fields:
service: 68_224_b2c_srv
scan.order: desc
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 3
output.logstash:
hosts: ["172.16.68.150:5044"]
————————————————————————————————————
#启动filebeat
#/etc/init.d/filebeat start
#ps -ef | grep filebeat
#ps -ef | awk '{print $1}'
六. 安装X-Pack(收费)
1.安装x-pack在es中
[es@elk bin]$ ./elasticsearch-plugin install file:///tmp/x-pack-6.2.3.zip -> Downloading file:///tmp/x-pack-6.2.3.zip [=================================================] 100%?? @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: plugin requires additional permissions @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ * java.io.FilePermission \\.\pipe\* read,write * java.lang.RuntimePermission accessClassInPackage.com.sun.activation.registries * java.lang.RuntimePermission getClassLoader * java.lang.RuntimePermission setContextClassLoader * java.lang.RuntimePermission setFactory * java.net.SocketPermission * connect,accept,resolve * java.security.SecurityPermission createPolicy.JavaPolicy * java.security.SecurityPermission getPolicy * java.security.SecurityPermission putProviderProperty.BC * java.security.SecurityPermission setPolicy * java.util.PropertyPermission * read,write See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html for descriptions of what these permissions allow and the associated risks. Continue with installation? [y/N]y @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ @ WARNING: plugin forks a native controller @ @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@ This plugin launches a native controller that is not subject to the Java security manager nor to system call filters. Continue with installation? [y/N]y Elasticsearch keystore is required by plugin [x-pack-security], creating... -> Installed x-pack with: x-pack-deprecation,x-pack-monitoring,x-pack-upgrade,x-pack-ml,x-pack-logstash,x-pack-watcher,x-pack-security,x-pack-graph,x-pack-core [es@elk bin]$ ll
2.设置默认用户的密码
[es@elk x-pack]$ ./setup-passwords interactive Initiating the setup of passwords for reserved users elastic,kibana,logstash_system. You will be prompted to enter passwords as the process progresses. Please confirm that you would like to continue [y/N]y Enter password for [elastic]: Reenter password for [elastic]: Enter password for [kibana]: Reenter password for [kibana]: Enter password for [logstash_system]: Reenter password for [logstash_system]: Changed password for user [kibana] Changed password for user [logstash_system] Changed password for user [elastic]
3.安装x-pack在kibana中
#./elasticsearch-plugin install file:///tmp/x-pack-6.2.3.zip [root@elk bin]# ./kibana-plugin install file:///tmp/x-pack-6.2.3.zip Attempting to transfer from file:///tmp/x-pack-6.2.3.zip Transferring 270035965 bytes.................... Transfer complete Retrieving metadata from plugin archive Extracting plugin archive Extraction complete Optimizing and caching browser bundles... Plugin installation complete [root@elk bin]# [root@elk bin]# [root@elk bin]# ./kibana-plugin list [email protected]
4.安装x-pack在logstash中
#./logstash-plugin install file:///tmp/x-pack-6.2.3.zip