Use the ELK suite to build a log analysis and monitoring platform under CentOS . See the attachment.
1 Overview
ELK suite ( ELK stack ) refers to the three-piece set of ElasticSearch , Logstash and Kibana . These three software can form a set of log analysis and monitoring tools.
Since the respective version numbers of the three software are too many, it is recommended to use the combination recommended by the ElasticSearch official website: http://www.elasticsearch.org/overview/elkdownloads/
2 Environment preparation
2.1 Software Requirements
The specific version requirements are as follows:
- Operating system version: CentOS 6.7 ;
- JDK version: 1.7.0 ;
- Logstash version: 1.4.2 ;
- ElasticSearch version: 1.4.2 ;
- Kibana version: 3.1.2 ;
2.2 Firewall Configuration
First install jdk
Install JDK with yum
1. Check which jdk versions are in the yum library ( only openjdk has been found for the time being)
[root@localhost ~]# yum search java|grep jdk
ldapjdk-javadoc.x86_64 : Javadoc for ldapjdk
java-1.6.0-openjdk.x86_64 : OpenJDK Runtime Environment
java-1.6.0-openjdk-demo.x86_64 : OpenJDK Demos
java-1.6.0-openjdk-devel.x86_64 : OpenJDK Development Environment
java-1.6.0-openjdk-javadoc.x86_64 : OpenJDK API Documentation
java-1.6.0-openjdk-src.x86_64 : OpenJDK Source Bundle
java-1.7.0-openjdk.x86_64 : OpenJDK Runtime Environment
java-1.7.0-openjdk-demo.x86_64 : OpenJDK Demos
java-1.7.0-openjdk-devel.x86_64 : OpenJDK Development Environment
java-1.7.0-openjdk-javadoc.noarch : OpenJDK API Documentation
java-1.7.0-openjdk-src.x86_64 : OpenJDK Source Bundle
java-1.8.0-openjdk.x86_64 : OpenJDK Runtime Environment
java-1.8.0-openjdk-demo.x86_64 : OpenJDK Demos
java-1.8.0-openjdk-devel.x86_64 : OpenJDK Development Environment
java-1.8.0-openjdk-headless.x86_64 : OpenJDK Runtime Environment
java-1.8.0-openjdk-javadoc.noarch : OpenJDK API Documentation
java-1.8.0-openjdk-src.x86_64 : OpenJDK Source Bundle
ldapjdk.x86_64 : The Mozilla LDAP Java SDK
2. Select the version and install it
// Select version 1.7 to install
[root@localhost ~]# yum install java-1.8.0-openjdk
// After installation, the default installation directory is : /usr/lib/jvm/java-1.8.0-openjdk -1.8.0.75.x86_64
3. Set environment variables
[root@localhost ~]# vi /etc/profile
Add the following to the profile file
#set java environment
JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75.x86_64
JRE_HOME=$JAVA_HOME/jre
CLASS_PATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib
PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin
export JAVA_HOME JRE_HOME CLASS_PATH PATH
Make changes take effect
[root@localhost java]# source /etc/profile
Check jdk to java -version
In order to use HTTP services normally, you need to turn off the firewall:
- # service iptables stop
Or you can not close the firewall, but open the relevant ports in iptables :
- # vim /etc/sysconfig/iptables
- -A INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT
- -A INPUT -m state --state NEW -m tcp -p tcp --dport 9200 -j ACCEPT
- -A INPUT -m state --state NEW -m tcp -p tcp --dport 9292 -j ACCEPT
- # service iptables restart
· #Unzip installation
· useradd elk
su – each
·
· tar -xvf elasticsearch-2.3.4.tar.gz
·
· cd elasticsearch-2.3.4
· #Install the Head plugin
· ./bin/plugin install mobz/elasticsearch-head
·
· Ls plugins /
· # ls can see the head file to indicate ok .
· [elk@hch_test_dbm1_121_62 elasticsearch-2.3.4]$ ll plugins/
· Total dosage 4
· drwxrwxr-x. 5 each each 4096 8月 2 17:26 head
·
· [Elk @ hch_test_dbm1_121_62 elasticsearch-2.3.4] $ vim config / elasticsearch.yml
·
· cluster.name: es_cluster
· node.name: node0
· path.data: /home/elk/data
· path.logs: /home/elk/logs
· # 当前的host ip地址
· network.host: 192.168.121.62
· network.port: 9200
·
启动es：
./bin/elasticsearch
ES 启动错误 解决方案 5.51版本
http://blog.csdn.net/u012371450/article/details/51776505
打开url地址http://192.168.0.65:9200/
访问
http://192.168.0.65:9200/_plugin/head/
安装logstash
logstash其实它就是一个 收集器 而已，我们需要为它指定Input和Output（当然Input和Output可以为多个）。由于我们需要把Java代码中Log4j的日志输出到ElasticSearch中，因此这里的Input就是Log4j，而Output就是ElasticSearch。
结构图如E:\u\elk\pic\02.png所示：
安装配置：
# 解压缩安装
tar -xvf logstash-2.3.4.tar.gz
cd logstash-2.3.4
# 将配置文件放置在config文件夹下面
mkdir config
vim config/log4j_to_es.conf
# For detail structure of this file
# Set: https://www.elastic.co/guide/en/logstash/current/configuration-file-structure.html
input {
# For detail config for log4j as input,
# See: https://www.elastic.co/guide/en/logstash/current/plugins-inputs-log4j.html
log4j {
mode => "server"
host => "192.168.121.62" #本机IP
port => 4567
}
}
filter {
#Only matched data are send to output.
}
output {
# For detail config for elasticsearch as output,
# See: https://www.elastic.co/guide/en/logstash/current/plugins-outputs-elasticsearch.html
elasticsearch {
action => "index" #The operation on ES
hosts => "192.168.121.62:9200" #ElasticSearch host, can be array.
index => "applog" #The index to write data to.
}
}
启动logstash，2个参数一个是agent一个是配置文件：
[elk@hch_test_dbm1_121_62 logstash-2.3.4]$ ./bin/logstash -f config/log4j_to_es.conf
Settings: Default pipeline workers: 32
log4j:WARN No appenders could be found for logger (org.apache.http.client.protocol.RequestAuthCache).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Pipeline main started
接下来，可以使用logstash来收集日志并保存到es中了，可以使用一段java代码来实现它。记得关闭本机防火墙