Logstash installation and construction (1)

http://www.cnblogs.com/Orgliny/p/5579832.html

　Logstash is an open source data collection engine with a real-time pipeline. Data from different sources can be dynamically unified and attributed to different destinations. Also a management event and logging tool. You can use it to collect logs, analyze them, and store them for later use.

　　Logstash is usually used with Kibana and Elasticsearch, but there are many other uses that deserve our attention. The configuration and construction of Elasticsearch can be found in this blog . This article will describe the installation and simple configuration of logstash in detail.

1. Download Logstash from the official website

# wget https://download.elastic.co/logstash/logstash/logstash-2.3.2.tar.gz

2. Download the rpm version of logstash, decompress and use the official startup script

# wget https://download.elastic.co/logstash/logstash/packages/centos/logstash-2.3.2-1.noarch.rpm

3. Java 8 download address:

http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

4. Configure the java environment

# tar zxf jdk-8u91-linux-x64.tar.gz -C /usr/local/
# vi /etc/profile
    export JAVA_HOME=/usr/local/jdk1.8.0_91
    export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
    export PATH=$PATH:$JAVA_HOME/bin
# source /etc/profile

Enter java -version If you see the following information, the java environment configuration is successful

java version "1.8.0_91"
Java(TM) SE Runtime Environment (build 1.8.0_91-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode)

5. Unzip the rpm package

# mv logstash-2.3.2-1.noarch.rpm /tmp
# cd /tmp/
# rpm2cpio logstash-2.3.2-1.noarch.rpm | cpio -div

6. Unzip the tar package and configure the startup script

# tar zxf logstash-2.3.2.tar.gz -C /usr/local/      
# cd /usr/local/
# mv logstash-2.3.2/ logstash
# groupadd -r logstash　　//创建logstash组
# useradd -r -g logstash -d /usr/local/logstash -s /sbin/nologin -c "logstash" logstash　　//创建logstash用户

将rpm软件包中的脚本复制到系统指定位置
# cp /tmp/etc/init.d/logstash /etc/init.d/
# cp /tmp/etc/sysconfig/logstash /etc/sysconfig/
# cp /tmp/etc/logrotate.d/logstash /etc/logrotate.d/
# chmod 0644 /etc/logrotate.d/logstash

创建logstash的日志、HOME以及配置文件目录
# mkdir -p /etc/logstash/conf.d/　　//配置文件目录
# mkdir /var/log/logstash　　//日志目录
# mkdir /var/lib/logstash　　//HOME目录
# chown logstash /var/log/logstash
# chown logstash:logstash /var/lib/logstash
# chown -R logstash:logstash /usr/local/logstash/

配置启动脚本中的变量，将其修改为logstash的实际路径
# vi /etc/init.d/logstash
    program=/usr/local/logstash/bin/logstash

此时就可以将自己写好的logstash配置文件放到 /etc/logstash/conf.d/ 下，并设置开机启动。

7、使用简单的配置文件测试

# cat /etc/logstash/conf.d/simple.conf
input {
  stdin {}
}
output {
  stdout {
    codec => rubydebug }
}

使用命令运行logstash

# /usr/local/logstash/bin/logstash -f /etc/logstash/conf.d/simple.conf     // -f 指定配置文件，在启动之前还可以使用 -t 参数指定配置文件检查配置是否正确
Settings: Default pipeline workers: 4
Pipeline main started

输入hello world ，查看输出结果

#/usr/local/logstash/bin/logstash -f /etc/logstash/conf.d/simple.conf 
Settings: Default pipeline workers: 4
Pipeline main started
hello world !
{
       "message" => "hello world !",
      "@version" => "1",
    "@timestamp" => "2016-06-13T02:35:01.737Z",
          "host" => "localhost.localdomain"
}

可以看到，输入什么内容logstash按照某种格式输出，使用CTRL-C命令可以退出之前运行的Logstash。

8、配置logstash使用elasticsearch作为logstash后端

# cat /usr/local/logstash/conf.d/logstash-es-simple.conf
  input {
    stdin {}
  }
  output {
    elasticsearch {
      hosts => "127.0.0.1"}
    stdout {
      codec => rubydebug }
  }

执行命令

 执行命令：
    # /usr/local/logstash/bin/logstash agent -f conf.d/logstash-es-simple.conf
        Settings: Default pipeline workers: 4
        Pipeline main started
        hello logstash
        {
               "message" => "hello logstash",
              "@version" => "1",
            "@timestamp" => "2016-06-13T02:39:25.112Z",
                  "host" => "localhost.localdomain"
        }

使用curl命令发送请求来查看ES是否接收到了数据：

# curl 'http://127.0.0.1:9200/_search?pretty'
        {
          "took" : 21,
          "timed_out" : false,
          "_shards" : {
            "total" : 5,
            "successful" : 5,
            "failed" : 0
          },
          "hits" : {
            "total" : 1,    
            "max_score" : 1.0,
            "hits" : [ {
              "_index" : "logstash-2016.06.13",
              "_type" : "logs",
              "_id" : "AVRg9UHczZ2iuimLmajG",
              "_score" : 1.0,
              "_source" : {
                "message" : "hello logstash",
                "@version" : "1",
                "@timestamp" : "2016-06-13T02:39:25.112Z",
                "host" : "localhost.localdomain"
              }
            } ]
          }
        }

此时已经成功利用elasticsearch和logstash收集数据。

Logstash installation and construction (1)

Guess you like