Install the Logstash dependency package JDK
The operation of Logstash depends on the Java operating environment. The latest version of Java is recommended for Logstash 1.5 or later versions not lower than java 7. Since we are just running Java programs, not developing, just download the JRE. First, download the new version of jre from Oracle, the download address: http://www.oracle.com/technetwork/java/javase/downloads/jre8-downloads-2133155.html
$ mkdir /usr/local/java
$ tar -zxf jdk-8u45-linux-x64.tar.gz -C /usr/local/java/
Set the JDK environment variables as follows:
$ vim ~/.bash_profile
export JAVA_HOME=/usr/local/java/jdk1.8.0_161
export PATH=$PATH:$JAVA_HOME/bin
exportCLASSPATH=.:$JAVA_HOME/lib/tools.jar:$JAVA_HOME/lib/dt.jar:$CLASSPATH
$ java -version (note -version not --version)
java version "1.8.0_161"
Java(TM) SE Runtime Environment (build 1.8.0_161-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)
Install Logstash
$ https://download.elastic.co/logstash/logstash/logstash-6.2.3.tar.gz
$ tar –zxf logstash-6.2.3.tar.gz -C /usr/local/
After the installation is complete, run the following command:
$ /usr/local/logstash-6.2.3/bin/logstash -e 'input { stdin { } } output { stdout {} }'
output:
Sending Logstash's logs to /usr/local/logstash-6.2.3/logs which is now configured via log4j2.properties
[2018-04-09T01:54:36,236][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/usr/local/logstash-6.2.3/modules/netflow/configuration"}
[2018-04-09T01:54:36,267][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/usr/local/logstash-6.2.3/modules/fb_apache/configuration"}
[2018-04-09T01:54:36,953][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-04-09T01:54:37,799][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.2.3"}
[2018-04-09T01:54:38,385][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2018-04-09T01:54:40,664][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>"main", "pipeline.workers"=>4, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2018-04-09T01:54:40,871][INFO ][logstash.pipeline ] Pipeline started succesfully {:pipeline_id=>"main", :thread=>"#<Thread:0x72564250 run>"}
The stdin plugin is now waiting for input:
[2018-04-09T01:54:40,981][INFO ][logstash.agent ] Pipelines running {:count=>1, :pipelines=>["main"]}
hello world
我们可以看到，我们输入什么内容logstash按照某种格式输出，其中-e参数参数允许Logstash直接通过命令行接受设置。这点可以帮助我们反复的测试配置是否正确而不用写配置文件。使用CTRL-C命令可以退出之前运行的Logstash。
使用-e参数在命令行中指定配置是很常用的方式，不过如果需要配置更多设置则需要很长的内容。这种情况，我们首先创建一个简单的配置文件，并且指定logstash使用这个配置文件。
例如：在logstash安装目录下创建一个“基本配置”测试文件logstash-simple.conf，文件内容如下：
$ cat logstash-simple.conf
input { stdin { } }
output {
stdout { codec=> rubydebug }
}
Logstash使用input和output定义收集日志时的输入和输出的相关配置，本例中input定义了一个叫"stdin"的input，output定义一个叫"stdout"的output。无论我们输入什么字符，Logstash都会按照某种格式来返回我们输入的字符，其中output被定义为"stdout"并使用了codec参数来指定logstash输出格式。
使用logstash的-f参数来读取配置文件，执行如下开始进行测试：
$ echo "`date` hello World"
Mon Apr 9 02:08:45 UTC 2018 hello World
$ /usr/local/logstash-6.2.3/bin/logstash -f logstash-simple.conf
Logstash startup completed
Mon Apr 9 02:08:45 UTC 2018 hello Worl #该行是执行echo “`date`hello World” 后输出的结果，直接粘贴到该位置
{
"@timestamp" => 2018-04-09T02:17:15.064Z,
"message" => "Mon Apr 9 02:08:45 UTC 2018 hello Worl",
"@version" => "1",
"host" => "5ef8026aa3bf"
}
安装Elasticsearch
$ tar -zxf elasticsearch-6.2.3.tar.gz -C /usr/local/
启动Elasticsearch
$ /usr/local/elasticsearch-6.2.3/bin/elasticsearch
OR后台执行
$ nohup /usr/local/elasticsearch-6.2.3/bin/elasticsearch &
这时有可能会直接被Killed掉，因为内存溢出（OOM），elastisearch占用的内存非常大，所以在内存比较小的服务器上运行要先修改jvm的内存大小
$ vim config/jvm.options
将22和23行的栈堆大小改为512M
-Xms512M
-Xmx512M
注：如果在运行过程中还出现Killed，继续将以上的值调小。
[2018-04-09T03:06:34,552][WARN ][o.e.b.ElasticsearchUncaughtExceptionHandler] [] uncaught exception in thread [main]
org.elasticsearch.bootstrap.StartupException: java.lang.RuntimeException: can not run elasticsearch as root
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:125) ~[elasticsearch-6.2.3.jar:6.2.3]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:112) ~[elasticsearch-6.2.3.jar:6.2.3]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86) ~[elasticsearch-6.2.3.jar:6.2.3]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:124) ~[elasticsearch-cli-6.2.3.jar:6.2.3]
at org.elasticsearch.cli.Command.main(Command.java:90) ~[elasticsearch-cli-6.2.3.jar:6.2.3]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) ~[elasticsearch-6.2.3.jar:6.2.3]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) ~[elasticsearch-6.2.3.jar:6.2.3]
Caused by: java.lang.RuntimeException: can not run elasticsearch as root
at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:105) ~[elasticsearch-6.2.3.jar:6.2.3]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:172) ~[elasticsearch-6.2.3.jar:6.2.3]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:323) ~[elasticsearch-6.2.3.jar:6.2.3]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:121) ~[elasticsearch-6.2.3.jar:6.2.3]
... 6 more
因为安全问题elasticsearch 不让用root用户直接运行，所以要创建新用户
$ groupadd tzhennan
$ useradd tzhennan -g tzhennan
Exception in thread "main" java.nio.file.AccessDeniedException: /usr/local/elasticsearch-6.2.3/config/jvm.options
at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
at java.nio.file.Files.newByteChannel(Files.java:361)
at java.nio.file.Files.newByteChannel(Files.java:407)
at java.nio.file.spi.FileSystemProvider.newInputStream(FileSystemProvider.java:384)
at java.nio.file.Files.newInputStream(Files.java:152)
at org.elasticsearch.tools.launchers.JvmOptionsParser.main(JvmOptionsParser.java:58)
$ chown -R elasticsearch:elasticsea elasticsearch-6.2.3/
查看elasticsearch的9200端口是否已监听
$ netstat -anp | grep :9200
tcp 0 0 127.0.0.1:9200 0.0.0.0:* LISTEN 1483/java
接下来我们在logstash安装目录下创建一个用于测试logstash使用elasticsearch作为logstash的后端的测试文件logstash-es-simple.conf，该文件中定义了stdout和elasticsearch作为output，这样的“多重输出”即保证输出结果显示到屏幕上，同时也输出到elastisearch中
input { stdin { } }
output {
elasticsearch {
hosts => "localhost:9200"
user => "tzhennan"
password => "xxx"
}
stdout {
codec=> rubydebug
}
}
执行命令
$ /usr/local/logstash-6.2.3/bin/logstash -f logstash-es-simple.conf
[2018-04-09T07:33:50,581][INFO ][logstash.outputs.elasticsearch] Installing elasticsearch template to _template/logstash
[2018-04-09T07:33:51,886][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2018-04-09T07:33:52,078][INFO ][logstash.pipeline ] Pipeline started succesfully {:pipeline_id=>"main", :thread=>"#<Thread:0x17ddf470 run>"}
The stdin plugin is now waiting for input:
[2018-04-09T07:33:52,202][INFO ][logstash.agent ] Pipelines running {:count=>1, :pipelines=>["main"]}
hello logstash
{
"@timestamp" => 2018-04-09T07:35:03.141Z,
"host" => "5ef8026aa3bf",
"@version" => "1",
"message" => "hello logstash"
}
可以使用curl命令发送请求来查看ES是否接收到了数据：
$ curl 'http://localhost:9200/_search?pretty'
输出：
{
"took" : 154,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 1,
"max_score" : 1.0,
"hits" : [
{
"_index" : "logstash-2018.04.09",
"_type" : "doc",
"_id" : "WulUqWIB-5JaC2wTa7rt",
"_score" : 1.0,
"_source" : {
"@timestamp" : "2018-04-09T07:35:03.141Z",
"host" : "5ef8026aa3bf",
"@version" : "1",
"message" : "hello logstash"
}
}
]
}
}
至此，已经成功利用Elasticsearch和Logstash来收集日志数据了
问题：
可以访问127.0.0.1:9200，但不能访问公网IP:9200 ，需要在在elasticsearch.yml文件中增加以下配置：
network.bind_host: 0.0.0.0
安装elasticsearch插件
$ cd /usr/local/elasticsearch-6.2.3/
安装Head插件
$ ./bin/plugin install mobz/elasticsearch-head
安装安装elasticsearch-kopf插件
$ ./plugin -install lmenezes/elasticsearch-kopf
安装完成后在plugins目录下可以看到
$ ls plugin
可以在浏览器访问http://ip:9200/_plugin/kopf浏览保存在Elasticsearch中的数据
安装Kibana
$ tar -zxf kibana-6.2.3-linux-x86_64.tar.gz -C /usr/local/
启动kibana
$ /usr/local/kibana-6.2.3-linux-x86_64/bin/kibana
1>使用http://ip:5601访问Kibana，登录后，首先，配置一个索引，默认，Kibana的数据被指向Elasticsearch，使用默认的logstash-*的索引名称，并且是基于时间的，点击“Create”即可
2>点击“Discover”，可以搜索和浏览Elasticsearch中的数据，默认搜索的是最近15分钟的数据。可以自定义选择时间
问题：
可以访问127.0.0.1:5601，但不能访问公网IP:9200 ，需要在在kibaba.yml文件中增加以下配置：
server.host: "localhost"
更改为
server.host: "0.0.0.0"
配置logstash作为Indexer
将logstash配置为索引器，并将logstash的日志数据存储到Elasticsearch
input {
file {
type =>"syslog"
path => ["/var/log/messages", "/var/log/syslog" ]
}
syslog {
type => "syslog"
port => "5544"
}
}
output {
stdout { codec=> rubydebug }
elasticsearch {
hosts => "localhost:9200"
user => "tzhennan"
password => "xxx"
}
}
启动
$ /usr/local/logstash-6.2.3/bin/logstash -flogstash-indexer.conf
使用echo命令模拟写入日志
$ echo "`date` 本地系统测试" >>/var/log/messages
...
参考文章：
http://blog.51cto.com/baidu/1676798
https://my.oschina.net/itblog/blog/547250