ELK的文档搭建

其他 2019-04-04 13:34:45 阅读次数: 0

一、安装elasticsearch

官网：https://www.elastic.co/guide/index.html

https://www.elastic.co/guide/en/elasticsearch/reference/2.3/index.html

1）环境准备

[root@k8s6 ~]# java -version
openjdk version "1.8.0_201"
OpenJDK Runtime Environment (build 1.8.0_201-b09)
OpenJDK 64-Bit Server VM (build 25.201-b09, mixed mode)

1) 
[root@k8s6 ~]# ls elktools/
elasticsearch-2.3.5.rpm  kibana-4.5.4-1.x86_64.rpm  logstash-2.3.4-1.noarch.rpm
View Code

2）rpm包安装

[root@k8s6 ~]# rpm -ivh elktools/elasticsearch-2.3.5.rpm

3）修改配置文件

备份配置文件
cp /etc/elasticsearch/elasticsearch.yml /etc/elasticsearch/elasticsearch.yml.bak
修改配置文件
[root@k8s6 elasticsearch]# diff /etc/elasticsearch/elasticsearch.yml /etc/elasticsearch/elasticsearch.yml.bak
17c17
< cluster.name: myes
---
> # cluster.name: my-application
23c23
< node.name: master_node
---
> # node.name: node-1
54c54
< network.host: 192.168.10.22
---
> # network.host: 192.168.0.1
58c58
< http.port: 9200
---
> # http.port: 9200
View Code

补充：

bootstrap.mlockall: false
bootstrap.system_call_filter: false

4）启动服务

[root@k8s6 elasticsearch]# systemctl start elasticsearch    启动服务
[root@k8s6 elasticsearch]# netstat -lntup|grep java
tcp6       0      0 192.168.10.22:9200      :::*                    LISTEN      25874/java          
tcp6       0      0 192.168.10.22:9300      :::*                    LISTEN      25874/java
View Code

5）对elasticsearch的api验证

[root@k8s6 elasticsearch]# curl http://192.168.10.22:9200
{
  "name" : "master_node",
  "cluster_name" : "myes",
  "version" : {
    "number" : "2.3.5",
    "build_hash" : "90f439ff60a3c0f497f91663701e64ccd01edbb4",
    "build_timestamp" : "2016-07-27T10:36:52Z",
    "build_snapshot" : false,
    "lucene_version" : "5.5.0"
  },
  "tagline" : "You Know, for Search"
}
View Code

二、安装elasticsearch的插件安装

1）查看执行的命令和插件位置

[root@k8s6 ~]# ls /usr/share/elasticsearch/
bin  lib  LICENSE.txt  modules  NOTICE.txt  plugins  README.textile

2）安装head插件

[root@k8s6 ~]# /usr/share/elasticsearch/bin/plugin install mobz/elasticsearch-head        从github下载
[root@k8s6 ~]# ls /usr/share/elasticsearch/plugins/head/
Dockerfile                          Gruntfile.js       LICENCE                       proxy           src
Dockerfile-alpine                   grunt_fileSets.js  package.json                  README.textile  test
elasticsearch-head.sublime-project  index.html         plugin-descriptor.properties  _site

2.1）访问测试

http://192.168.10.22:9200/_plugin/head/

3.1）安装 kopf 插件

[root@k8s6 ~]# /usr/share/elasticsearch/bin/plugin install lmenezes/elasticsearch-kopf

3.2）访问测试

http://192.168.10.22:9200/_plugin/kopf/

 

 4）模拟提交post请求

刷新连接

 

 二、es的集群测试

1）另一台机器同样安装好es,修改配置文件（主播模式）测试vm会存在问题

[root@node01 elasticsearch]# grep '^[a-Z]' elasticsearch.yml
cluster.name: myes
node.name: node01
path.data: /data/es-data
path.logs: /var/log/elasticsearch
bootstrap.mlockall: false
bootstrap.system_call_filter: false
network.host: 192.168.10.23
http.port: 9200

创建目录，并授权

[root@node01 elasticsearch]# mkdir -p /data/es-data
[root@node01 elasticsearch]# chown elasticsearch:elasticsearch /data/es-data/

2）单播模式 

[root@node01 elasticsearch]# grep '^[a-Z]' elasticsearch.yml
cluster.name: myes
node.name: node01
path.data: /data/es-data
path.logs: /var/log/elasticsearch
bootstrap.mlockall: true
network.host: 192.168.10.23
http.port: 9200
discovery.zen.ping.unicast.hosts: ["192.168.10.22", "192.168.10.23"]

3）启动服务

强调，一点要关闭防火墙。否则无法弄成集群

[root@node01 elasticsearch]# /etc/init.d/elasticsearch start

 4）查询集群的状态

[root@node01 ~]# curl -XGET 'http://192.168.10.22:9200/_cluster/health?pretty=true'
{
  "cluster_name" : "myes",
  "status" : "green",
  "timed_out" : false,
  "number_of_nodes" : 2,    # 2个节点
  "number_of_data_nodes" : 2,
  "active_primary_shards" : 5,
  "active_shards" : 10,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 0,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 100.0
}
View Code

 三、在主节点安装 logstash

1）安装logstash

[root@k8s6 elktools]# rpm -ivh logstash-2.3.4-1.noarch.rpm 
准备中...                          ################################# [100%]
正在升级/安装...
   1:logstash-1:2.3.4-1               ################################# [100%]
[root@k8s6 elktools]# rpm -qa|grep logstash
logstash-2.3.4-1.noarch
[root@k8s6 elktools]# rpm -ql logstash    # 查看安装生成了哪些文件
View Code

 2）模拟日志标准输出：/opt/logstash/bin/logstash -e 'input { stdin{} } output { stdout{} }'

[root@k8s6 elktools]# /opt/logstash/bin/logstash -e 'input { stdin{} } output { stdout{} }'
Settings: Default pipeline workers: 4
Pipeline main started
hello
2019-03-15T04:42:43.106Z k8s6 hello
hello world
2019-03-15T04:42:51.121Z k8s6 hello world
View Code

json格式输出： /opt/logstash/bin/logstash -e 'input { stdin{} } output { stdout{ codec => rubydebug } }'

[root@k8s6 elktools]# /opt/logstash/bin/logstash -e 'input { stdin{} } output { stdout{ codec => rubydebug } }'
Settings: Default pipeline workers: 4
Pipeline main started
hello world
{
       "message" => "hello world",
      "@version" => "1",
    "@timestamp" => "2019-03-15T04:47:12.509Z",
          "host" => "k8s6"
}
View Code

3）命令行模拟输出到elasticsearch： /opt/logstash/bin/logstash -e 'input { stdin{} } output { elasticsearch{ hosts => ["192.168.10.22:9200"] index => "logstash-%{+YYYY.MM.dd}" } }'

[root@k8s6 elktools]# /opt/logstash/bin/logstash -e 'input { stdin{} } output { elasticsearch{ hosts => ["192.168.10.22:9200"] index => "logstash-%{+YYYY.MM.dd}" } }'
Settings: Default pipeline workers: 4
Pipeline main started
hhhee
View Code

 查询到输出的值

4）同时进行屏幕输出并写入es

[root@k8s6 elktools]# /opt/logstash/bin/logstash -e 'input { stdin{} } output { stdout{ codec => rubydebug }  elasticsearch{ hosts => ["192.168.10.22:9200"] index => "logstash-%{+YYYY.MM.dd}" } }'
Settings: Default pipeline workers: 4
Pipeline main started
天天向上
{
       "message" => "天天向上",
      "@version" => "1",
    "@timestamp" => "2019-03-15T05:02:55.388Z",
          "host" => "k8s6"
}
View Code

 四、使用配置文件启动logstash

检测语法

[root@node01 ~]# /opt/logstash/bin/logstash -t -f /etc/logstash/conf.d/nginx.conf
Configuration OK

1）进入到配置文件目录。cd /etc/logstash/conf.d/   ，该目录由/etc/init.d/logstash 这里定义的

[root@k8s6 elktools]# cd /etc/logstash/conf.d/
[root@k8s6 conf.d]# ls
[root@k8s6 conf.d]#
View Code

编辑 demo.conf文件，既输出在屏幕，也存入es

YYYY.MM.dd  每天

[root@k8s6 conf.d]# cat demo.conf
input {
    stdin{}
}

filter{
}

output{
    elasticsearch {
        hosts => ["192.168.10.22:9200"]
        index => "logstash-%{+YYYY.MM.dd}"
    }
    
    stdout{ codec => rubydebug }
}
View Code

启动服务：[root@k8s6 conf.d]# /opt/logstash/bin/logstash -f /etc/logstash/conf.d/demo.conf   

 2)使用file插件，指定要记录日志的目录

[root@k8s6 conf.d]# cat file.conf 
input {
    file{
        path => ["/var/log/messages", "/var/log/secure"]
        type => "system-log"
        start_position => "beginning"
    }
}

filter{
}

output{
    elasticsearch {
        hosts => ["192.168.10.22:9200"]
        index => "system-log-%{+YYYY.MM}"
    }    
}
file.conf

启动服务： /opt/logstash/bin/logstash -f /etc/logstash/conf.d/file.conf

 五、安装kibana

1）在主节点安装kibana

[root@k8s6 elktools]# rpm -ivh kibana-4.5.4-1.x86_64.rpm

查看安装到了哪些位置

[root@k8s6 elktools]# rpm -ql kibana
View Code

 2）修改配置文件

[root@k8s6 config]# pwd
/opt/kibana/config
[root@k8s6 config]# grep '^[a-Z]' kibana.yml 
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.url: "http://192.168.10.22:9200"
kibana.index: ".kibana"
/opt/kibana/config/kibana.yml

启动服务：[root@k8s6 config]# /etc/init.d/kibana start

[root@k8s6 config]# netstat -lntup|grep 5601
tcp        0      0 0.0.0.0:5601            0.0.0.0:*               LISTEN      31271/node

网页访问：192.168.10.22:5601

 2）配置索引，配置哪个，显示哪个的日志文件

匹配到了创建，即可

 

猜你喜欢

转载自www.cnblogs.com/linu/p/10533396.html
今日推荐
《美国对全球网络空间安全与发展的威胁和破坏》报告发布
火速冲上 GitHub 热榜 —— 开源编程语言、框架哪有这么可爱？
北京人形机器人创新中心发布全球首个纯电驱拟人奔跑的全尺寸人形机器人“天工”
LFOSSA 源来如此公开课 | 掌握云原生未来：CNCF 认证全面攻略与备考秘籍
周排行
让自己的头脑极度开放
CentOS 6.5(x64) 和Redhat6.5操作系误删libc
高可用注册中心
【日记】12.28/【题解】AtCoder AGC041
XML（5）_XML 约束_DTD
Java集合Map（四）
树梅派安装桌面环境教程
pipenv 的 使用和安装
小程序白屏问题和内存研究
C语言简单选择排序
每日归档
更多
2024-05-02(0)
2024-05-01(4)
2024-04-30(1)
2024-04-29(40)
2024-04-28(0)
2024-04-27(56)
2024-04-26(39)
2024-04-25(22)
2024-04-24(36)
2024-04-23(26)

代码天地 © 2018 codetd.com

境外服务器   最新电视剧2022