Big Data Elasticsearch of lightweight log collection tool FileBeat installation, configuration and use

---

beats

beats is an agent to send different types of data to elasticsearch. beats can directly transmit data to elasticsearch, may be transmitted via logstash elasticsearch data.

---

Filebeat

Filebeat File acquisition is dedicated beats, for collecting logs

---

1. Download & unzip

$ wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.3.2-linux-x86_64.tar.gz
$ tar -zxvf filebeat-7.3.2-linux-x86_64.tar.gz

---

2. 配置 filebeat.yml

     #定义日志文件路径
     filebeat.inputs:
     - type: log
        enabled: true
        paths:
             - /var/log/*.log
    #发送到Elasticseach
     output.elasticsearch:
        hosts: ["192.168.0.22:9200"]
    #如果要用kibana仪表盘
     setup.kibana:
        host: "localhost:5601"
    #如果有安全策略，则需要指定访问凭据
    output.elasticsearch:
        hosts: ["elasticsearchlocalhost:9200"]
        username: "elasticsearch"
        password: "pwd"
     setup.kibana:
        host: "kibanalocalhost:5601"
        username: "kibana"
        password: "pwd"
    #如果使用logstash
     output.logstash:
        hosts: ["127.0.0.1:5044"]
     #索引模板加载。索引模板用于定义设置和映射，以确定如何分析字段。默认模板文件 fields.yml
     #加载不同的模板
     setup.template.name: "template"
     setup.template.fields: "path/to/fields.yml"
    #覆盖一个已存在的模板
     setup.template.overwrite: true
    #禁用自动加载模板
     setup.template.enabled: false
    #修改索引名称
     # 默认情况下，Filebeat写事件到名为filebeat-6.3.2-yyyy.MM.dd的索引，其中yyyy.MM.dd是事件被索引的日期。为了用一个不同的名字，你可以在Elasticsearch输出中设置index选项。例如：
     output.elasticsearch.index: "customname-%{[beat.version]}-%{+yyyy.MM.dd}"
     setup.template.name: "customname"
     setup.template.pattern: "customname-*"
     setup.dashboards.index: "customname-*"    

---

3. Start Filebeat

$ ./filebeat -e -c filebeat.yml -d "publish"