Logstash 是一个具有 real-time pipelining功能的开源数据收集引擎。
Logstash可以动态地统一来自不同源的数据，并将数据规范化到选择的目的地。为不同的高级下游分析和可视化用例清理并民主化所有数据。
虽然Logstash最初推动了日志收集方面的创新，但是它的能力远远超出了这个。任何类型的事件都可以通过广泛的输入、过滤器和输出插件数组来丰富和转换，许多本地编解码器进一步简化了摄取过程。Logstash通过利用更大容量和多样的数据来加速您的洞察力。
- The ingestion workhorse for Elasticsearch and more
- Pluggable pipeline architecture
- Community-extensible and developer-friendly plugin ecosystem
Logstash管道有两个必需的元素，输入和输出，以及一个可选的元素filter。输入插件消耗源数据，筛选器插件根据您的指定修改数据，输出插件将数据写入目的地。
1. 安装
1.1 yum
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
vim /etc/yum.repos.d/logstash.repo
[logstash-6.x] name=Elastic repository for 6.x packages baseurl=https://artifacts.elastic.co/packages/6.x/yum gpgcheck=1 gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch enabled=1 autorefresh=1 type=rpm-md
yum install logstash
1.2 rpm
wget https://artifacts.elastic.co/downloads/logstash/logstash-6.5.4.rpm
rpm -iv logstash-6.5.4.rpm
2. 配置
cd /etc/logstash
vim logstash.yml
path.data: /var/lib/logstash
pipeline.workers: 20 # 并行执行管道的过滤器和输出阶段的worker的数量。如果发现事件正在备份，或者CPU未饱和，请考虑增加这个数目以更好地利用机器处理能力。默认值为主机CPU核数。
pipeline.batch.size: 4000 # 单个工作线程在尝试执行其筛选器和输出之前将从输入中收集的事件的最大数量。较大的批量大小通常更有效，但是要以增加的内存开销为代价。
log.level: info
path.logs: /var/log/logstash
vim pipelines.yml
- pipeline.id: main
path.config: "/etc/logstash/conf.d/*.conf"
3.
测试
/usr/share/logstash/bin/logstash -e 'input { stdin { } } output { stdout { } }'
......
hello world
{
"@version" => "1",
"host" => "dbwtest03bc.daodao.com",
"@timestamp" => 2018-12-24T06:35:19.938Z,
"message" => "hello world"
}
===================================================================
实验：
写一个脚本，每两秒生成一条日志。使用filebeat跟踪logstash，发送到 logstash。
------------------------------------------
生成日志脚本：
import time
import random
import string
import logging
import socket
logfile = '/root/test_elk/test_elk.log'
logger = logging.getLogger('[Test-ELK]')
logger.setLevel(logging.INFO)
handler = logging.FileHandler(logfile, mode='a')
handler.setLevel(logging.INFO)
formatter = logging.Formatter('%(asctime)s %(name)-12s %(levelname)-6s %(message)s')
handler.setFormatter(formatter)
logger.addHandler(handler)
for i in xrange(1000):
# TIMESTAMP=time.strftime('%Y-%m-%d %H:%M:%S')
id = i
myname = socket.getfqdn(socket.gethostname( ))
myaddr = socket.gethostbyname(myname)
c1 = random.randint(0,100)
c2 = random.choice(("a","b","c","d"))
c3 = random.choice(("f","t"))
c4 = ''.join(random.sample(string.ascii_letters + string.digits, 6))
logger.info('id=%s, myname=%s, myaddr=%s, c1=%s, c2=%s, c3=%s, c4=%s'% (id, myname, myaddr, c1, c2, c3, c4))
time.sleep(5)
------------------------------------------
filebeat 主要配置（输入，输出）：
filebeat.inputs:
- type: log
enabled: true
paths:
- /root/test_elk/test_elk*.log
field:
log_topic: test_elk
output.logstash:
hosts: ["192.168.3.51:5044"]
------------------------------------------
使用logstash，从beat输入，输出到stdout
/usr/share/logstash/bin/logstash -e 'input { beat { port => 5044 } } output { stdout { } }'
{
"@version" => "1",
"offset" => 18200,
"message" => "2018-12-24 01:58:19,149 [Test-ELK] INFO id=2, myname=dbwtest03bc.daodao.com, myaddr=192.168.4.17, c1=14, c2=b, c3=f, c4=lIJKq8",
"input" => {
"type" => "log"
},
"tags" => [
[0] "beats_input_codec_plain_applied"
],
"@timestamp" => 2018-12-24T06:58:22.494Z,
"beat" => {
"hostname" => "dbwtest03bc.daodao.com",
"version" => "6.5.4",
"name" => "dbwtest03bc.daodao.com"
},
"host" => {
"architecture" => "x86_64",
"os" => {
"version" => "7 (Core)",
"codename" => "Core",
"platform" => "centos",
"family" => "redhat"
},
"containerized" => true,
"id" => "6787d9310dd84654ab8871f64df6f6d7",
"name" => "dbwtest03bc.daodao.com"
},
"source" => "/root/test_elk/test_elk.log",
"prospector" => {
"type" => "log"
}
}