Beats data collection
Beats is a lightweight data acquisition product from elastic company, which includes several sub-products:
packetbeat (for monitoring network traffic),
filebeat (for monitoring log data, can replace logstash-input-file),
topbeat (used to collect process information, load, memory, disk and other data),
winlogbeat (for collecting windows event logs)
In addition, the community also provides tools such as dockerbeat. Since they are all written based on libbeat, the configuration is basically the same, but the input is different.
This article will be introduced in sequence as follows:
Background knowledge:
Download, deploy, use, and result sample
filebeat download, deploy, use, sample
topbeat sample
winlogbeat sample using Powershell
About Powershell
If you want to use it under linux, you can skip this section.
The Beats in elastic basically use Powershell scripts in the windows environment, so users must have a certain understanding of Powershell. Powershell can be understood as an advanced encapsulation of the command line by windows, adding a shell to support more advanced usage. Since Windows 7, the system has built-in Powershell tools. Therefore, if your system is an older version of xp, you need to install Powershell yourself.
Start Powershell
Under Windows, there are two ways to open Powershell (to be opened as an administrator).
Open by icon Open
search under windows, enter powershell, right click and run as administrator.
Start through the command line
In the system path C:\Windows\System32, start cmd.exe as an administrator (right-click and select run as administrator).
Enter the command Powershell to enter the Powershell command window.
C:\Windows\system32>Powershell
Windows PowerShell
Copyright (C) 2009 Microsoft Corporation. all rights reserved.
PS C:\Windows\system32>
Enable script restriction
By default, the system will prohibit running scripts and return the following error message:
PS E:\packetbeat> .\install-service-packetbeat.ps1 The file E:\packetbeat\install-service-packetbeat.ps1 cannot be loaded because script
execution is prohibited on this system .
See "get-help about_signing" for details.
Location Line: 1 Character: 33
- .\install-service-packetbeat.ps1 <<<<
- CategoryInfo : NotSpecified: (:) [], PSSecurityException
- FullyQualifiedErrorId : RuntimeException
You need to modify this parameter and execute the following command to enable the Powershell script function:
PS E:\packetbeat> set-ExecutionPolicy RemoteSigned
Packetbeat Network traffic monitoring
Packetbeat is a part of the beats product, which is responsible for network packet analysis. It can:
Monitor traffic for a specific network card;
you can set the relevant monitoring object and port number, support dns, http, memcahce, mysql, pgsql, redis, thrift, mongodb, etc.;
can output to a specific destination, such as elasticsearch, logstash, file, console, etc.
Download
https://www.elastic.co/downloads/beats
Deploy the
Linux environment
The first step is to decompress
After downloading the installation package of .tar.gz, decompress it:
tar -zxvf packetbeat-1.2.3-x86_64.tar.gz
Enter the decompressed folder, there are 3 files in it:
--- packetbeat #Startup file
--- packetbeat.template.json #Mapping configuration in
Elasticsearch--- packetbeat.yml #Packetbeat configuration file The
second step, modify the configuration file The
configuration file includes several parts:
Configure the graphics card for network monitoring
interfaces:
device:any
configuration protocol
protocols:
http:
ports:[80,8080,9000]
redis:
ports:[6379]
configure output
output:
elasticsearch:
hosts:["localhost:9200"]
inex:"packetbeat"
template:
name:"packetbeat"
path:"packetbeat.template.json"
overwrite:false
logstah:
hosts:["localhost:5044"]
file:
path:"/tmp/packetbeat"
filename:packetbeat
console:
shipper:
logging:
第三步，运行
正常的运行：
./packetbeat
If you want to run in the background, you can do it like this:
nohup ./packetbeat &
default logs will be output to nohup.out.
Windows environment The
first step, decompression
Compared with linux, there are two more powershell scripts.
--- install-service-packetbeat.ps1 # Registration script
--- uninstall-service-packetbeat.ps1 # Logout script
--- packetbeat.exe # Startup file
--- packetbeat.template.json # Mapping configuration in
Elasticsearch- -- packetbeat.yml #Packetbeat configuration file The
second step is to enter the command line as an administrator and run the registration script
. Enter the specified directory and run the registration script.
.\install-service-winlogbeat.ps1
The third step is to start the service
Start-Service packetbeat.exe to connect
with Elasticsearch
Packetbeat and the configuration is as follows:
elasticsearch:
hosts: ["localhost:9200"]
index: "packetbeat"
template:
name: "packetbeat"
path: "packetbeat.template.json"
对接logstash
Packetbeat配置如下:
logstash:
# The Logstash hosts
hosts: ["localhost:5044"]
logstash uses logstash-input-beats to receive, the configuration can refer to the following:
input{
beats{
port => 5044
}
stdin{}
}
output{
stdout{
codec => rubydebug
}
file{
path => "E:\server.log"
}
}
存储到file
PacketBeat配置:
file:
path: "E:/packetbeat"
filename: packetbeat
polls by file size by default.
Log Management
Log can set the location and level of output. Similar to the usual log4j:
logging:
files:
path: E:/mybeat
name: mybeat
level: debug
Packetbeat监听到的内容
{
"_index": "packetbeat-2016.08.01",
"_type": "dns",
"_id": "AVZELeQzbZnlZq0jh6Vk",
"_version": 1,
"_score": 1,
"_source": {
"@timestamp": "2016-08-01T03:37:53.106Z",
"beat": {
"hostname": "XINGHL",
"name": "XINGHL"
},
"bytes_in": 31,
"bytes_out": 260,
"client_ip": "10.4.45.44",
"client_port": 51599,
"client_proc": "",
"client_server": "",
"count": 1,
"direction": "out",
"dns": {
"additionals": [
{
"class": "IN",
"data": "115.239.210.176",
"name": "ns4.a.shifen.com",
"ttl": 281,
"type": "A"
},
{
"class": "IN",
"data": "119.75.222.17",
"name": "ns5.a.shifen.com",
"ttl": 281,
"type": "A"
},
{
"class": "IN",
"data": "61.135.165.224",
"name": "ns1.a.shifen.com",
"ttl": 281,
"type": "A"
},
{
"class": "IN",
"data": "180.149.133.241",
"name": "ns2.a.shifen.com",
"ttl": 281,
"type": "A"
},
{
"class": "IN",
"data": "61.135.162.215",
"name": "ns3.a.shifen.com",
"ttl": 281,
"type": "A"
}
],
"additionals_count": 5,
"answers": [
{
"class": "IN",
"data": "www.a.shifen.com",
"name": "sp1.baidu.com",
"ttl": 33,
"type": "CNAME"
},
{
"class": "IN",
"data": "61.135.169.125",
"name": "www.a.shifen.com",
"ttl": 282,
"type": "A"
},
{
"class": "IN",
"data": "61.135.169.121",
"name": "www.a.shifen.com",
"ttl": 282,
"type": "A"
}
],
"answers_count": 3,
"authorities": [
{
"class": "IN",
"data": "ns5.a.shifen.com",
"name": "a.shifen.com",
"ttl": 1182,
"type": "NS"
},
{
"class": "IN",
"data": "ns1.a.shifen.com",
"name": "a.shifen.com",
"ttl": 1182,
"type": "NS"
},
{
"class": "IN",
"data": "ns3.a.shifen.com",
"name": "a.shifen.com",
"ttl": 1182,
"type": "NS"
},
{
"class": "IN",
"data": "ns2.a.shifen.com",
"name": "a.shifen.com",
"ttl": 1182,
"type": "NS"
},
{
"class": "IN",
"data": "ns4.a.shifen.com",
"name": "a.shifen.com",
"ttl": 1182,
"type": "NS"
}
],
"authorities_count": 5,
"flags": {
"authoritative": false,
"recursion_allowed": true,
"recursion_desired": true,
"truncated_response": false
},
"id": 32509,
"op_code": "QUERY",
"question": {
"class": "IN",
"name": "sp1.baidu.com",
"type": "A"
},
"response_code": "NOERROR"
},
"ip": "210.83.210.155",
"method": "QUERY",
"port": 53,
"proc": "",
"query": "class IN, type A, sp1.baidu.com",
"resource": "sp1.baidu.com",
"responsetime": 1,
"server": "",
"status": "OK",
"transport": "udp",
"type": "dns"
}
}
filebeat log monitoring
filebeat is an important part of Beats, it can be used as a lightweight Data collection engine, replacing the previous logstash-forward.
Download
https://www.elastic.co/downloads/beats
Explain that
filebeat.yml is the configuration file of filebeat, including the following parts:
-- filebeat # Configure the object monitored by filebeat, that is, the file path or directory path
-- output # Output configuration, support es, logstash, file, console, etc.
-- shipper
-- logging # Configure log
filebeat.template.json to provide by default The elasticsearch mapping template
filebeat as the main executor
Run the
linux environment and
run the command to decompress the installation package——filebeat.tar.gz
tar -zxvf filebeat.tar.gz
编辑 filebeat.yml
vim filebeat.yml
dynamic filebeat
nohup ./filebeat &
windows environment
Run cmd as an administrator, and execute the Powershell command to enter PS mode. Start the filebeat registration script:
C:\Windows\system32>Powershell
Windows PowerShell
Copyright (C) 2009 Microsoft Corporation. all rights reserved.
PS C: \ Windows \ system32> e:
PS E:> cd. \ Filebeat-1.2.3-windows
PS E: \ filebeat-1.2.3-windows> dir
目录: E:\filebeat-1.2.3-windowsMode LastWriteTime Length Name
---- ------------- ------ ----
----- 2016/5/18 4:33 10361856 filebeat.exe
----- 2016/5/18 4:33 814 filebeat.template.json
----- 2016/5/18 4:33 17533 filebeat.yml
----- 2016/5/18 4:33 442 install-service-filebeat.ps1
----- 2016/5/18 4:33 184 uninstall-service-filebeat.ps1
PS E:\filebeat-1.2.3-windows> .\install-service-filebeat.ps1
Status Name DisplayName
------ ---- -----------
Stopped filebeat filebeat
Edit the configuration file, filebeat.yml
Start the filebeat file
PS E:\filebeat-1.2.3-windows> Start-Service filebeat
样例
{
"_index": "filebeat-2016.08.01",
"_type": "log",
"_id": "AVZE1AMfbZnlZq0jh6cF",
"_version": 1,
"_score": 1,
"_source": {
"@timestamp": "2016-08-01T06:39:15.193Z",
"beat": {
"hostname": "XINGHL",
"name": "XINGHL"
},
"count": 1,
"fields": null,
"input_type": "log",
"message": "hello filebeat",
"offset": 22988,
"source":"e:\logs\test.log", The startup methods are similar to the previous ones, so I won't go into details here.topbeat monitors process resource information}}
"type": "log"
topbeat - windows版
{
"_index": "topbeat-windows-2016.08.01",
"_type": "process",
"_id": "AVZE7zC6bZnlZq0jh8QD",
"_version": 1,
"_score": 1,
"_source": {
"@timestamp": "2016-08-01T07:09:01.206Z",
"beat": {
"hostname": "XINGHL",
"name": "XINGHL"
},
"count": 1,
"proc": {
"cmdline": "%SystemRoot%\system32\csrss.exe ObjectDirectory=\Windows SharedSection=1024,20480,768 Windows=On SubSystemType=Windows ServerDll=basesrv,1 ServerDll=winsrv:UserServerDllInitialization,3 ServerDll=winsrv:ConServerDllInitialization,2 ServerDll=sxssrv,4 ProfileControl=Off MaxRequestThreads=16",
"cpu": {
"user": 5538,
"user_p": 0,
"system": 7753,
"total": 13291,
"start_time": "Jan01"
},
"mem": {
"size": 3391488,
"rss": 3366912,
"rss_p": 0,
"share": 0
},
"name": "csrss.exe",
"pid": 544,
"ppid": 0,
"state": "running",
"username": "NT AUTHORITY\SYSTEM"
},
"type": "process"
}
}
topbeat - linux版本
{
"_index": "topbeat-2016.08.01",
"_type": "process",
"_id": "AVZE6Mh4bZnlZq0jh6jT",
"_version": 1,
"_score": 1,
"_source": {
"@timestamp": "2016-08-01T07:01:09.641Z",
"beat": {
"hostname": "10.0.67.101",
"name": "10.0.67.101"
},
"count": 1,
"proc": {
"cpu": {
"user": 0,
"user_p": 0,
"system": 0,
"total": 0,
"start_time": "Jul06"
},
"mem": {
"size": 0,
"rss": 0,
"rss_p": 0,
"share": 0
},
"name": "migration/0",
"pid": 5,
"ppid": 2,
"state": "sleeping",
"username": "root"
},
"type": "process" The startup method is similar to the previous ones, so I won't go into details here.winlogbeat windows event listener}
}
{
"_index": "winlogbeat-2015.11.09",
"_type": "wineventlog",
"_id": "AVZE_J7FbZnlZq0jh_sL",
"_version": 1,
"_score": 1,
"_source": {
"@timestamp": "2015-11-09T00:28:50.953Z",
"beat": {
"hostname": "XINGHL",
"name": "XINGHL"
},
"computer_name": "xinghailong",
"count": 1,
"event_id": 35,
"level": "信息",
"log_name": "System",
"message": "时间服务现在用时间源 time.neusoft.com,0x9 (ntp.m|0x9|0.0.0.0:123->202.118.6.8:123) 同步系统时间。",
"record_number": "25479",
"source_name": "Microsoft-Windows-Time-Service",
"type": "wineventlog",
"user": {
"domain": "NT AUTHORITY",
"identifier": "S-1-5-19",
"name": "LOCAL SERVICE",
"type": "Well Known Group"
}
}
}
参考
1 官方文档
2 ELK Beats Documentation
The blog is no longer maintained, please follow the new blog: https://xinghalo.github.io/
http://www.cnblogs.com/xing901022/category/677227.html
http://www.cnblogs.com/xing901022/category/642865.html