1, ELK Profile
1.1 What is the ELK
Elasticsearch is an open source distributed search engine, its features are: distributed, zero-configuration, auto-discovery, auto-slice index, index replication mechanism, restful style interfaces, multiple data sources, such as automatic load search.
Logstash is a fully open source tools, he can collect your log, filter, and store it for later use (such as search).
Kibana is also an open source and free tools, log it Kibana can provide analysis for the Logstash and ElasticSearch friendly Web interface that can help you summarize, analyze and search for important data logs.
1.2, ELK application
electricity provider architecture
issue
1, API is not the same, how are we going to integrate? - "dubbo defined uniform api specification
2, the operation will have marks (track user behavior) between subsystems ---" log
3, each subsystem generates a respective log Log Consolidation --- - "logstash
. 4 , the AOP buried asynchronous log output
specific scenario 1
is moved through a third party prepaid recharge
Log output: every time a print asynchronous call log
Distributed Load Balancing:
Too many machines can recharge (dynamically to select a relatively idle current of the machine to perform this task)
problem:
A: Brother, today to help check the phone number 138 001 380 000 recharge logging (success)
B: Wait
5 minutes later
A: how about
B: Wait, leaving three machines did not check End
Conclusion: if we can sort out all the logs together, there would not be a problem to an investigation
solution:
- Can you put a log on the database.
Too much data, and the log no standard log formats, database program is not recommended, and the pressure is too great
- Data processing scheme using large logs
Cost is too high, and the distributed environment of each system log rules are not the same.
specific business practices
Log Collection: Logstash
log storage: ElasticSearch
log shows: Kibana
for Taiwan server log is not unified, offers a variety of retrieval rules to facilitate the visual display
Summary
Distributed problem to bring: multiple nodes, load balancing, log dispersion, high operation and maintenance costs (requires human tracking)
1.3 Centralized log management system
Some of the current mainstream centralized log management system
1, simple: Rsyslog
2. Commercialization: Splunk
3. Open Source: Scribe (FaceBook), Chukwa (Apache)
4, ELK most extensive (Elastic Stack) (java language)
1.4、ELK
| ElasticSearch |
Java |
Real-time distributed search and analysis engine, he can be used for full-text search , structured search and analysis, lucene. Solr |
| Logstash |
JRuby |
Channel capacity of real-time data collection engine, comprising an input filter, the output module, generally do parsing log format in the filtration module |
| Kibana |
JavaScript |
ElasticSerach platform to provide analysis and visualization of the Web platform . He can ElasticSerach index lookup, call data, and generate a variety of dimensions table Figure |
1.5 log
Log: running track recording program ---
levels: ERROR, INFO, DEBUG, WARN
purpose: finding information easy to locate and record additional information to remove foreign business links
Filebeat Introduction
When you have to face hundreds of thousands, even tens of thousands of log servers, virtual machines and containers generated, please farewell SSH it. Filebeat will provide you with a lightweight type method, and for forwarding the summary log file, so that the simple things are not complicated.
When transmitting data to or Logstash Elasticsearch, Filebeat backpressure sensitive protocol, to account for a larger amount of data. If Logstash is busy processing data, you can let Filebeat know to slow down the reading speed. Once the congestion is resolved, Filebeat will be restored to its original pace and continue to run.
Whether in any environment, always lurking risk of application interruption. Filebeat able to read and forward the log line, if there is an interruption will be back to normal after all, continues from the stopped position before the interruption.
2. Preparation
2.1, install Centos7
More than 2G of memory is recommended
2.2, Basic Configuration
Set the IP address
vi /etc/sysconfig/network-scripts/ifcfg-eno33
service network restart add users and authorized
[root@localhost ~]# adduser elk1
[root@localhost ~]# passwd elk1
[root@localhost ~]# whereis sudoers
[root@localhost ~]# ls -l /etc/sudoers
[root@localhost ~]# chmod -v u+w /etc/sudoers
[root@localhost ~]# vi /etc/sudoers
## Allow root to run any commands anywher
root ALL=(ALL) ALL
linuxidc ALL=(ALL) ALL #这个是新增的用户
[root@localhost ~]# chmod -v u-w /etc/sudoers
[root@localhost ~]# su elk1
3、ElasticSerach
3.1, Java installation environment
extracting installation package
[root@localhost jdk1.8]# tar -zxvf jdk-8u171-linux-x64.tar.gz Setting up the Java environment variables
[root@localhost jdk1.8.0_171]# vi /etc/profileIn the file add
export JAVA_HOME=/home/elk1/jdk1.8/jdk1.8.0_171
export JRE_HOME=$JAVA_HOME/jre
export CLASSPATH=.:$JAVA_HOME/LIB:$JRE_HOME/LIB:$CLASSPATH
export PATH=$JAVA_HOME/bin:$JRE_HOME/bin:$PATH
[root@localhost jdk1.8.0_171]# source /etc/profile
[root@localhost jdk1.8.0_171]# java -version
java version "1.8.0_171"
Java(TM) SE Runtime Environment (build 1.8.0_171-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode)
3.2, ElasticSerach stand-alone installation
[root@localhost elasticserach]# tar -zxvf elasticsearch-6.3.1.tar.gz
[root@localhost elasticserach]# cd elasticsearch-6.3.1/bin
[root@localhost bin]# ./elasticsearch
[root@localhost bin]# su elk1
[elk1@localhost bin]$ ./elasticsearch
[root@localhost bin]# chown -R elk1:elk1 /home/elk1/elasticsearch
[elk1@localhost bin]$ ./elasticsearch
[elk1@localhost config]$ vi jvm.options
[elk1@localhost bin]$ ./elasticsearch
[root@localhost jdk1.8.0_171]# curl 127.0.0.1:9200
#后台启动
[elk1@localhost bin]$ ./elasticsearch -d
#关闭程序
[elk1@localhost bin]$ ps -ef|grep elastic
[elk1@localhost bin]$ kill 10097
#设置浏览器访问
[root@localhost bin]systemctl stop firewalld
[root@localhost bin]vi config/elasticsearch.yml
Installation Issues:
[1] [2] Solution
[root@localhost bin]# vi /etc/security/limits.conf
[3] Solution
[root@localhost bin]# vi /etc/sysctl.conf
[root@localhost bin]# sysctl -p
3.3, ElasticSerach cluster installation
Modify the configuration file elasticserach.yml
vim /elasticsearch.ymlcluster.name: aubin-cluster#必须相同
# 集群名称（不能重复）
node.name: els1（必须不同）
# 节点名称，仅仅是描述名称，用于在日志中区分（自定义）
#指定了该节点可能成为 master 节点，还可以是数据节点
node.master: true
node.data: true
path.data: /var/lib/elasticsearch
# 数据的默认存放路径（自定义）
path.logs: /var/log/elasticsearch
# 日志的默认存放路径
network.host: 192.168.0.1
# 当前节点的IP地址
http.port: 9200
# 对外提供服务的端口
transport.tcp.port: 9300
#9300为集群服务的端口
discovery.zen.ping.unicast.hosts: ["172.18.68.11", "172.18.68.12","172.18.68.13"]
# 集群个节点IP地址，也可以使用域名，需要各节点能够解析
discovery.zen.minimum_master_nodes: 2
# 为了避免脑裂，集群节点数最少为 半数+1
Note: Empty data and logs data
192.168.14.12:9200/_cat/nodes?v
3.4, install the plug-in head
Download the plug head
wget https://github.com/mobz/elasticsearch-head/archive/elasticsearch-head-master.zipYou can also use git to download the assumption yum install git
unzip elasticsearch-head-master.zip install node.js
wget https://npm.taobao.org/mirrors/node/latest-v4.x/node-v4.4.7-linux-x64.tar.gztar -zxvf node-v9.9.0-linux-x64.tar.gz node.js to add environment variables
source /etc/profile Test
node -v
npm -v installation grunt (grunt is a very handy build tool, you can work compressed package, test, execution, etc.)
Into elasticsearch-head-master
npm install -g grunt-cli
npm install
(npm install -g cnpm --registry=https://registry.npm.taobao.org) modify Elasticsearch configuration file
editing elasticsearch-6.3.1 / config / elasticsearch.yml, add the following:
http.cors.enabled: true
http.cors.allow-origin: "*" modified Gruntfile.js (note '')
opens elasticsearch-head-master / Gruntfile.js, connect to find the following properties, the new hostname: '*':
connect: {
server: {
options: {
hostname: '*',
port: 9100,
base: '.',
keepalive: true
}
}
} Start elasticsearch-head
进入elasticsearch-head目录，执行命令：grunt server background start elasticsearch-head
nohup grunt server &exit closed head plug
ps -aux|grep head
kill 进程号3.5、ElasticSerach API
elasticsearch rest api follow the format:
curl -X<REST Verb> <Node>:<Port>/<Index>/<Type>/<ID> Check the version information es
curl IP:9200 Check whether the health cluster
http://IP:9200/_cat/health?v view the list of nodes
http://IP:9200/_cat/nodes?v list all indexes and storage size
http://IP:9200/_cat/indices?v Creating an index
curl -XPUT 'IP:9200/XX?pretty' Add a Type
curl -XPUT 'IP:9200/XX/external/2?pretty' -d '
{
"gwyy": "John"
}' a type of update
curl -XPOST 'IP:9200/XX/external/1/_update?pretty' -d '
{
"doc": {"name": "Jaf"}
}' delete the specified index
curl -XDELETE 'IP:9200/_index?pretty'3.6, configuration details
ElasticSearch.yml l
ES related configuration
# 集群的名字，以此作为是否同一集群的判断条件
cluster.name: elasticsearch
# 节点名字，以此作为集群中不同节点的区分条件
node.name: node-1
#设置当前节点既可以为主节点也可以为数据节点
node.master: true
node.data: true
# 索引分片个数，默认为5片
#index.number_of_shards: 5
# 索引副本个数，默认为1个副本
#index.number_of_replicas: 1
# 数据存储目录（多个路径用逗号分隔）
discovery.zen.ping.unicast.hosts: ["192.168.14.14","192.168.14.15"]
discovery.zen.minimum_master_nodes: 2
#数据目录
path.data: /home/elk1/elasticserach/data
# 日志目录
path.logs: /home/elk1/elasticserach/logs
# 修改一下ES的监听地址，这样别的机器才可以访问
network.host: 192.168.14.13
# 设置节点间交互的tcp端口（集群）,默认是9300
transport.tcp.port: 9300
# 监听端口（默认的就好）
http.port: 9200
# 增加新的参数，这样head插件才可以访问es
http.cors.enabled: true
http.cors.allow-origin: "*"
Jvm.options l
the JVM's configuration
Log4j2.properties
log configuration
3.7, Elasticserach mode
Development and Production is divided into two modes
way to distinguish
以transport的地址是否绑定在localhost为标准（实际地址）
即：elasticserach.yml文件中的network.host配置 mode difference
(1) the warning will prompt for configuration check when you start the next Development Mode Exceptions
(2) under the Production Mode will prompt manner error configuration check at startup exception and launch
3.8, elasticserach operation
basic concepts
Document: Document Object
Index: Index (library)
Type: Data type index (Table)
Field,: fields, document properties (fields)
Query DSL: query syntax (SQL)
CRUD operations
create a document
request:
POST /newbies/student/1
{
"name":"zhangsan",
"clazz":"0115bigdata",
"description":"we are family"
}return:
{
"_index": "newbies",
"_type": "student",
"_id": "1",
"_version": 1,
"result": "created",
"_shards": {
"total": 2,
"successful": 2,
"failed": 0
},
"_seq_no": 0,
"_primary_term": 1
} obtain the document
request:
GET newbies/student/1return:
{
"_index": "newbies",
"_type": "student",
"_id": "1",
"_version": 1,
"found": true,
"_source": {
"name": "zhangsan",
"clazz": "0115bigdata",
"description": "we are family"
}
} update the document
request:
POST /newbies/student/1/_update
{
"doc":{
"description":"hello world"
}
}return:
{
"_index": "newbies",
"_type": "student",
"_id": "1",
"_version": 2,
"result": "updated",
"_shards": {
"total": 2,
"successful": 2,
"failed": 0
},
"_seq_no": 1,
"_primary_term": 1
} delete a document
request:
DELETE newbies/student/1search result:
{
"_index": "newbies",
"_type": "student",
"_id": "1",
"found": false
} Elasticserach Query
Query String
GET /newbies/student/_sea'rch?q=关键字return:
{
"took": 8,
"timed_out": false,
"_shards": {
"total": 5,
"successful": 5,
"skipped": 0,
"failed": 0
},
"hits": {
"total": 1,
"max_score": 0.2876821,
"hits": [
{
"_index": "newbies",
"_type": "student",
"_id": "1",
"_score": 0.2876821,
"_source": {
"name": "zhangsan",
"clazz": "0115bigdata",
"description": "we are family"
}
}
]
}
} Query DSL
GET newbies/student/_search
{
"query":{
"term":{
"name":{
"value":"zhangsan"
}
}
}
}4、Logstash
4.1, install logstash
[root@localhost logstash]# tar -zxvf logstash-6.3.1.tar.gz
[root@localhost logstash-6.3.1]# cd config
[root@localhost config]# vi log4j_to_es.conf
input {
file {
path=>[""]
type=>""
start_position=>"beginning"
}
}
output {
stdout {
codec=>rubydebug
}
}
[root@localhost logstash-6.3.1]# ./bin/logstash -f config/log4j_to_es.conf4.2, input, output, filter
Input
input{file{path=>”/tomcat/logs/abc.log”}} Output
output{stdout{codec=>rubydebug}} filter insert
Grok
. 1, based on regular expressions provides a rich reusable pattern (pattern)
2, based on this structure can be unstructured data processing
Date
converts a character string type of time stamp field types, to facilitate subsequent data processing
mutate
add, modify, delete, and other fields related to the replacement process
4.3, logstash format nginx log content
Creating nginx_logstash.conf file
input {
stdin { }
}
filter {
grok {
match => {
"message" => '%{IPORHOST:remote_ip} - %{DATA:user_name} \[%{HTTPDATE:time}\] "%{WORD:request_action} %{DATA:request} HTTP/%{NUMBER:http_version}" %{NUMBER:response} %{NUMBER:bytes} "%{DATA:referrer}" "%{DATA:agent}"'
}
}
date {
match => [ "time", "dd/MMM/YYYY:HH:mm:ss Z" ]
locale => en
}
geoip {
source => "remote_ip"
target => "geoip"
}
useragent {
source => "agent"
target => "user_agent"
}
}
output {
stdout {
codec => rubydebug
}
}
Logstash start parsing file nginx
head -n 2 /home/elk1/nginx_logs|./logstash -f ../config/nginx_logstash.conf
l results
{
"user_name" => "-",
"referrer" => "-",
"@timestamp" => 2015-05-17T08:05:32.000Z,
"request" => "/downloads/product_1",
"time" => "17/May/2015:08:05:32 +0000",
"geoip" => {
"country_code3" => "NL",
"longitude" => 4.8995,
"continent_code" => "EU",
"latitude" => 52.3824,
"timezone" => "Europe/Amsterdam",
"country_code2" => "NL",
"ip" => "93.180.71.3",
"country_name" => "Netherlands",
"location" => {
"lat" => 52.3824,
"lon" => 4.8995
}
},
"@version" => "1",
"http_version" => "1.1",
"remote_ip" => "93.180.71.3",
"message" => "93.180.71.3 - - [17/May/2015:08:05:32 +0000] \"GET /downloads/product_1 HTTP/1.1\" 304 0 \"-\" \"Debian APT-HTTP/1.3 (0.8.16~exp12ubuntu10.21)\"",
"bytes" => "0",
"user_agent" => {
"minor" => "3",
"os" => "Debian",
"name" => "Debian APT-HTTP",
"os_name" => "Debian",
"build" => "",
"major" => "1",
"device" => "Other"
},
"agent" => "Debian APT-HTTP/1.3 (0.8.16~exp12ubuntu10.21)",
"host" => "localhost.localdomain",
"response" => "304",
"request_action" => "GET"
}
5, Kibana
5.1, Kibana AnSo
[root@localhost kibana]# tar -zxvf kibana-6.3.1-linux-x86_64.tar.gz
[root@localhost kibana]# cd kibana-6.3.1-linux-x86_64/config
[root@localhost config]# vi kibana.yml
[root@localhost bin]# ./kibana
5.2, kibana arrangement
Configuration folder in the config file
Kibana.yml common configuration instructions
Server.host/server.port:访问的端口号和地址(地址设置后才能被外网访问)
Elasticsearch.url:访问elasticserach的地址5.3, kibana Features
Discover:数据搜索查看
Visualize:图标制作
Dashboard:仪表盘制作
Timeline:时序数据的高级可视化分析
DevTools:开发者工具
Management:kibana相关配置
6、Filebeat和packetbeat
6.1、Filebeat
Download Filebeat
https://www.elastic.co/cn/downloads/beats/filebeat
view the system median: getconf LONG_BIT
2.2、Packetbeat
Packetbeat Introduction
(1) real-time to crawl network packets
(2) automatically parse the application layer protocol (capture)
the DNS, Http, Redis, Mysql, etc.
Packetbeat crawl elasticserach request data
(1) enters the packetbeat directory, create a file es.yml
(2) file editing es.yml
packetbeat.interfaces.device: ens33#网卡
packetbeat.protocols.http:
ports: [9200]#es端口
send_request: true#抓取请求信息
include_body_for: ["application/json", "x-www-form-urlencoded"]#包含内容
output.console:
pretty: true#控制台输出
(3) Start packetbeat
sudo ./packetbeat -e -c es.yml -strict.perms=false7、Nginx
安装nginx
#安装依赖环境
yum install gcc-c++
yum install pcre-devel
yum install zlib zlib-devel
yum install openssl openssl-deve
#//一键安装上面四个依赖
#yum -y install gcc zlib zlib-devel pcre-devel openssl openssl-devel
#解压
tar -xvf nginx-1.13.7.tar.gz
#进入nginx目录
cd /usr/local/nginx #执行命令
./configure
#执行make命令make//执行make install命令
make
make install
//启动命令
nginx/sbin/nginx
//停止命令
nginx/sbin/nginx -s stop或者 : nginx -s quit
//重启命令
nginx -s reload8, the actual data visualization Demo
8.1, the actual description
demand:
Collection query Elasticserach cluster
Analysis of common query statement, response and long
Program
Data collection: Packetbeat + logstash
Data analysis: Kibana + Elasticsearch
8.2, preparation
Production Cluster(生产环境)
1、Elasticsearch 192.168.14.13:9200
2、Kibana 192.168.14.15:5601
Monitoring Cluster(监控环境)
1、Elasticsearch 192.168.14.16:8200
2、Kibana 192.168.14.16:8601
Logstash\packetbeat
8.3, combat
Start collecting data cluster
Start ES:
./elasticsearch
Modify the configuration kibana
./kibana #启动 Start the data analysis cluster
(1) ES start
ditto
(2) Start logstash
input {
beats {
port => 5044
}
}
filter {
if "search" in [request]{
grok {
match => { "request" => ".*\n\{(?<query_body>.*)"}
}
grok {
match => { "path" => "\/(?<index>.*)\/_search"}
}
if [index] {
} else {
mutate {
add_field => { "index" => "All" }
}
}
mutate {
update => { "query_body" => "{%{query_body}"}}
}
# mutate {
# remove_field => [ "[http][response][body]" ]
# }
}
output {
#stdout{codec=>rubydebug}
if "search" in [request]{
elasticsearch {
hosts => "127.0.0.1:9200"
}
}
}
(3) Start
./bin/logstash -f config/log4j_to_es.conf附录：防火墙配置
1、firewalld的基本使用
启动： systemctl start firewalld
关闭： systemctl stop firewalld
查看状态： systemctl status firewalld
开机禁用 ： systemctl disable firewalld
开机启用 ： systemctl enable firewalld
2.systemctl是CentOS7的服务管理工具中主要的工具，它融合之前service和chkconfig的功能于一体。
启动一个服务：systemctl start firewalld.service
关闭一个服务：systemctl stop firewalld.service
重启一个服务：systemctl restart firewalld.service
显示一个服务的状态：systemctl status firewalld.service
在开机时启用一个服务：systemctl enable firewalld.service
在开机时禁用一个服务：systemctl disable firewalld.service
查看服务是否开机启动：systemctl is-enabled firewalld.service
查看已启动的服务列表：systemctl list-unit-files|grep enabled
查看启动失败的服务列表：systemctl --failed
3.配置firewalld-cmd
查看版本： firewall-cmd --version
查看帮助： firewall-cmd --help
显示状态： firewall-cmd --state
查看所有打开的端口： firewall-cmd --zone=public --list-ports
更新防火墙规则： firewall-cmd --reload
查看区域信息: firewall-cmd --get-active-zones
查看指定接口所属区域： firewall-cmd --get-zone-of-interface=eth0
拒绝所有包：firewall-cmd --panic-on
取消拒绝状态： firewall-cmd --panic-off
查看是否拒绝： firewall-cmd --query-panic
4.那怎么开启一个端口呢
添加
firewall-cmd --zone=public --add-port=80/tcp --permanent （--permanent永久生效，没有此参数重启后失效）
重新载入
firewall-cmd --reload
查看
firewall-cmd --zone= public --query-port=80/tcp
删除
firewall-cmd --zone= public --remove-port=80/tcp --permanent
