EPK开源日志可视化管理服务部署解析
服务结构
- 由原始的ELK（elasticsearch+logstash+kibana）更换为（elasticsearch+python+kibana）服务
- 材料信息：
| 服务 | 描述 |
|---|---|
| elasticsearch | (6.4.2) |
| 端口控制 | 9200 |
| 访问链接 | http://IP:9200/_cat/indices?v |
| kibana | (6.4.2) |
| 端口控制 | 5601 |
| 访问链接 | http://IP:5601 |
| python | 2.7.+ |
| git地址 | https://github.com/dyiwen |
elasticsearch部署
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-6.4.2.tar.gz;
tar xvf elasticsearch-6.4.2.tar.gz;
useradd es;
passwd es;
chown -R es:es ./elasticsearch-6.4.2;
vim elasticsearch-6.4.2/config/elasticsearch.yml; #修改es配置文件
------------------------------------------------------------------
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: epk-colony #集群名字，如果不配置该项，系统默认取elasitcsearch
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node-1 #节点名称，同一集群的节点名称不能相同，如果不配置该项，系统会随机分配一个名称。
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
node.master: True #指定是否为主节点。该属性可不指定，节点之间自主选举。
node.data: false #数据节点，是否存储数据
node.ingest: True
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /data/dyiwen/elasticsearch-6.4.2/data/store #数据存储路径
#
# Path to log files:
#
path.logs: /data/dyiwen/elasticsearch-6.4.2/data/log #日志存储路径
#
#path.conf: /path/to/conf #配置文件所在位置
#path.plugins: /path/to/plugins #插件安装位置
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
bootstrap.memory_lock: true #是否锁定内存，提高ES性能
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: xxx.xxx.xxx.xxx #本机内网IP
#
# Set a custom port for HTTP:
#
#http.port: 9200 #访问端口
#
transport.tcp.port: 9300 #数据传输端口
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#集群内节点的主机
discovery.zen.ping.unicast.hosts: ["172.18.xxx.xxx:9300","172.18.xxx.xxx:9200"]
#
# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
#设置这个参数来保证集群中的节点可以知道其它N个有master资格的节点。默认为1，对于大的集群来说，可以设置大一点的值（2-4）
discovery.zen.minimum_master_nodes: 1
#
# For more information, consult the zen discovery module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#设置集群中N个节点启动时进行数据恢复，默认为1。
#gateway.recover_after_nodes: 3
#
#设置该集群中可存在的节点上限：gateway.expected_nodes: 2
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#设置是否可以通过正则或者_all删除或者关闭索引
#action.destructive_requires_name: true
------------------------------------------------------------------
- 启动elasticsearch
rpm -ivh jdk-8u191-linux-x64.rpm;
cd ./elasticsearch-6.4.2;
su es;
bin/elasticsearch;#首次启动避免后台方便查看日志
nohup ./bin/elasticsearch -d >/dev/null 2>&1 & #后台运行
#删除ES索引
curl -XDELETE 'http://es地址:9200/{索引名称}
问题1：
max file descriptors [4096] for elasticsearch process is too low, increase to at least [65536]
#root用户修改
vim /etc/security/limits.conf
#最后追加
************************************************************************************
#-----------------------------------------2018-12-03 by dyiwen----------------------
es hard nofile 65536
es soft nofile 65536
*************************************************************************************
#修改后es用户检查是否修改成功
su es;
ulimit -Hn;
65536 #修改成功
问题2：
max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
#提高vm.max_map_count 的大小
#root用户
vim /etc/sysctl.conf
#追加
***************************************************************
--------------------------------20190216 by dyiwen--------------
vm.max_map_count=262144
***************************************************************
#检查
sysctl -p;
问题3：
Java HotSpot™ 64-Bit Server VM warning: INFO: os::commit_memory(0x0000000085330000, 2060255232, 0) failed; error=‘Cannot allocate memory’ (errno=12)
修改jvm空间分配
vim config/jvm.options;
-Xms512m
-Xmx512m
kibana部署
#下载kibana
wget https://artifacts.elastic.co/downloads/kibana/kibana-6.4.2-linux-x86_64.tar.gz;
tar xvf kibana-6.4.2-linux-x86_64.tar.gz;
vim kibana-6.4.2-linux-x86_64/config/kibana.yml;
###################################################
#修改以下几项：
server.port: 5601
server.host: "xxx.xxx.xxx.xxx"
server.name: "kibana_6.4.2"
elasticsearch.url: "http://xxx.xxx.xxx.xxx:9200"
###################################################
#启动,注意root权限启动
cd kibana-6.4.2-linux-x86_64；
./bin/kibana；
- 添加ES索引
python2.7采集日志
- 流程思路
from elasticsearch import Elasticsearch
from elasticsearch import helpers
es=Elasticsearch(hosts='',port=9200)
helpers.bulk(client=es,actions=actions,raise_on_error=True,request_timeout=30)#将解析后的日志推送至es，入参actions
-----------------------------------------------
actions格式：
action = {'_op_type':'index',
'_index':index_,
'_type':'doc',
# '_id':i,
'_source':{
u'日期':time_,
u'对象':obj_,
u'请求':request_,
'message':line
},
'fields':{
"@timestamp":[time_]}}
-----------------------------------------------
- 启用日志收集
pip install -r requirements.txt;
#安装elasticsearch==6.3.1
#配置server.conf
[LOG]
environment = xxxx #配置索引名称
log_path = /dyiwen/xxx/xxx/xxx/logs #配置需要收集日志的地址
#启动
bash run.sh;
#定时收集
crontab -e;
30 1 * * * /opt/epk_push_server/run.sh > /dev/null 2>&1 &;
#每晚一点收集日志