Simple ELK log display (docker)
introduction
At present, many data services are arranged on k8s, and the logs of the services are scattered on different machines. Because it is an online environment, try to make couples. For developers, the k8s cluster is a black box, and ordinary developers are not allowed. Go to the machine to view the log. In the beginning, use flume to collect logs to kafka and hive (unified requirements, small companies do not have too many resources for testing), because the synchronization of hive is once an hour, plus the synchronization time, the problem cannot be found in time, and the fastest time is required Solving their problem, so it took less than 3 hours to solve the log problem perfectly with ELK.
Build
es (I deployed 12 units, here are three units as an example)
docker-compose-server.yml (create related directories)
version: '3.7'
services:
elasticsearch-server:
image: elasticsearch:6.7.0
restart: always
network_mode: "host"
container_name: "elasticsearch"
ports:
- "9200:9200"
- "9300:9300"
volumes:
- "/xxx/elasticsearch/config/elasticsearch.yml:/usr/share/elasticsearch/config/elasticsearch.yml"
- "/xxx/elasticsearch/config/jvm.options:/usr/share/elasticsearch/config/jvm.options"
- "/xxx/elasticsearch/plugins:/usr/share/elasticsearch/plugins"
- "/xxx/elasticsearch/log:/usr/share/elasticsearch/log"
- "/xxx/elasticsearch/data:/usr/share/elasticsearch/data"
- "/etc/localtime:/etc/localtime:ro"
Change setting
vim /etc/sysctl.conf
添加：vm.max_map_count=655360
sysctl -p /etc/sysctl.conf/
必要时需要
chmod 777 /xxx/elasticsearch/data
elasticsearch.yml
#集群名称
cluster.name: data-es-cluster
#当前该节点的名称
node.name: es01
#是不是有资格竞选主节点
node.master: true
#是否存储数据
node.data: true
#最大集群节点数
node.max_local_storage_nodes: 3
#数据存档位置
path.data: /usr/share/elasticsearch/data
#日志存放位置
path.logs: /usr/share/elasticsearch/log
# ---------------------------------- Network -----------------------------------
#然后启动一直报无效的IP地址，无法注入9300端口，这里只需要填写0.0.0.0
network.host: 0.0.0.0
#设置其它结点和该结点交互的ip地址，如果不设置它会自动判断，值必须是个真实的ip地址，设置当前物理机地址,
#如果是docker安装节点的IP将会是配置的IP而不是docker网管ip
network.publish_host: 10.10.30.203
#设置映射端口
http.port: 9200
#内部节点之间沟通端口
transport.tcp.port: 9300
# --------------------------------- Discovery ----------------------------------
#集群发现默认值为127.0.0.1:9300,如果要在其他主机上形成包含节点的群集,如果搭建集群则需要填写
discovery.seed_hosts: ["es01:9300","es02:9300","es03:9300"]
cluster.initial_master_nodes: ["es01"]
discovery.zen.minimum_master_nodes: 1
# ---------------------------------- Gateway -----------------------------------
gateway.recover_after_nodes: 2
logstash
docker run -itd -p 5044:5044 -p 9600:9600 --name logstash -v /xxx/logstash/data:/usr/share/logstash/data --privileged=true logstash:7.5.1 /bin/bash
kibana
docker-compose.yml
version: '3.7'
services:
kibana:
image: kibana:6.7.0
restart: always
network_mode: "host"
container_name: "kibana"
ports:
- "5601:5601"
volumes:
- "/xxxx/kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml"
- "/etc/localtime:/etc/localtime:ro"
kibana.yml
erver.name: "data-kibana"
server.host: "0"
elasticsearch.hosts: ["http://es01:9200","http://es02:9200","http://es03:9200"]
xpack.monitoring.ui.container.elasticsearch.enabled: true
i18n.locale: "zh-CN"
example
1. Log data
2020/11/26 18:49:37 ◊ INFO ◊ ****/common/endpointkit/baseRequestEndpoint.go ◊ (*BaseRequestEndpoint):60 ◊ {
"api":"/****/v1/******?/******?=/******?","caller":"******?","method":"POST","logid":"20201126184937-36283567-df060e15-ed8d-48c4-9a95-095f788ea102","ts":"1606387777765628577"} ◊ {
"id":"3907944","limit":"1","pool_level":"0"} ◊ {
"status":200,"message":"","data":{
"id":[243715],"comments":null}}
2. The
grok grammar of logstage.conf is a big problem for beginners, here is the most violent way to solve it
input {
kafka{
bootstrap_servers=> "kafka01:9092,kafka02:9092,kafka03:9092"
group_id =>"es"
topics =>"service_log"
consumer_threads =>3
decorate_events =>true
}
}
filter {
grok {
match => {
"message" => "%{
GREEDYDATA:ctime}\ [◊]\ %{
GREEDYDATA:level}\ [◊]\ %{
GREEDYDATA:fliename}\ [◊]\ %{
GREEDYDATA:point}\ [◊]\ %{
GREEDYDATA:base_json}\ [◊]\ %{
GREEDYDATA:request_json}\ [◊]\ %{
GREEDYDATA:response_json}"}
remove_field => "message"
remove_field => "point"
}
date {
match => ["ctime", "yyyy-MM-dd HH:mm:ss"]
target => "@timestamp"
}
mutate {
split => ["ctime"," "]
add_field => {
"date" => "%{
[ctime][0]}"
}
remove_field => ["ctime"]
}
mutate {
split => ["date","/"]
add_field => {
"idx" => "%{
[date][0]}.%{
[date][1]}.%{
[date][2]}"
}
remove_field => ["date"]
}
}
output{
elasticsearch{
hosts => ["es01:9200","es02:9200","es03:9200"]
index =>"service_log_%{
idx}"
document_type => "doc"
}
}
Note: service_log_yyyy-MM-ddThe advantage of this is that it is convenient to delete the index regularly
3. Run
## 进入logstash的容器
docker exec -it logstash bash
## 执行
bin/logstash -f data/logstage.conf --path.data tmp/log
effect
