Kubernetes combat summary-EFK deployment (v7.6.0) EFK (Elasticsearch + Filebeat + Kibana) collect container logs

Basic concepts

• Elasticsearch is a real-time, distributed and scalable search engine that allows full-text, structured search. It is usually used to index and search large amounts of log data, and can also be used to search many different types of documents.
• Beats is a powerful tool for data collection. Place Beats on your server with your container, or deploy Beats as a function, and then centrally process the data in Elastisearch. If you need more powerful processing performance, Beats can also send the data to Logstash for conversion and analysis.
• Kibana core products are equipped with a number of classic functions: histogram, line graph, pie chart, sun chart, etc. Not only that, you can also use Vega syntax to design your own visual graphics. All of these take advantage of the full aggregation capabilities of Elasticsearch.

 

Elasticsearch is usually deployed with Kibana. Kibana is a powerful data visualization Dashboard of Elasticsearch. Kibana allows you to browse Elasticsearch log data through a web interface.

 

 

 

---

Installation and deployment

Here we use Helm for deployment, so we need to install Helm first.

wget -O helm.tgz https://get.helm.sh/helm-v3.1.1-linux-amd64.tar.gz
tar -zxvf helm.tgz
cd linux-amd64
mv helm /usr/local/bin && chmod a+x /usr/local/bin
helm version

For details, refer to: Kubernetes combat summary-Helm deployment (v3.1.1)

 

Then we download the Helm warehouse source code

wget -O helm-charts.tgz  https://github.com/elastic/helm-charts/archive/7.6.2.tar.gz
tar -zxvf helm-charts.tgz
cd helm-charts-7.6.2

 

Create a PV for elasticsearch. NFS is used here, but you can also use other methods.

# es-pv.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
  name: elasticsearch
spec:
  capacity:
    storage: 30Gi
  accessModes:
    - ReadWriteOnce
  persistentVolumeReclaimPolicy: Retain
  nfs:
    path: /nfsdata/elasticsearch
    server: *.*.*.*

kubectl create -f es-pv.yaml

 

Deploy elasticsearch

elasticsearch vim / values.yaml 

# according to their number of nodes change 
Replicas: 1 
minimumMasterNodes: 1     
....... 

# change the image source and version of 
Image: " registry.cn-shanghai.aliyuncs.com/leozhanggg/elastic/elasticsearch " 
imageTag: " 7.6.0 " 
imagePullPolicy: " IfNotPresent " 
...... 

# own resources to adjust the size of 
resources: 
  Requests: 
    the CPU: " 1000m " 
    Memory: " 2Gi " 
  Limits: 
    the CPU: " 1000m " 
    Memory:"2Gi"
......

# 存在对应PV
volumeClaimTemplate:
  accessModes: [ "ReadWriteOnce" ]
  resources:
    requests:
      storage: 30Gi

helm install es --namespace=efk ./elasticsearch

 

Deploy filebeat

filebeat vim / values.yaml 

# log collection catalog 
hostPathRoot: / var / lib 
hostNetworking: false 

# change the image source and version of 
Image: " registry.cn-shanghai.aliyuncs.com/leozhanggg/elastic/filebeat " 
imageTag: " 7.6.0 " 
imagePullPolicy: " IfNotPresent " 

helm install fb --namespace = efk ./filebeat

 

Department kibana

kibana vim / values.yaml
 
# change the image source and version of 
Image: " registry.cn-shanghai.aliyuncs.com/leozhanggg/elastic/kibana " 
imageTag: " 7.6.0 " 
imagePullPolicy: " IfNotPresent " 
...... 

# Change to NodePort mode 
service: 
  type: NodePort 
  port: 5601 
  nodePort: "30005" 
  
helm install kb --namespace = efk ./kibana

Wait for deployment to complete

[root~ ]# kubectl get pod -n efk
NAME                             READY   STATUS    RESTARTS   AGE
elasticsearch-master-0           1/1     Running   0          28h
filebeat-filebeat-967m2          1/1     Running   0          28h
filebeat-filebeat-lr79k          1/1     Running   0          28h
kibana-kibana-64f5869d86-qrkrq   1/1     Running   0          28h

 

Visit masterip: port to view logs

 

 

---

Of course, this is not the best implementation. In this architecture, according to the characteristics of the business, you can also add some middleware, such as Redis, Kafak, etc.

For more details refer to >>>   How to monitor K8S cluster logs         EFK (Elasticsearch + Filebeat + Kibana) to collect container logs

 

Author: Leozhang GG

Source: https://www.cnblogs.com/leozhanggg/p/12700363.html

The copyright of this article belongs to the author and the blog garden. Welcome to reprint, but this paragraph statement must be retained without the author's consent, and the original text link is given in an obvious position on the article page, otherwise the right to pursue legal responsibility is reserved.