** efk的搭建**Elasticsearch is a real-time, distributed and scalable search engine that allows full-text and structured search, it is usually used to index and search a large amount of log data can also be used to search for many different types of documents.
Elasticsearch usually deployed together with Kibana, Kibana is a function Elasticsearch powerful data visualization Dashboard, Kibana allows you to browse Elasticsearch log data via a web interface.
Fluentd is a popular open source data collector, will be mounted on Kubernetes Fluentd cluster node, by acquiring the container log files, filtering and conversion log data, then passes the data to the cluster Elasticsearch, it is indexed and stored in the cluster .
Let's start to configure a scalable Elasticsearch cluster, and then create a Kibana application in Kubernetes cluster, and finally run through DaemonSet Fluentd, so that it can run a Pod Kubernetes work on each node.
1. Create a cluster Elasticsearch
Before you create Elasticsearch clusters, we first create a namespace.
Create a kube-efk.yaml
kubectl -f apply for efk.yaml
kubectl get ns see if there is this efk namespace
Here we use three Elasticsearch Pod to avoid the emergence of multi-node cluster "split brain" high availability problem, when one or more nodes will not have a "split brain" when communicating with other nodes, there may be several master node .
A key point is that you should set the parameters discover.zen.minimum_master_nodes = N / 2 + 1, where N is the number of nodes in line with Elasticsearch cluster master node, such as we have here three nodes, meaning that N should be set to 2. Thus, if a node is temporarily disconnected from the cluster, then the other two nodes may select a new primary node, the cluster continues to run and can rejoin at the end of a node attempts, when expanding Elasticsearch cluster, you must remember this parameter.
First create a headless service called elasticsearch, the new file elasticsearch-svc.yaml, document reads as follows:
Service defines a named elasticsearch, specifying label app = elasticsearch, when we Elasticsearch StatefulSet associated with this service, the service will return the DNS A record with the label app = elasticsearch of Elasticsearch Pods, and then set clusterIP = None, the the service set up to service headless. Finally, we define ports 9200,9300, respectively, for interacting with the REST API, as well as for inter-node communication.
Then we create the headless Service
kubectl apply -f elasticsearch-svc.yaml
Now that we have set up a headless services and a stable domain name .elasticsearch.logging.svc.cluster.local as Pod, then we create a specific application Elasticsearch Pod by StatefulSet.
Kubernetes StatefulSet allows us to assign a stable identity and persistent storage for the Pod, Elasticsearch need to ensure stable Pod store data after re-scheduling or restart remains the same, so it is necessary to use StatefulSet to manage Pod.
We used a StorageClass object called es-data-db, so we need to create the object in advance, here we use NFS as the storage back end, so it is necessary to install a corresponding provisioner drive,
we create elasticsearch-storageclass.yaml
Then we create pvc correspond to this storageclass
Elastic search-pvc.yaml
Finally, we create the statefulset
Elastic search-statefulset.yaml
We then use to create kubectl
kubectl apply -f elasticsearch-storageclass.yaml
kubectl apply -f elasticsearch-pvc.yaml
kubectl apply -f elasticsearch-statefulset.yaml
Then we look at the operation of the pod
Pods after the deployment is complete, we can check if Elasticsearch cluster is working properly by requesting a REST API. Use the following command to forward to a local port Elasticsearch node 9200 (e.g., es-cluster-0) corresponding to the port:
Then we open another window
Normally, it should have such information.
See the above information to show that we Elasticsearch cluster called k8s-logs of successfully created three nodes: es-cluster-0, es-cluster-1, and es-cluster-2, the current master node is es-cluster- 0.
2. Create Kibana service
Elasticsearch cluster startup is successful, then we can deploy Kibana service, create a file named kibana.yaml, the corresponding document reads as follows:
上面我们定义了两个资源对象，一个 Service 和 Deployment，为了测试方便，我们将 Service 设置为了 NodePort 类型，Kibana Pod 中配置都比较简单，唯一需要注意的是我们使用 ELASTICSEARCH_URL 这个环境变量来设置Elasticsearch 集群的端点和端口，直接使用 Kubernetes DNS 即可，此端点对应服务名称为 elasticsearch，由于是一个 headless service，所以该域将解析为3个 Elasticsearch Pod 的 IP 地址列表。
然后我们创建这个服务
kubectl apply -f kibana.yaml
过了一会，我们的kibana的服务就起来了。
如果 Pod 已经是 Running 状态了，证明应用已经部署成功了，然后可以通过 NodePort 来访问 Kibana 这个服务，在浏览器中打开http://<任意节点IP>:30245即可，如果看到如下欢迎界面证明 Kibana 已经成功部署到了 Kubernetes集群之中。
3.部署 Fluentd
Fluentd 是一个高效的日志聚合器，是用 Ruby 编写的，并且可以很好地扩展。对于大部分企业来说，Fluentd 足够高效并且消耗的资源相对较少，另外一个工具Fluent-bit更轻量级，占用资源更少，但是插件相对 Fluentd 来说不够丰富，所以整体来说，Fluentd 更加成熟，使用更加广泛，所以我们这里也同样使用 Fluentd 来作为日志收集工具。
工作原理
Fluentd 通过一组给定的数据源抓取日志数据，处理后（转换成结构化的数据格式）将它们转发给其他服务，比如 Elasticsearch、对象存储等等。Fluentd 支持超过300个日志存储和分析服务，所以在这方面是非常灵活的。主要运行步骤如下：
首先 Fluentd 从多个日志源获取数据
结构化并且标记这些数据
然后根据匹配的标签将数据发送到多个目标服务去
日志源配置
比如我们这里为了收集 Kubernetes 节点上的所有容器日志，就需要做如下的日志源配置：
路由配置
上面是日志源的配置，接下来看看如何将日志数据发送到 Elasticsearch：
match：标识一个目标标签，后面是一个匹配日志源的正则表达式，我们这里想要捕获所有的日志并将它们发送给 Elasticsearch，所以需要配置成**。
id：目标的一个唯一标识符。
type：支持的输出插件标识符，我们这里要输出到 Elasticsearch，所以配置成 elasticsearch，这是 Fluentd 的一个内置插件。
log_level：指定要捕获的日志级别，我们这里配置成info，表示任何该级别或者该级别以上（INFO、WARNING、ERROR）的日志都将被路由到 Elsasticsearch。
host/port：定义 Elasticsearch 的地址，也可以配置认证信息，我们的 Elasticsearch 不需要认证，所以这里直接指定 host 和 port 即可。
logstash_format：Elasticsearch 服务对日志数据构建反向索引进行搜索，将 logstash_format 设置为true，Fluentd 将会以 logstash 格式来转发结构化的日志数据。
Buffer： Fluentd 允许在目标不可用时进行缓存，比如，如果网络出现故障或者 Elasticsearch 不可用的时候。缓冲区配置也有助于降低磁盘的 IO。
4.安装
要收集 Kubernetes 集群的日志，直接用 DasemonSet 控制器来部署 Fluentd 应用，这样，它就可以从 Kubernetes 节点上采集日志，确保在集群中的每个节点上始终运行一个 Fluentd 容器。当然可以直接使用 Helm 来进行一键安装，为了能够了解更多实现细节，我们这里还是采用手动方法来进行安装。
First of all, we have to specify the object by ConfigMap Fluentd profile, the new fluentd-configmap.yaml file, as follows:
The above configuration file we configure the log directory and docker docker container to collect kubelet application logs collected data is sent to elasticsearch after treatment: 9200 service.
Then create a fluentd-daemonset.yaml files, reads as follows:
We will fluentd-config ConfigMap created above the target volumes to mount through the Fluentd container can be collected in addition to the log which nodes can be flexibly controlled, so here we also added a nodSelector property:
Also due to the use of our cluster is kubeadm built, master node tainted by default, so in order to log also collects master node, you need to add on tolerance:
Then we create the above objects and daemonset service configmap
We can see the view pod is up and running.
Then we enter kibana page, click discover
Here you can configure Elasticsearch index we need, in front of Fluentd profile logs we collected using logstash format, here only in the text box logstash- * to match all the log data Elasticsearch cluster, and then click Next, go to the following page:
This page is used to configure which field filtering log data by time, in the drop-down list, select @timestamp field, and then click Create index pattern, creation is complete, click on the left navigation menu Discover, then you can see some of the histogram FIGS recently collected and the log data:
