docker: ELK build open source log analysis system

ELK is a log analysis system consists of three parts,

Elasticsearch: Based on analysis json search engine, Elasticsearch is an open source distributed search engine, its features are: distributed, zero-configuration, auto-discovery, auto-index fragmentation,

Index copy mechanism, restful style interfaces, multiple data sources, such as automatic load search.

Logstash: dynamic data collection pipes, Logstash is a fully open source tool that can be collected for your log, analyze, and store it for later use

Kibana: visualizations elasticsearh the data collected by the presentation view. kibana is an open source and free tools that can Logstash and
log analysis provided ElasticSearch friendly Web interface that can help you summarize, analyze and search for important data logs.

First, the use of docker integrated mirror
mounted docker
Elk integrated mirror package name sebp / elk

1. Install docke, start

yum install docke

service docker start

2. Download sebp / elk

docker pull sebp/elk

Can not be downloaded, error:
Unauthorized: authentication required
this is a problem overseas network

NetEase solve a mirror with
vim /etc/docker/daemon.json this json file does not exist, do not worry, directly edit
the following into paste, save, you can restart

{
"registry-mirrors": [ "http://hub-mirror.c.163.com"]
}

# service docker restart

No effect, or get down

2 acceleration to solve with Ali cloud images

Ali register a cloud account, log in
here to copy their accelerated address

then

Install / upgrade your client Docker

Recommended installation 1.10.0or later Docker client, refer to the documentation docker-ce

How to configure Mirror accelerator

Docker for the user of the client version greater than 1.10.0

You can modify the daemon configuration file /etc/docker/daemon.jsonto use the accelerator:

sudo mkdir -p /etc/docker
sudo tee /etc/docker/daemon.json <<-'EOF'
{
  "registry-mirrors": ["https://34sii7qf.mirror.aliyuncs.com"] } EOF sudo systemctl daemon-reload sudo systemctl restart docker

Download:
# Docker pull sebp / Elk
......

617ab16bcfa1: Pull complete
Digest: sha256:b6b8dd20b1a9aaf47b11fe9c66395a462f5e65c50dcff725e9bf83576d4ed241
Status: Downloaded newer image for docker.io/sebp/elk:latest

查看镜像：
[root@bogon docker]# docker images
REPOSITORY TAG IMAGEID CREATED SIZE
docker.io/sebp/elk latest 4b52312ebe8d 12 days ago 1.15 GB

3. boot image

# docker run -p 5601:5601 -p 9200:9200 -p 5044:5044 -it --name elk sebp/elk

Error:

Couln't start Elasticsearch. Exiting.
Elasticsearch log follows below.

原因：

waiting for Elasticsearch to be up (xx/30) counter goes up to 30 and the container exits with Couln't start Elasticsearch. Exiting. and Elasticsearch's logs are dumped, then read the recommendations in the logs and consider that they must be applied.

In particular, in case (1) above, the message max virtual memory areas vm.max_map_count [65530] likely too low, increase to at least [262144]means that the host's limits on mmap counts must be set to at least 262144.

Docker have at least 3GB of allocated memory;

Elasticsearch separate at least 2G memory;

vm.max_map_count need at least 262144, modify vm.max_map_count parameters

solve:

# Vi /etc/sysctl.conf

Add at the end of a line
vm.max_map_count = 262144

View Results
# sysctl -p

vm.max_map_count = 262144

Restart:

# docker run -p 5601:5601 -p 9200:9200 -p 5044:5044 -it --name elk sebp/elk

/usr/bin/docker-current: Error response from daemon: Conflict. The container name "/elk" is already in use by container ece053f704db663e03355a679e25ec732bd08668dae1a87a89567e0e7c950749. You have to remove (or rename) that container to be able to reuse that name..
See '/usr/bin/docker-current run --help'.
[root@bogon ~]# docker run -p 5601:5601 -p 9200:9200 -p 5044:5044 -it --name elk123 sebp/elk
/usr/bin/docker-current: Error response from daemon: Conflict. The container name "/elk123" is already in use by container caf492e3acff7506c5e70b896bd82a33d757816ca3f55911a2aa9aef4cd74670. You have to remove (or rename) that container to be able to reuse that name..
See '/usr/bin/docker-current run --help'.
[root@bogon ~]# docker run -p 5601:5601 -p 9200:9200 -p 5044:5044 -it --name elk1 sebp/elk
* Starting periodic command scheduler cron [ OK ]
* Starting Elasticsearch Server [ OK ]
waiting for Elasticsearch to be up (1/30)
waiting for Elasticsearch to be up (2/30)
waiting for Elasticsearch to be up (3/30)
waiting for Elasticsearch to be up (4/30)
waiting for Elasticsearch to be up (5/30)
waiting for Elasticsearch to be up (6/30)
waiting for Elasticsearch to be up (7/30)
waiting for Elasticsearch to be up (8/30)
waiting for Elasticsearch to be up (9/30)
waiting for Elasticsearch to be up (10/30)
Waiting for Elasticsearch cluster to respond (1/30)
logstash started.
* Starting Kibana5 [ OK ]
==> /var/log/elasticsearch/elasticsearch.log <==
[2018-04-03T06:58:14,192][INFO ][o.e.d.DiscoveryModule ] [JI2Uv6k] using discovery type [zen]
[2018-04-03T06:58:15,132][INFO ][o.e.n.Node ] initialized
[2018-04-03T06:58:15,132][INFO ][o.e.n.Node ] [JI2Uv6k] starting ...
[2018-04-03T06:58:15,358][INFO ][o.e.t.TransportService ] [JI2Uv6k] publish_address {172.17.0.2:9300}, bound_addresses {[::]:9300}
[2018-04-03T06:58:15,378][INFO ][o.e.b.BootstrapChecks ] [JI2Uv6k] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2018-04-03T06:58:18,517][INFO ][o.e.c.s.MasterService ] [JI2Uv6k] zen-disco-elected-as-master ([0] nodes joined), reason: new_master {JI2Uv6k}{JI2Uv6kAQ1ymPZoVhL5AAg}{Rr9Wa_H-RKGaBC4IDpjAMA}{172.17.0.2}{172.17.0.2:9300}
[2018-04-03T06:58:18,529][INFO ][o.e.c.s.ClusterApplierService] [JI2Uv6k] new_master {JI2Uv6k}{JI2Uv6kAQ1ymPZoVhL5AAg}{Rr9Wa_H-RKGaBC4IDpjAMA}{172.17.0.2}{172.17.0.2:9300}, reason: apply cluster state (from master [master {JI2Uv6k}{JI2Uv6kAQ1ymPZoVhL5AAg}{Rr9Wa_H-RKGaBC4IDpjAMA}{172.17.0.2}{172.17.0.2:9300} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])
[2018-04-03T06:58:18,563][INFO ][o.e.h.n.Netty4HttpServerTransport] [JI2Uv6k] publish_address {172.17.0.2:9200}, bound_addresses {[::]:9200}
[2018-04-03T06:58:18,563][INFO ][o.e.n.Node ] [JI2Uv6k] started
[2018-04-03T06:58:18,577][INFO ][o.e.g.GatewayService ] [JI2Uv6k] recovered [0] indices into cluster_state

==> /var/log/logstash/logstash-plain.log <==

==> /var/log/kibana/kibana5.log <==

4. Verify test

Open the browser, type:http://<your-host>:5601，看到如下界面说明安装成功

The relationship between port

5601 (Kibana web interface). Front interface

9200 (Elasticsearch JSON interface) search.

5044 (Logstash Beats interface, receives logs from Beats such as Filebea 日志传输

Test:
Test to deliver a message

1, use the command:docker exec -it <container-name> /bin/bash 进入容器内

2, execute the command:/opt/logstash/bin/logstash -e 'input { stdin { } } output { elasticsearch { hosts => ["localhost"] } }'

报错：如果看到这样的报错信息 Logstash could not be started because there is already another instance using the configured data directory.
If you wish to run multiple instances, you must change the "path.data" setting.

Solution: execute the command: service logstash stop and then execute it.

再次执行：看到：Successfully started Logstash API endpoint {:port=>9600} 即可

3.输入测试信息 :this is a test

4、打开浏览器，输入：http://<your-host>:9200/_search?pretty 如图，就会看到我们刚刚输入的日志内容

ELK 是由三部分组成的一套日志分析系统，

Elasticsearch: 基于json分析搜索引擎,Elasticsearch是个开源分布式搜索引擎，它的特点有：分布式，零配置，自动发现，索引自动分片，

索引副本机制，restful风格接口，多数据源，自动搜索负载等。

Logstash:动态数据收集管道,Logstash是一个完全开源的工具，它可以对你的日志进行收集、分析，并将其存储供以后使用

Kibana:可视化视图,将elasticsearh所收集的data通过视图展现。kibana 是一个开源和免费的工具，它可以为 Logstash 和
ElasticSearch 提供的日志分析友好的 Web 界面，可以帮助您汇总、分析和搜索重要数据日志。

一、使用docker集成镜像
安装docker
elk集成镜像包名字是 sebp/elk

1.安装 docke、启动

yum install docke

service docker start

2.下载 sebp/elk

docker pull sebp/elk

无法下载、报错：
unauthorized: authentication required
这是国外网络的问题

解决1 用网易镜像
vim /etc/docker/daemon.json 这个json文件不存在的，不需要担心，直接编辑
把下面的贴进去，保存，重启即可

{
"registry-mirrors": [ "http://hub-mirror.c.163.com"]
}

# service docker restart

没效果，还是下不来

解决2 用阿里云镜像加速

注册一个阿里云账号，登陆
到这里复制自己的加速地址

然后

安装／升级你的Docker客户端

推荐安装1.10.0以上版本的Docker客户端，参考文档 docker-ce

如何配置镜像加速器

针对Docker客户端版本大于1.10.0的用户

您可以通过修改daemon配置文件/etc/docker/daemon.json来使用加速器：

sudo mkdir -p /etc/docker
sudo tee /etc/docker/daemon.json <<-'EOF'
{
  "registry-mirrors": ["https://34sii7qf.mirror.aliyuncs.com"] } EOF sudo systemctl daemon-reload sudo systemctl restart docker

下载：
# docker pull sebp/elk
……

617ab16bcfa1: Pull complete
Digest: sha256:b6b8dd20b1a9aaf47b11fe9c66395a462f5e65c50dcff725e9bf83576d4ed241
Status: Downloaded newer image for docker.io/sebp/elk:latest

查看镜像：
[root@bogon docker]# docker images
REPOSITORY TAG IMAGEID CREATED SIZE
docker.io/sebp/elk latest 4b52312ebe8d 12 days ago 1.15 GB

3.启动镜像

# docker run -p 5601:5601 -p 9200:9200 -p 5044:5044 -it --name elk sebp/elk

报错：

Couln't start Elasticsearch. Exiting.
Elasticsearch log follows below.

原因：

Docker至少得分配3GB的内存；

Elasticsearch至少需要单独2G的内存；

vm.max_map_count至少需要262144，修改vm.max_map_count 参数

解决：

# vi /etc/sysctl.conf

末尾添加一行
vm.max_map_count=262144

查看结果
# sysctl -p

vm.max_map_count = 262144

重新启动：

# docker run -p 5601:5601 -p 9200:9200 -p 5044:5044 -it --name elk sebp/elk

==> /var/log/logstash/logstash-plain.log <==

==> /var/log/kibana/kibana5.log <==

4.验证、测试

打开浏览器，输入：http://<your-host>:5601，看到如下界面说明安装成功

端口之间关系

5601 (Kibana web interface). 前台界面

9200 (Elasticsearch JSON interface) 搜索.

5044 (Logstash Beats interface, receives logs from Beats such as Filebea 日志传输

测试：
测试传递一条信息

1、使用命令：docker exec -it <container-name> /bin/bash 进入容器内

2、执行命令：/opt/logstash/bin/logstash -e 'input { stdin { } } output { elasticsearch { hosts => ["localhost"] } }'

解决：请执行命令：service logstash stop 然后在执行就可以了。

再次执行：看到：Successfully started Logstash API endpoint {:port=>9600} 即可

3.输入测试信息 :this is a test

4、打开浏览器，输入：http://<your-host>:9200/_search?pretty 如图，就会看到我们刚刚输入的日志内容

ELK 是由三部分组成的一套日志分析系统，

Elasticsearch: 基于json分析搜索引擎,Elasticsearch是个开源分布式搜索引擎，它的特点有：分布式，零配置，自动发现，索引自动分片，

索引副本机制，restful风格接口，多数据源，自动搜索负载等。

Logstash:动态数据收集管道,Logstash是一个完全开源的工具，它可以对你的日志进行收集、分析，并将其存储供以后使用

一、使用docker集成镜像
安装docker
elk集成镜像包名字是 sebp/elk

1.安装 docke、启动

yum install docke

service docker start

2.下载 sebp/elk

docker pull sebp/elk

无法下载、报错：
unauthorized: authentication required
这是国外网络的问题

解决1 用网易镜像
vim /etc/docker/daemon.json 这个json文件不存在的，不需要担心，直接编辑
把下面的贴进去，保存，重启即可

{
"registry-mirrors": [ "http://hub-mirror.c.163.com"]
}

# service docker restart

没效果，还是下不来

解决2 用阿里云镜像加速

注册一个阿里云账号，登陆
到这里复制自己的加速地址

然后

安装／升级你的Docker客户端

推荐安装1.10.0以上版本的Docker客户端，参考文档 docker-ce

如何配置镜像加速器

针对Docker客户端版本大于1.10.0的用户

您可以通过修改daemon配置文件/etc/docker/daemon.json来使用加速器：

sudo mkdir -p /etc/docker
sudo tee /etc/docker/daemon.json <<-'EOF'
{
  "registry-mirrors": ["https://34sii7qf.mirror.aliyuncs.com"] } EOF sudo systemctl daemon-reload sudo systemctl restart docker

下载：
# docker pull sebp/elk
……

617ab16bcfa1: Pull complete
Digest: sha256:b6b8dd20b1a9aaf47b11fe9c66395a462f5e65c50dcff725e9bf83576d4ed241
Status: Downloaded newer image for docker.io/sebp/elk:latest

查看镜像：
[root@bogon docker]# docker images
REPOSITORY TAG IMAGEID CREATED SIZE
docker.io/sebp/elk latest 4b52312ebe8d 12 days ago 1.15 GB

3.启动镜像

# docker run -p 5601:5601 -p 9200:9200 -p 5044:5044 -it --name elk sebp/elk

报错：

Couln't start Elasticsearch. Exiting.
Elasticsearch log follows below.

原因：

Docker至少得分配3GB的内存；

Elasticsearch至少需要单独2G的内存；

vm.max_map_count至少需要262144，修改vm.max_map_count 参数

解决：

# vi /etc/sysctl.conf

末尾添加一行
vm.max_map_count=262144

查看结果
# sysctl -p

vm.max_map_count = 262144

重新启动：

# docker run -p 5601:5601 -p 9200:9200 -p 5044:5044 -it --name elk sebp/elk

==> /var/log/logstash/logstash-plain.log <==

==> /var/log/kibana/kibana5.log <==

4.验证、测试

打开浏览器，输入：http://<your-host>:5601，看到如下界面说明安装成功

端口之间关系

5601 (Kibana web interface). 前台界面

9200 (Elasticsearch JSON interface) 搜索.

5044 (Logstash Beats interface, receives logs from Beats such as Filebea 日志传输

测试：
测试传递一条信息

1、使用命令：docker exec -it <container-name> /bin/bash 进入容器内

2、执行命令：/opt/logstash/bin/logstash -e 'input { stdin { } } output { elasticsearch { hosts => ["localhost"] } }'

解决：请执行命令：service logstash stop 然后在执行就可以了。

再次执行：看到：Successfully started Logstash API endpoint {:port=>9600} 即可

3.输入测试信息 :this is a test

4、打开浏览器，输入：http://<your-host>:9200/_search?pretty 如图，就会看到我们刚刚输入的日志内容

ELK 是由三部分组成的一套日志分析系统，

Elasticsearch: 基于json分析搜索引擎,Elasticsearch是个开源分布式搜索引擎，它的特点有：分布式，零配置，自动发现，索引自动分片，

索引副本机制，restful风格接口，多数据源，自动搜索负载等。

Logstash:动态数据收集管道,Logstash是一个完全开源的工具，它可以对你的日志进行收集、分析，并将其存储供以后使用

一、使用docker集成镜像
安装docker
elk集成镜像包名字是 sebp/elk

1.安装 docke、启动

yum install docke

service docker start

2.下载 sebp/elk

docker pull sebp/elk

无法下载、报错：
unauthorized: authentication required
这是国外网络的问题

解决1 用网易镜像
vim /etc/docker/daemon.json 这个json文件不存在的，不需要担心，直接编辑
把下面的贴进去，保存，重启即可

{
"registry-mirrors": [ "http://hub-mirror.c.163.com"]
}

# service docker restart

没效果，还是下不来

解决2 用阿里云镜像加速

注册一个阿里云账号，登陆
到这里复制自己的加速地址

然后

安装／升级你的Docker客户端

推荐安装1.10.0以上版本的Docker客户端，参考文档 docker-ce

如何配置镜像加速器

针对Docker客户端版本大于1.10.0的用户

您可以通过修改daemon配置文件/etc/docker/daemon.json来使用加速器：

sudo mkdir -p /etc/docker
sudo tee /etc/docker/daemon.json <<-'EOF'
{
  "registry-mirrors": ["https://34sii7qf.mirror.aliyuncs.com"] } EOF sudo systemctl daemon-reload sudo systemctl restart docker

下载：
# docker pull sebp/elk
……

617ab16bcfa1: Pull complete
Digest: sha256:b6b8dd20b1a9aaf47b11fe9c66395a462f5e65c50dcff725e9bf83576d4ed241
Status: Downloaded newer image for docker.io/sebp/elk:latest

查看镜像：
[root@bogon docker]# docker images
REPOSITORY TAG IMAGEID CREATED SIZE
docker.io/sebp/elk latest 4b52312ebe8d 12 days ago 1.15 GB

3.启动镜像

# docker run -p 5601:5601 -p 9200:9200 -p 5044:5044 -it --name elk sebp/elk

报错：

Couln't start Elasticsearch. Exiting.
Elasticsearch log follows below.

原因：

Docker至少得分配3GB的内存；

Elasticsearch至少需要单独2G的内存；

vm.max_map_count至少需要262144，修改vm.max_map_count 参数

解决：

# vi /etc/sysctl.conf

末尾添加一行
vm.max_map_count=262144

查看结果
# sysctl -p

vm.max_map_count = 262144

重新启动：

# docker run -p 5601:5601 -p 9200:9200 -p 5044:5044 -it --name elk sebp/elk

==> /var/log/logstash/logstash-plain.log <==

==> /var/log/kibana/kibana5.log <==

4.验证、测试

打开浏览器，输入：http://<your-host>:5601，看到如下界面说明安装成功

端口之间关系

5601 (Kibana web interface). 前台界面

9200 (Elasticsearch JSON interface) 搜索.

5044 (Logstash Beats interface, receives logs from Beats such as Filebea 日志传输

测试：
测试传递一条信息

1、使用命令：docker exec -it <container-name> /bin/bash 进入容器内

2、执行命令：/opt/logstash/bin/logstash -e 'input { stdin { } } output { elasticsearch { hosts => ["localhost"] } }'

解决：请执行命令：service logstash stop 然后在执行就可以了。

Is performed again: see: Successfully started Logstash API endpoint {: port => 9600} to

3. Enter the test information: this is a test

4, open the browser, type: http: // <your-host>: 9200 / _search pretty figure, you will see the contents of the log we just entered?