Docker + Consul + registrator achieve service discovery and nginx reverse proxy

A. Architecture design

In reality, we have been eager to pursue high-quality, high-availability architecture service system, while reducing unnecessary deployment and maintenance costs, reduce fault rate. Faced with such high demand, there are two architectural approach:
Docker + Etcd + Confd + Nginx
Docker Consul + + Nginx
this paper, we mainly introduce Docker + Etcd + Confd + Nginx scheme that is more efficient, quick, and maintenance lower costs and fault tolerance, support more distributed, as shown below:

A schematic view of above about process is as follows:
1, the docker01 host in a manner binary package deployment consul services and running in the background, its identity is Leader;
2, docker02, docker03 run consul and services in the manner of the container, and added to a consul cluster docker01 concentrated ;
3, the host docker02, the background registrator docker03 container to automatically discover services provided by the docker container;
4, deployed on Nginx docker01, reverse proxy provides services based on the image Nginx docker02, docker03 host, each of two runs a web container, providing different page document, so that the test results;
5, mounted on the consul-template command docker01, information is collected (gathered information container registrator) is written in the template template, and finally writes the Nginx configuration file.
6. At this point, the client achieved by accessing Nginx reverse proxy server (docker01), get docker02, web file server running on Nginx docker03 provided by the container.
Note: registrator is an automatic service discovery docker container provided, and registration services in the back-end service registry (data center). The main container used to collect information to run the service, and sent to the consul. In addition to the data center consul, there etcd, zookeeper like.

II. Advantage Architecture

Docker + Consul + Nginx although the look is the use of three components, but it proved to be an organic whole. They are interrelated and interact, to fully meet our requirements for high availability, and efficient service architecture program, is one of the most ideal combination Docker ecosystem has the following advantages :

1. Discover and register the component consul Raft use algorithms to ensure consistency, more direct than the complex Paxos algorithm. In comparison, the Paxos ZooKeeper uses, and is used etcd Raft;
2. multiple data centers, data center cluster plurality avoid single points of failure of a single data center, and etcd ZooKeeper not provide multiple data center functions support;
3, real-time discovery and no refresh aware services, availability of resources, elasticity, flexibility;
4. health check, dynamic load can be balanced across the available service instances, etcd not provide this functionality;
5. more than enough Docker containers (premise infrastructure resources are sufficient to ensure the performance of support);
6.http and dns protocol interface, zookeeper integration is more complex, etcd only support http protocol;
facilitate quick adjustment 7. scale, the official web management interface, etcd no such function;
8 .nsul template with consul used to support multiple access layer, such as Nginx, Haproxy.

III. Experimental environment

Host computer	iP address	service
docker01	192.168.1.11	consul+consul-template+nginx
docker02	192.168.1.13	consul + registrar
docker03	192.168.1.20	consul + registrar

Three hosts turn off the firewall, disable selinux, change the host name as described above.

Deployment of the consul Service

(1) docker01 Quguan network https://www.consul.io/downloads.html download consul Service

[root@docker01 ~]# unzip consul_1.5.1_linux_amd64.zip 
//现在是本地导入压缩包，需要解压              
[root@docker01 ~]# mv consul /usr/local/bin/
//移动服务到bin目录
[root@docker01 ~]# chmod +x /usr/local/bin/consul
//给予一个可执行权限

(2) Start consul

[root@docker01 ~]# consul agent -server -bootstrap -ui -data-dir=/var/lib/consul-data -bind=192.168.1.11 -client=0.0.0.0 -node=master

PS: // - bootstrap: If you use this option, generally used in a single server node when the self-elected as leader.

Parameter Description:
-server: add a service
-bootstrap: general use in a single server node when the self-elected as leader.
-data-dir: key / volume data stored in the specified directory
-ui: turn on the internal web interface
-bind: Specifies the ip open service
-client: the client to specify access
-node: in the name of internal communication cluster, by default CPU name.

Now this is for external use ip

PS: open port
8300 cluster nodes
inside the cluster 8301 access
communication across data centers 8302
8500 web ui interface
8600 using dns protocol to view node information of port

Refer to mean the view in FIG port:

At this time, this command will start consul occupies the terminal, you can use the nohup command to keep it running in the background.

[root@docker01 ~]# nohup consul agent -server -bootstrap -ui -data-dir=/var/lib/consule-data -bind=192.168.1.11  -client=0.0.0.0 -node=master &

(3) View consul port information **

[root@docker01 ~]# consul info

Five. Docker01 download deployment consul-template

In https://github.com/hashicorp/consul-template , the consul-template download

[root@docker01 ~]# unzip consul-template_0.19.5_linux_amd64.zip
//解压安装好的consul-template包
[root@docker01 ~]# mv consul-template /usr/local/bin/
//移动到命令目录
[root@docker01 ~]# chmod +x /usr/local/bin/consul-template 
//给予一个可执行权限

Sixth and seventh steps outline

And operates on docker01 docker02

First look at the docker thinking about server operations:

分别在两台docker服务器上都创建registrator容器，注意到consul服务中心；
在docker01上运行两台nginx容器（端口随机生成），在docker02上运行两台nginx容器（端口随机生成）；
修改这4台nginx容器中的index.html页面内容为（xgp-web01、xgp-web02、xgp-web03、xgp-web04）
访问consul web界面验证
访问nginx服务器地址 http://192.168.1.11:8000 进行验证；

六. docker02，docker03，加入consul集群

这里我们采用容器的方式去运行consul服务。

（1）下载consu所需的l镜像

[root@docker02 ~]# docker pull consul

（2）基于consul镜像开启一台容器

[root@docker02 ~]# docker run -d --name consul -p 8301:8301 -p 8301:8301/udp -p 8500:8500 -p 8600:8600 -p 8600:8600/udp --restart always progrium/consul -join 192.168.1.11 -advertise 192.168.1.13 -client 0.0.0.0 -node=node01

参数解释：

-d：守护进程

--name：容器名称

--restart：容器随着docker服务一直运行

-advertise:声明本机地址

-join:声明服务端地址

-node:consul集群中的名称

（3）docker查看consul集群成员的信息

[root@docker01 ~]# consul members

（4）两台docker开启容器后，docker01查看

（5）浏览器访问http://192.168.1.11:8500

七. docker02、docker03 上部署registrator服务

registrator是一个能自动发现docker container提供的服务,并在后端服务注册中心注册服务或取消服务的工具，后端注册中心支持conusl、etcd、 skydns2、zookeeper等。

（1）下载registrator镜像

[root@docker02 ~]# docker pull registrator
//下载registrator镜像

（2）基于registrator镜像，开启一台容器

[root@docker02 ~]# docker run -d  --name registrator -v /var/run/docker.sock:/tmp/docker.sock  --restart always gliderlabs/registrator consul://192.168.1.13:8500

参数说明：

--network：把运行的docker容器设定为host网络模式；
-v /var/run/docker.sock：把宿主机的Docker守护进程(Docker daemon)默认监听的Unix域套接字挂载到容器中；
--ip : 刚才把network指定了host模式，所以我们指定下IP为宿主机的IP；
consul:j最后这个选项是配置consul服务器的IP和端口。

（3）开启一台nginx容器

[root@docker02 ~]# docker run -d —P --name nginx nginx:latest 

（4）浏览器查看一下http://192.168.1.11:8500/ui/dc1/nodes

八.docker01部署一个nginx服务

配置nginx，大概配置的思路为：

在/usr/local/nginx/conf中创建目录consul，目录名自定义；
在consul目录中创建nginx.ctmpl模板；
在nginx.conf配置中添加include项并指向consul目录 ；
重启nginx服务；

（1）安装开启nginx服务

安装nginx依赖包

[root@docker01 ~]#  yum -y install pcre pcre-devel openssl openssl-devel zlib zlib-devel

编译安装nginx

[root@docker01 ~]# cd nginx-1.14.0/
[root@docker01 nginx-1.14.0]# ./configure --user=nginx --group=nginx  --with-http_stub_status_module --with-http_realip_module --with-pcre --with-http_ssl_module && make && make install

创建所需用户和链接命令目录

[root@docker01 nginx-1.14.0]# useradd -M -s /sbin/nologin nginx
[root@docker01 nginx-1.14.0]# ln -s /usr/local/nginx/sbin/* /usr/local/bin/

检查nginx是否有问题，并开启nginx

[root@docker01 nginx-1.14.0]# nginx -t
nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful
[root@docker01 nginx-1.14.0]# nginx

PS:这里nginx作为反向代理，代理后端docker02、 docker03 上nginx的容器服务,所以我们先去docker02、docker03. 上部署一些服务， 为了方便等会看到负载的效果，所以，我们运行完成容器之后，做一个主界面内容的区分。

（2）安装完成之后，本机测试访问

[root@docker01 nginx-1.14.0]# curl 127.0.0.1

（3）部署环境

主机	服务
docker02	nginx	web01，web02
docker03	nginx	web03，web04

<1>下载nginx镜像（docker02，docker03都要）

[root@docker02 ~]# docker pull nginx
//下载nginx镜像

<2>docker01操作

基于nginx镜像运行上述所说的容器并设置测试页面

web01
[root@docker02 ~]# docker run  -itd --name web01 -P nginx:latest
[root@docker02 ~]# docker exec -it web01 /bin/bash
root@44b59d07202f:/# cd /usr/share/nginx/html/
root@44b59d07202f:/usr/share/nginx/html# echo web01 > index.html

web02
[root@docker02 ~]# docker run  -itd --name web02 -P nginx:latest
[root@docker02 ~]# docker exec -it web02 /bin/bash
root@44b59d07202f:/# cd /usr/share/nginx/html/
root@44b59d07202f:/usr/share/nginx/html# echo web02 > index.html

<3>docker02操作

基于nginx镜像运行上述所说的容器并设置测试页面

web03
[root@docker03 ~]#  docker run  -itd --name web03 -P nginx:latest
[root@docker03 ~]#  docker exec -it web03 /bin/bash
root@fd8e8b2df136:/# cd /usr/share/nginx/html/
root@fd8e8b2df136:/usr/share/nginx/html#  echo web03 > index.html
root@fd8e8b2df136:/usr/share/nginx/html# exit

web04
[root@docker03 ~]#  docker run  -itd --name web04 -P nginx:latest
[root@docker03 ~]#  docker exec -it web04 /bin/bash
root@fd8e8b2df136:/# cd /usr/share/nginx/html/
root@fd8e8b2df136:/usr/share/nginx/html#  echo web04 > index.html
root@fd8e8b2df136:/usr/share/nginx/html# exit

（4）docker01更改nginx配置文件

[root@docker01 ~]# cd /usr/local/nginx/
//进入nginx配置文件目录
[root@docker01 nginx]# mkdir consul
//创建consul目录
[root@docker01 nginx]# cd consul/
//进入consul目录

<1>创建nginx.ctmpl模板

[root@docker01 consul]# vim nginx.ctmpl
upstream http_backend {
        {{range service "nginx"}}
        server {{ .Address }}:{{ .Port }};
        {{ end }}
}

server {
        listen 8000;
        server_name localhost;
        location / {
                proxy_pass http://http_backend;
        }
}

nginx.ctmpl模板中的内容就是两段意思，熟悉nginx的朋友一看也能明白：第1 定义nginx upstream一个简单模板，第2 定义一个server，监听8000端口，反向代理到upstream。

<2>修改nginx配置文件，通过 include 参数包含刚刚创建的文件

[root@docker01 consul]# cd /usr/local/nginx/conf/
[root@docker01 conf]# vim nginx.conf
  include /usr/local/nginx/consul/*.conf;  #文件最后添加（要在大括号里面）

<3> 生成一个vhost.conf配置文件，并重启nginx（会占用终端)

使用consul-template命令，根据模板生产新的配置文件，并重新加载nginx的配置文件。

[root@docker01 conf]# consul-template -consul-addr 192.168.1.11:8500 -template "/usr/local/nginx/consul/nginx.ctmpl:/usr/local/nginx/consul/vhost.conf:/usr/local/bin/nginx -s reload"

参数说明：

--consul-addr：指定consul服务的ip和端口；
./nginx.ctmpl：这是用nginx.ctmpl这个模板来启动进程，这是写的相对路径，也可以写绝对路径；
vhost.conf：nginx.ctmpl模板生成后的文件名，这也可以写绝对路径，如果不写绝对路径，这个文件就在当前目录生成（/usr/local/nginx/consul/）

这时，这条命令会占用终端，可以使用nohup命令让它保持后台运行,并重启nginx服务。

[root@docker01 conf]# nohup consul-template -consul-addr 192.168.1.11:8500 -template   "/usr/local/nginx/consul/nginx.ctmpl:/usr/local/nginx/consul/vhost.conf:/usr/local/sbin/nginx -s reload"  &

查看一下文件是否生成，里面是否有内容

[root@docker01 ~]# cd /usr/local/nginx/consul/
[root@docker01 consul]# ls
nginx.ctmpl  vhost.conf

[root@docker01 consul]# cat vhost.conf 

此时，应该能够看到，新生产的vhost.conf配置文件已经生效，访问本机8000端口可以得到不同容器提供的服务。

<4>测试访问

[root@docker01 consul]# curl 127.0.0.1:8000
web01

此时可以看到负载均衡的效果！

<5>如果访问不成功

查看端口8000是否开启

[root@docker01 consul]# ss -lnt

检查nginx配置文件

[root@docker01 consul]# nginx -t
nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
nginx: configuration file /usr/local/nginx/conf/nginx.conf test is successful

检查自己编写的nginx配置文件

[root@docker01 consul]# cd /usr/local/nginx/consul/
[root@docker01 consul]# cat nginx.ctmpl 
upstream http_backend {
    {{range service "nginx"}}
    server {{ .Address }}:{{ .Port }};
    {{ end }}
}

server {
    listen 8000;
    server_name localhost;
    location / {
        proxy_pass http://http_backend;
    }
}

如果nginx配置文件没问题，重启nginx

[root@docker01 consul]# nginx -s reload

<6>测试自动发现

docker02 创建测试容器

[root@docker02 ~]# docker run  -itd --name web05 -P nginx:latest
[root@docker02 ~]# docker exec -it web05 /bin/bash
root@44b59d07202f:/# cd /usr/share/nginx/html/
root@44b59d07202f:/usr/share/nginx/html# echo web02 > index.html
[root@docker02 ~]# docker ps 

docker01查看

[root@docker01 consul]# cd /usr/local/nginx/consul/
[root@docker01 consul]# cat vhost.conf 

docker01测试访问

[root@docker01 consul]# curl 127.0.0.1:8000

//同上

此时可以看到负载均衡的效果！

At this time no need to consider the back-end web servers are added or deleted automatically updated, because the consul-template after you run this command to add / usr / local / sbin / nginx -s reload role!