docker-consul container service automatic discovery and update

foreword

• Service registration and discovery are indispensable and important components in the microservice architecture. At first, the services were single-node, which did not guarantee high availability and did not consider the pressure bearing of the service. The calls between services were simply accessed through the interface. Until the distributed architecture of multiple nodes appeared later, the initial solution was to balance the load on the service front-end, so that the front-end must know the network locations of all back-end services and configure them in the configuration file. There will be a few questions here:

• If you need to call the backend service AN, you need to configure the network locations of N services, which is very troublesome to configure

• The network location change of the backend service requires changing the configuration of each caller

• Since there are these problems, service registration and discovery are to solve these problems. The back-end service AN can register its current network location to the service discovery module, and the service discovery will be recorded in the form of KV. K is generally the service name, and V is IP:PORT (port). The service discovery module conducts health checks regularly, and polls to see if these back-end services can be accessed. When the front-end calls the back-end service AN, it runs to the service discovery module to ask their network locations, and then calls their services. In this way, the above problems can be solved. The front-end does not need to record the network locations of these back-end services at all, and the front-end and back-end are completely decoupled.

I. Overview

1. What is consul

• consul is a service management software developed by google open source using go language. Supports multi-data centers, distributed high availability, service discovery and configuration sharing. The Raft algorithm is used to ensure high availability of services. Built-in service registration and discovery framework, distributed consistency protocol implementation, health check, Key/Value storage, multi-data center solution, no longer need to rely on other tools (such as ZooKeeper, etc.)
• Service deployment is simple, with only one executable binary package. Each node needs to run agent, which has two modes of operation: server and client. The official recommendation of each data center requires 3 or 5 server nodes to ensure data security and ensure that the election of the server-leader can be performed correctly
• Installing consul is used for service registration, that is, some information of the container itself is registered in consul, and other programs can obtain registered service information through consul, which is service registration and discovery

2. Two modes of consul

• **In client mode,** all services registered to the current node will be forwarded to the server node, which does not persist the information itself
• In the server mode , the function is similar to the client mode, the only difference is that it will persist all the information to the local, so that the information can be retained in case of failure

server-leader is the boss of all server nodes. It is different from other server nodes in that it needs to be responsible for synchronizing registered information to other server nodes, and is also responsible for the health monitoring of each node

Each port explained:

8300: Port for unified synchronization server data information

8301: Port for unified synchronization of client data information (lan port)

8302: Port for communicating with each other across data centers (wan port)

8500: DNS port, used for registered service discovery, port for viewing node information using DNS protocol

8600: http port, used for http interface and web ui access

3. Features of consul

• Service registration and discovery : Consul makes service registration and service discovery easy through the DNS or HTTP interface, and some external services, such as those provided by saas, can also be registered in the same way
• Health check : Health check enables consul to quickly alert the operation in the cluster. Integration with service discovery to prevent service forwarding to failed services
• Key/Value Storage : A system for storing dynamic configurations. Provides a simple HTTP interface that can be operated anywhere
• Multi-datacenter : supports any number of regions without complicated configuration

Installing consul is used for service registration, that is, some information of the container itself is registered in consul, and other programs can obtain the registration-related service information through consul, which is service registration and discovery

Brief description:

1. Supports health checks, allows storage of key-value pairs
2. Based on Golong language, strong portability
3. Support ACL access control

4. The principle of service discovery and update

• Discover:

Consul will install a registrator on the docker server, which will detect the state changes (increase and decrease) of the container on the current host by itself, and can make adjustments by itself. When it finds that the container has changed, it will tell the change information
to Consul's server end and register to consul's server end, after registration, you can view the changes of nodes in the web page on the UI interface

• renew:

The Consul agent controls the consul template template, uses the template component to connect with nginx.conf, and uses the template method to build the template. There are variables in the template, and the reference variable is used to load the backend after consul is registered to the server. The server will tell the agent this information, and the agent will connect with the template and write it into the template. After the update, it will be recognized by the front-end nginx as a sub-configuration file of nginx.conf. Finally, the consul agent will control nginx to reload. After reloading, changes in the upstream pool and proxy_pass information in the nginx.conf configuration file will be recognized, which means that the back-end nodes can be identified, so that the back-end resources can be dynamically adjusted in the address pool

• Brief description:

template template (update)
registrator (automatic discovery)
every time a container is built in the backend, it will register with the registrator, control consul to complete the update operation, and consul will trigger the consul template template to perform hot update
core mechanism: consul: automatic discovery, automatic update, Provide services for containers (add, delete, life cycle)

Two, Consul deployment

server type	system	IP address	Components that need to be installed
consul-server	CentOS7.4(64 bit)	192.168.113.125	Run consul service, nginx service, consul-template daemon process
registrator server	CentOS7.4(64 bit)	192.168.113.127	Run the registrator container and nginx service

#以上两台机器都要关闭防火墙核心防护
systemctl stop firewalld
setenforce 0

#1.建立 consul 服务（192.168.113.125）
[root@consul ~]# mkdir /opt/consul
[root@consul ~]# cd /opt/consul
[root@consul consul]# unzip consul_0.9.2_linux_amd64.zip 
[root@consul consul]# mv consul /usr/local/bin/

#2.设置代理，后台启动 consul 服务器
[root@consul consul]# consul agent \
-server \
-bootstrap \
-ui \
-data-dir=/var/lib/consul-data \
-bind=192.168.113.125 \
-client=0.0.0.0 \
-node=consul-server01 &> /var/log/consul.log &

#查看端口信息是否启动
[root@consul consul]# netstat -natp | grep consul

#查看谁是leader
[root@consul consul]# consul info | grep leader

#查看集群状态
[root@consul consul]# consul operator raft list-peers

#通过 http api 获取集群信息
查看集群server成员		  curl 127.0.0.1:8500/v1/status/peers
集群Raf leader			 curl 127.0.0.1:8500/v1/status/leader
注册的所有服务              curl 127.0.0.1:8500/v1/catalog/services
查看nginx服务信息           curl 127.0.0.1:8500/v1/catalog/nginx
集群节点详细信息            curl 127.0.0.1:8500/v1/catalog/nodes

Description of the above command options

options	illustrate
- start as server	The default is client
-bootstrap	It is used to control whether a server is in bootstrap mode. There can only be one server in bootstrap mode in a data center. When a server is in bootstrap mode, it can be elected as server-leader by itself
-bootstrap-expect=2	The minimum number of servers required by the cluster, when it is lower than this number, the cluster will fail
-ui	Specify to enable the UI interface, so that the web UI interface that comes with consul can be accessed through an address such as http://localhost:8500/ui.
-data-dir	Specify data storage journal
-bind	Specify the communication address used within the cluster. All nodes in the cluster must be reachable to this address. The default is 0.0.0.0
-client	Specifies which client address consul is bound to. This address provides services such as HTTP, DNS, and RPC. The default is 127.0.0.1
-node	Specifies which client address consul is bound to. This address provides services such as HTTP, DNS, and RPC. The default is 127.0.0.1
-datacenter	Specify the data center name, the default is dc1

#1.registrator部署 （192.168.113.127）
#容器服务自动加入Nginx集群
#Gliderlabs/Registrator可检查容器运行状态自动注册，还可注销docker容器的服务到服务配置中心
#目前支持Consul、Etcd和SkyDNS2

#安装Gliderlabs/Registrator
[root@registrator ~]# docker run -d \
--name=registrator \
--net=host \
-v /var/run/docker.sock:/tmp/docker.sock \
--restart=always \
gliderlabs/registrator:latest \
-ip=192.168.113.127 \
consul://192.168.113.125:8500
#下面是参数解释

#查看一下是否安装创建成功
docker ps -a

parameter	illustrate
–net=host	Set the running docker container to host network mode
-v /var/run/docker.sock:/tmp/docker.sock	Mount the Unix domain socket that the host's Docker daemon (Docker daemon) listens to by default into the container
–restart=always	Set to always restart the container when the container exits
-ip	Just now we specified the host mode for the network, so we specified the ip as the host's ip
consul	Specify the IP and port of the consul server

#2.安装Nginx、httpd测试镜像
[root@registrator ~]# docker run -itd -p:20:80 --name test-01 -h test1 nginx
[root@registrator ~]# docker run -itd -p:21:80 --name test-02 -h test2 nginx
[root@registrator ~]# docker run -itd -p:23:80 --name test-03 -h test3 httpd
[root@registrator ~]# docker run -itd -p:24:80 --name test-04 -h test4 httpd

#3.在consul服务器中查看镜像是否注册(192.168.113.125)
[root@consul consul]# curl 127.0.0.1:8500/v1/catalog/services

#4. Check whether these services have been found in the registration center in the web browser

192.168.113.125:8500 (consul server IP address)

3. Configure the template template to automatically update

• Consul-Template is an application that automatically replaces configuration files based on Consul. Consul-Template is a daemon process used to query Consul cluster information in real time, update any number of specified templates on the file system, and generate configuration files. After the update is complete, you can choose to run the shell command to perform the update operation and reload Nginx

• Consul-Template can query the service directory, Key, Key-values, etc. in Consul. This powerful abstraction and query language template can make Consul-Template especially suitable for dynamically creating configuration files. For example: Create Apache/Nginx Proxy Balancers Haproxy Backends, etc.

1. Prepare the template nginx template file

#在consul服务器上进行操作（192.168.113.125）
#1.添加nginx.ctmpl配置文件
[root@consul consul]# pwd 
/opt/consul
[root@consul consul]# vim nginx.ctmpl
#定义nginx upstream 一个简单模板
upstream http_backend {
    
    
  {
    
    {
    
    range service "nginx"}}
   server {
    
    {
    
    .Address}}:{
    
    {
    
    .Port}};
   {
    
    {
    
    end}}
}
#定义一个server,监听8000端口，反向代理到upstream
server {
    
    
 listen 8000;
 server_name localhost 192.168.113.125;
 access_log /var/log/nginx/bxh.com-access.log;  
 index index.html index.php;
 location / {
    
    
        proxy_set_header HOST $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Client-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_pass http://nginx_slb;
  }     
}

#2.传入nginx安装包并解压
root@consul consul]# cd ..   #就是切换到opt目录
[root@consul opt]# rz -E
[root@consul opt]# tar zxvf nginx-1.12.2.tar.gz 

#3.创建nginx程序用户并安装依赖包
[root@consul opt]# useradd -M -s /sbin/nologin nginx
[root@consul opt]# yum -y install gcc pcre-devel zlib-devel gcc-c++ make

#4.编译安装后优化路径
[root@consul opt]# cd nginx-1.12.2/
[root@consul nginx-1.12.2]# ./configure --prefix=/usr/local/nginx --user=nginx --group=nginx && make && make install
[root@consul nginx-1.12.2]# ln -s /usr/local/nginx/sbin/nginx /usr/local/sbin/

#5.修改nginx配置文件
[root@consul nginx-1.12.2]# cd /usr/local/nginx/conf/
[root@consul conf]# vim nginx.conf

#6.创建nginx服务的vhost和日志目录
[root@consul conf]# pwd
/usr/local/nginx/conf
[root@consul conf]# mkdir vhost
[root@consul conf]# mkdir /var/log/nginx
[root@consul conf]# nginx   #启动nginx
[root@consul conf]# netstat -natp | grep nginx

#7.传入consul[root@consul conf]# cd /opt-template_0.19.3_linux_amd64.zip压缩包并解压
[root@consul conf]# cd /opt
[root@consul opt]# unzip consul-template_0.19.3_linux_amd64.zip 
[root@consul opt]# mv consul-template /usr/local/bin/

#8.前台启动consul-template（后台也可以）
[root@consul opt]# consul-template --consul-addr 192.168.113.125:8500 \
--template "/opt/consul/nginx.ctmpl:/usr/local/nginx/conf/vhost/bxh.conf:/usr/local/nginx/sbin/nginx -s reload" \
--log-level=info


#9.新开一个终端查看配置文件是否生成
[root@consul opt]# cd /usr/local/nginx/conf/vhost/
[root@consul vhost]# vim bxh.conf 

#10.在registrator服务器上添加端口为85的nginx容器(192.168.113.127)
[root@registrator ~]# docker run -itd -p:85:80 --name test-05 -h test5 nginx

#11.查看consul服务器运行template服务页面发生了变化(192.168.113.125)

#12.再次查看一下配置文件（可以看到增加一个85端口，使用docker stop停止一个容器的话，配置文件也会相对应改变）

#13.修改容器nginx站点目录中默认的html.index文件(192.168.113.127)
#分别登陆容器添加默认web页面显示
[root@registrator ~]# docker exec -it test-01 bash
root@test1:/# echo "this is test1 web" >> /usr/share/nginx/html/index.html 
root@test1:/# exit 

[root@registrator ~]# docker exec -it test-02 bash
root@test2:/# echo "this is test2 web" >> /usr/share/nginx/html/index.html 
root@test2:/# exit

[root@registrator ~]# docker exec -it test-05 bash
root@test5:/# echo "this is test5 web" >> /usr/share/nginx/html/index.html 
root@test5:/# exit


#14.在浏览器中进行访问测试
192.168.113.125:8000

2. Consul multi-node configuration

#新开一台机器测试是否能 自动发现、更新（192.168.113.128）
#1.先建立 consul 服务
[root@nginx1 ~]# cd /opt
[root@nginx1 opt]# unzip consul_0.9.2_linux_amd64.zip 
[root@nginx1 opt]# mv consul /usr/local/bin/

#2.添加一台已有docker环境的服务器加入到已有的集群中
[root@nginx1 opt]# consul agent \
-server \
--bootstrap \
-ui \
-data-dir=/var/lib/consul-data \
-bind=192.168.113.128 \
-client=0.0.0.0 \
-node=consul-server02 \
-enable-script-checks=true \
-datacenter=dc1 \
-join 192.168.113.125 &> /var/log/consul.log &

parameter	illustrate
–enable-script-ckecks=true	Set check service to available
-datacenter	data center name
-join	Join an existing cluster

#3.在consul服务器上查看
[root@consul opt]# consul members

Browser access can also see 192.168.113.128:8500