confd+etcd

转自http://www.361way.com/confd-etcd/5470.html
Confd是一个轻量级的配置管理工具。通过查询Etcd，结合配置模板引擎，保持本地配置最新，同时具备定期探测机制，配置变更自动reload。其后端支持的数据类型有：etcd、consul、vault、environment variables、redis、zookeeper、dynamodb、stackengine、rancher。不过一般使用Confd和etcd的配合使用比较多。其常用架构如下：

etcd-confd-nginx
一、简单配置
1、配置etcd数据

具体步骤这里略过，这里只配置两条数据

etcdctl set /myapp/database/url www.361way.com
etcdctl set /myapp/database/user rob

2、confd安装

confd比较简单就一个文件，拿过来就可以执行，可以从github上下载：https://github.com/kelseyhightower/confd/releases ，并将其放到/usr/local/bin目录下即可。不过使用前需要创建相应的配置目录：

[root@etcd1 bin]# mkdir -p /etc/confd/{conf.d,templates}

3、创建confd配置文件

# vim  /etc/confd/conf.d/myconfig.toml
[template]
src = "myconfig.conf.tmpl"
dest = "/tmp/myconfig.conf"
keys = [
    "/myapp/database/url",
    "/myapp/database/user",
]

4、创建模板文件

# vim /etc/confd/templates/myconfig.conf.tmpl
[myconfig]
database_url = {{getv "/myapp/database/url"}}
database_user = {{getv "/myapp/database/user"}}

5、执行生成配置文件

confd -onetime -backend etcd -node http://127.0.0.1:2379    只一次
confd -interval=60 -backend etcd -node http://127.0.0.1:2379 &   按时间轮询

使用onetime参数的，配置文件生成一次后，confd程序就退出了，下面的那句，会每隔60秒轮询一次。一旦后端etcd相应的值发生变化就会重新生成相应的配置文件。
6、验证文件生成

[root@etcd1 tmp]# cat /tmp/myconfig.conf
[myconfig]
database_url = www.361way.com
database_user = rob

二、etcd+confd+nginx配置
1、创建数据

etcdctl set /myapp/subdomain myapp
etcdctl set /myapp/upstream/app2 "10.0.1.100:80"
etcdctl set /myapp/upstream/app1 "10.0.1.101:80"
etcdctl set /yourapp/subdomain yourapp
etcdctl set /yourapp/upstream/app2 "10.0.1.102:80"
etcdctl set /yourapp/upstream/app1 "10.0.1.103:80"

2、创建配置文件

# cat /etc/confd/conf.d/myapp-nginx.toml
[template]
prefix = "/myapp"
src = "nginx.tmpl"
dest = "/tmp/myapp.conf"
owner = "nginx"
mode = "0644"
keys = [
  "/subdomain",
  "/upstream",
]
check_cmd = "/usr/sbin/nginx -t -c {{.src}}"
reload_cmd = "/usr/sbin/service nginx reload"
# cat /etc/confd/conf.d/yourapp-nginx.toml
[template]
prefix = "/yourapp"
src = "nginx.tmpl"
dest = "/tmp/yourapp.conf"
owner = "nginx"
mode = "0644"
keys = [
  "/subdomain",
  "/upstream",
]
check_cmd = "/usr/sbin/nginx -t -c {{.src}}"
reload_cmd = "/usr/sbin/service nginx reload"

这里创建了两个配置文件。接下来创建一个模板文件，两个配置文件会根据该模板文件生成配置：

# cat /etc/confd/templates/nginx.tmpl
upstream {{getv "/subdomain"}} {
{{range getvs "/upstream/*"}}
    server {{.}};
{{end}}
}
server {
    server_name  {{getv "/subdomain"}}.example.com;
    location / {
        proxy_pass        http://{{getv "/subdomain"}};
        proxy_redirect    off;
        proxy_set_header  Host             $host;
        proxy_set_header  X-Real-IP        $remote_addr;
        proxy_set_header  X-Forwarded-For  $proxy_add_x_forwarded_for;
   }
}

3、验证

在进行验证启用的时候，会发现有如下报错：

[root@etcd1 conf.d]# 2017-05-08T19:06:07+08:00 etcd1 confd[10949]: INFO Target config /tmp/myapp.conf out of sync
2017-05-08T19:06:07+08:00 etcd1 confd[10949]: ERROR "nginx: [emerg] \"upstream\" directive is not allowed here in /tmp/.myapp.conf835093196:1\nnginx: configuration file /tmp/.myapp.conf835093196 test failed\n"
2017-05-08T19:06:07+08:00 etcd1 confd[10949]: ERROR Config check failed: exit status 1
2017-05-08T19:06:07+08:00 etcd1 confd[10949]: INFO Target config /tmp/yourapp.conf out of sync
2017-05-08T19:06:07+08:00 etcd1 confd[10949]: ERROR "nginx: [emerg] \"upstream\" directive is not allowed here in /tmp/.yourapp.conf196880350:1\nnginx: configuration file /tmp/.yourapp.conf196880350 test failed\n"
2017-05-08T19:06:07+08:00 etcd1 confd[10949]: ERROR Config check failed: exit status 1

原因很简单，注意配置文件中的check_cmd 命令，该命令会进行配置文件检测，检测不通过时，配置文件不会修改，且不会执行后面的reload_cmd命令。这里想不报错也很简单，将配置文件中nginx的配置指向正确的位置，而且让nginx可以正常检测，且检测结果没有错误。再次执行命令，并修改配置文件，会发现有如下信息：

# cat myapp.conf
upstream myapp {
    server 10.0.1.100:80;
    server 10.0.1.101:80;
}
server {
    server_name  myapp.example.com;
    location / {
        proxy_pass        http://myapp;
        proxy_redirect    off;
        proxy_set_header  Host             $host;
        proxy_set_header  X-Real-IP        $remote_addr;
        proxy_set_header  X-Forwarded-For  $proxy_add_x_forwarded_for;
   }
}

三、其他

在模板文件中，会用到一些函数：map、base、exists、get、gets、getenv、datetime等 ，关于这些函数的使用，可以参看官方文档templates 。

etcd+confd 基本上可以和任何应用进行组合，如网上比较常见的etcd+confd+nginx 、etcd+confd+haproxy 、etcd+confd+k8s、etcd+confd+tomcat等等。和tomcat的整合，在官方上也有相应的示例，具体可以参看官方文档tomcat-sample 。

[root@k8s-master-1 confd-dns]# cat /root/github/confd-dns/confd.toml
backend = “etcd”
client_cakeys = “/etc/etcd/ssl/etcd-ca.pem”
client_cert = “/etc/etcd/ssl/etcd.pem”
client_key = “/etc/etcd/ssl/etcd-key.pem”
confdir = “/root/github/confd-dns/etc/confd”
log-level = “debug”
interval = 600
nodes = [
“https://109.105.1.253:2379”,
“https://109.105.1.254:2379”,
“https://109.105.1.208:2379”,
“https://109.105.1.176:2379”,
“https://109.105.1.246:2379”,
]
noop = false
prefix = “/confd-dns”
scheme = “https”
watch = true
[root@k8s-master-1 confd-dns]# cat etc/confd/conf.d/gcloud.srcb.com.zone.toml
[template]
src = “gcloud.srcb.com.zone.tmpl”
dest = “/var/named/gcloud.srcb.com.zone”
keys = [
“/gpu-infra/dns/serial”,
“/gpu-infra/dns/names”,
“/gpu-infra/dns/nodes”,
]
check_cmd = “chown named:named {{.src}} && named-checkzone gcloud.srcb.com.zone {{.src}}”
reload_cmd = “rndc reload”

[root@k8s-master-1 confd-dns]# cat etc/confd/templates/gcloud.srcb.com.zone.tmpl
$TTL 600
@ IN SOA @ rname.invalid. (
{{getv “/gpu-infra/dns/serial”}} ; serial
1D ; refresh
1H ; retry
1W ; expire
3H ) ; minimum
NS @
A 109.105.1.253
A 109.105.1.254
A 109.105.1.246
AAAA ::1

     IN   NS  ns1
     IN   NS  ns2

ns1 IN A 109.105.1.253
ns2 IN A 109.105.1.254

{{ $nodes := jsonArray (getv “/gpu-infra/dns/nodes”) }}

{{range jsonArray (getv “/gpu-infra/dns/names”) }}
{{.}}{{range $nodes}} IN A {{.}}
{{end}}{{end}}

在etcd几点执行：
etcdctl set /confd-dns/gpu-infra/dns/nodes [“109.105.1.253”,“109.105.1.254”,“109.105.1.246”,“109.105.1.208”,“109.105.1.180”,“109.105.1.176”]