目录：
第一节 多节点OpenStack Charms 部署指南0.0.1.dev223–1--OpenStack Charms 部署指南
第二节 多节点OpenStack Charms 部署指南0.0.1.dev223–2-安装MAAS
第三节 多节点OpenStack Charms 部署指南0.0.1.dev223–3-安装Juju
第四节 多节点OpenStack Charms 部署指南0.0.1.dev223–4-安装openstack
第五节 多节点OpenStack Charms 部署指南0.0.1.dev223–5--使bundle安装openstack
第六节 多节点OpenStack Charms 部署指南0.0.1.dev223–6--配置vault和设置数字证书生命周期
第七节 多节点OpenStack Charms 部署指南0.0.1.dev223–7--juju 离线部署bundle
第八节 多节点OpenStack Charms 部署指南0.0.1.dev223–8--配置 OpenStack
第九节 多节点OpenStack Charms 部署指南0.0.1.dev223–9--网络拓扑
第十节 多节点OpenStack Charms 部署指南0.0.1.dev223–10–OpenStack 高可用基础架构实际
第十一节 多节点OpenStack Charms 部署指南0.0.1.dev223–11–访问Juju仪表板
参考文档：
《Specific series upgrade procedures-percona-cluster charm: series upgrade to Focal》
《OpenStack Charms Deployment Guide0.0.1dev276》
《ReleaseNotes1501》
[BUG] openstack hacluster apache2 service not running, wrong ssl cert name
背景说明：本文是在bundle openstack-base #70部署完毕后，手动基础设施HA化的步骤。
根据根据《多节点OpenStack Charms 部署指南0.0.1.dev223–附录T–OpenStack 高可用性》中，HA可以分为两类，一类是原生HA，一类是非原生HA。
原生HA包括：
| 服务 | 应用/Charm | 备注 |
|---|---|---|
| Ceph | ceph-mon, ceph-osd | |
| MySQL | percona-cluster | MySQL 5. x; 客户端访问所需的外部 高可用技术; 可在 Ubuntu 20.04 LTS 之前使用 |
| MySQL | mysql-innodb-cluster | MySQL 8. x; 从 Ubuntu 20.04 LTS 开始使用 |
| OVN | ovn-central, ovn-chassis | OVN是高可用的设计，可以应用在 OpenStack Ussuri 上，从 Ubuntu 18.04 LTS 和 Ubuntu 20.04 LTS 开始使用 |
| RabbitMQ | rabbitmq-server | |
| Swift | swift-storage |
部署rabbitmq server 集群：
原文中，rabbitmq-server高可用的命令为：
juju deploy -n 3 --to lxd,lxd,lxd --config min-cluster-size=3 rabbitmq-server
本文变更为:
juju add-unit --to lxd:0 rabbitmq-server
juju add-unit --to lxd:1 rabbitmq-server
其他非原生HA为：
三个单元组成的集群的通用部署命令。
juju deploy -n 3 --config vip=<ip-address> <charm-name>
juju deploy --config cluster_count=3 hacluster <charm-name>-hacluster
juju add-relation <charm-name>-hacluster:ha <charm-name>:ha
部署keystone集群:
keystone 高可用配置方法为：
juju deploy -n 3 --to lxd:0,lxd:1,lxd:2 --config vip=10.0.7.12 keystone
juju deploy --config cluster_count=3 hacluster keystone-hacluster
juju add-relation keystone-hacluster:ha keystone:ha
由于bundle openstack-base-70中keystone 已经安装，所以以上命令失败。
看了文档，意思可以如下添加unit扩展，部署keystone集群：
juju add-unit --to lxd:1 keystone
juju add-unit --to lxd:2 keystone
juju set keystone vip=10.0.7.12
juju deploy --config cluster_count=3 --series focal hacluster keystone-hacluster
juju add-relation keystone-hacluster:ha keystone:ha
juju set 命令juju版本2.8已经不支持
发现命令变更为
juju add-unit --to lxd:1 keystone
juju add-unit --to lxd:2 keystone
juju config keystone vip=10.0.7.12
juju deploy --config cluster_count=3 --series focal hacluster keystone-hacluster
juju add-relation keystone-hacluster:ha keystone:ha
#重建keystone集群,不建议，会hook fail：
#juju remove-unit keystone/0 --force --no-wait
#juju remove-applicationg keystone --force --no-wait
#juju deploy -n 3 --to lxd:0,lxd:1,lxd:2 --config vip=10.0.7.13 --series focal ./openstack-base-1/keystone --debug
#juju deploy --config cluster_count=3 hacluster keystone-hacluster
#juju add-relation keystone-hacluster:ha keystone:ha
部署vault 集群：
在原文《OpenStack Charms Deployment Guide-0.0.1dev276-Infrastructure high availability》中的方法为：
Havault 部署除了 hacluster 和 MySQL 之外，还需要 etcd 和 easyrsa 应用程序。此外，集群中的每个vault单元都必须有自己的未密封的vault实例。
在这些示例命令中，为了简单起见，使用了单个 percona-cluster 单元
juju deploy --to lxd:1 percona-cluster mysql
juju deploy -n 3 --to lxd:0,lxd:1,lxd:2 --config vip=10.246.114.11 vault
juju deploy --config cluster_count=3 hacluster vault-hacluster
juju deploy -n 3 --to lxd:0,lxd:1,lxd:2 etcd
juju deploy --to lxd:0 cs:~containers/easyrsa
juju add-relation vault:ha vault-hacluster:ha
juju add-relation vault:shared-db percona-cluster:shared-db
juju add-relation etcd:db vault:etcd
juju add-relation etcd:certificates easyrsa:client
但是，由于在openstack-base中，数据库使用的是 mysql-innodb-cluster，且已经集群化了，因为在focal中，percona-cluster已经被mysql-innodb-cluster取代。
所以juju命令根据实际情况，应该有以下变更：
#juju remove-unit vault/0 --force --no-wait
#juju remove-application vault --force --no-wait
#juju deploy -n 3 --to lxd:0,lxd:1,lxd:2 --config vip=10.0.7.22 --series focal vault --debug
juju add-unit --to lxd:1 vault
juju add-unit --to lxd:2 vault
juju config vault vip=10.0.7.21
juju deploy --config cluster_count=3 --series focal hacluster vault-hacluster
juju add-relation vault:ha vault-hacluster:ha
在vault HA之前的截图：
分别对三个vault单元解封：
解封vault/0：
export VAULT_ADDR="http://10.0.1.248:8200"
vault operator init -key-shares=5 -key-threshold=3
vault operator unseal FyoFAkE7rlqfVSnDwm4943tYAwx51UfSntW73rQdK7SX
vault operator unseal sj38M2qmnOAegNijJ1XYtxer17rGqtrJP7OPCeG8Tq1Q
vault operator unseal /s5IYKaUo4u4vvkP6fUEDwxtHjHdtek6HIgQ+GQ4okaG
export VAULT_TOKEN=s.YpBOElRdghjenojFo4YrXNPe
vault token create -ttl=720h
juju run-action --wait vault/leader authorize-charm token=s.ajIKkgKxDjy28EqiRqZWgkS5
juju run-action --wait vault/leader 'generate-root-ca'
查看vault状态：
juju status vault
Model Controller Cloud/Region Version SLA Timestamp
openstack maas-controller mymaas/default 2.8.7 unsupported 14:42:06+08:00
App Version Status Scale Charm Store Rev OS Notes
vault 1.5.4 blocked 3 vault local 0 ubuntu
vault-hacluster active 3 hacluster jujucharms 72 ubuntu
vault-mysql-router 8.0.23 active 3 mysql-router local 0 ubuntu
Unit Workload Agent Machine Public address Ports Message
vault/0* active idle 0/lxd/7 10.0.1.248 8200/tcp Unit is ready (active: true, mlock: disabled)
vault-hacluster/0* active idle 10.0.1.248 Unit is ready and clustered
vault-mysql-router/0* active idle 10.0.1.248 Unit is ready
vault/1 blocked idle 1/lxd/8 10.0.2.12 8200/tcp Unit is sealed
vault-hacluster/1 active idle 10.0.2.12 Unit is ready and clustered
vault-mysql-router/1 active idle 10.0.2.12 Unit is ready
vault/2 blocked idle 2/lxd/7 10.0.2.11 8200/tcp Unit is sealed
vault-hacluster/2 active idle 10.0.2.11 Unit is ready and clustered
vault-mysql-router/2 active idle 10.0.2.11 Unit is ready
Machine State DNS Inst id Series AZ Message
0 started 10.0.0.159 node4 focal default Deployed
0/lxd/7 started 10.0.1.248 juju-2c0e84-0-lxd-7 focal default Container started
1 started 10.0.0.156 node2 focal default Deployed
1/lxd/8 started 10.0.2.12 juju-2c0e84-1-lxd-8 focal default Container started
2 started 10.0.0.157 node1 focal default Deployed
2/lxd/7 started 10.0.2.11 juju-2c0e84-2-lxd-7 focal default Container started
juju run-action vault/0 pause --wait #可以不执行
juju status vault
接着解封vault/1:
export VAULT_ADDR="http://10.0.2.12:8200"
vault operator unseal FyoFAkE7rlqfVSnDwm4943tYAwx51UfSntW73rQdK7SX
vault operator unseal sj38M2qmnOAegNijJ1XYtxer17rGqtrJP7OPCeG8Tq1Q
vault operator unseal /s5IYKaUo4u4vvkP6fUEDwxtHjHdtek6HIgQ+GQ4okaG
juju status vault
juju status --format=yaml vault | grep public-address | awk '{print $2}'
juju run-action vault/0 resume --wait
再解封vault/2:
export VAULT_ADDR="http://10.0.2.11:8200"
vault operator unseal FyoFAkE7rlqfVSnDwm4943tYAwx51UfSntW73rQdK7SX
vault operator unseal sj38M2qmnOAegNijJ1XYtxer17rGqtrJP7OPCeG8Tq1Q
vault operator unseal /s5IYKaUo4u4vvkP6fUEDwxtHjHdtek6HIgQ+GQ4okaG
启动三个vault单元：
juju run-action vault/0 resume --wait
juju run-action vault/1 resume --wait
juju run-action vault/2 resume --wait
juju status vault
部署etcd作为vault存储后端,easyrsa作为etcd的tls证明来源。
注：
部署完etcd再部署easyrsa，不要着急
juju deploy -n 3 --config channel=3.1/stable --to lxd:0,lxd:1,lxd:2 --series focal cs:etcd-546
juju add-relation vault:shared-db mysql-innodb-cluster:shared-db
juju add-relation etcd:db vault:etcd
juju deploy --to lxd:0 --series focal cs:~containers/easyrsa
juju add-relation etcd:certificates easyrsa:client
显示 vault etcd easyrsa状态:
juju status vault etcd easyrsa
显示全部状态：
部署placement 集群：
juju add-unit --to lxd:0 placement
juju add-unit --to lxd:1 placement
juju config placement vip=10.0.7.32
juju deploy --config cluster_count=3 --series focal hacluster placement-hacluster
juju add-relation placement-hacluster:ha placement:ha
部署ceph-radosgw 集群：
juju add-unit --to lxd:1 ceph-radosgw
juju add-unit --to lxd:2 ceph-radosgw
juju config ceph-radosgw vip=10.0.7.42
juju deploy --config cluster_count=3 --series focal hacluster ceph-radosgw-hacluster
juju add-relation ceph-radosgw-hacluster:ha ceph-radosgw:ha
部署cinder 集群：
juju add-unit --to lxd:0 cinder
juju add-unit --to lxd:2 cinder
juju config cinder vip=10.0.7.47
juju deploy --config cluster_count=3 --series focal hacluster cinder-hacluster
juju add-relation cinder-hacluster:ha cinder:ha
部署glance 集群：
juju add-unit --to lxd:0 glance
juju add-unit --to lxd:1 glance
juju config glance vip=10.0.7.52
juju deploy --config cluster_count=3 --series focal hacluster glance-hacluster
juju add-relation glance-hacluster:ha glance:ha
部署neutron-api 集群：
juju add-unit --to lxd:0 neutron-api
juju add-unit --to lxd:1 neutron-api
juju config neutron-api vip=10.0.7.57
juju deploy --config cluster_count=3 --series focal hacluster neutron-api-hacluster
juju add-relation neutron-api-hacluster:ha neutron-api:ha
部署nova-cloud-controller集群：
juju add-unit --to lxd:1 nova-cloud-controller
juju add-unit --to lxd:2 nova-cloud-controller
juju config nova-cloud-controller vip=10.0.7.62
juju deploy --config cluster_count=3 --series focal hacluster nova-cloud-controller-hacluster
juju add-relation nova-cloud-controller-hacluster:ha nova-cloud-controller:ha
部署完毕后发现nava-cloud-controller状态block,显示miss relation with memcached。
经查询资料，《ReleaseNotes1501》，memcached须如下部署并添加关系。
juju deploy -n 3 --to lxd:0,lxd:1,lxd:2 --series focal memcached --debug
juju add-relation nova-cloud-controller memcached
部署openstack-dashboard 集群：
juju add-unit --to lxd:0 openstack-dashboard
juju add-unit --to lxd:2 openstack-dashboard
juju config openstack-dashboard vip=10.0.7.67
juju deploy --config cluster_count=3 --series focal hacluster openstack-dashboard-hacluster --debug
juju add-relation openstack-dashboard-hacluster:ha openstack-dashboard:ha
除easyrsa外，全部部署完HA的样子，juju status：
如果以上组状态为block，且有"Services not running that should be: apache2"，应重新导入证书。
juju run-action --wait vault/0 reissue-certificates