作者：张 发表于：2014-08-19
版权声明：可以任意转载，转载时请务必以超链接形式标明文章原始出处和作者信息及本版权声明
(http://blog.csdn.net/mrz001 )
1 PaceMaker&CoroSync
1.1 简介
PaceMaker是一种群集资源管理器。它是开源的高可用解决方案之一。它通过服务级的监控检测，并通过利用可选的群集基础结构（无论是OpenAIS或心跳）提供的消息和能力，实现了节点和资源级故障恢复的最大可用性集群服务（亦称资源）。它可以配置任意规模的集群，并配有强大的依赖模型，它允许管理员准确地表达群集资源之间的关系（包括顺序和位置）。几乎任何脚本可以作为Pacemaker的一部分来进行管理。
CoroSync是一组通信系统，用来做高可用集群的引擎。Redhat的RHCS集群套件就是基于CoroSync实现。
1.2 系统及软件版本
系统版本：
CentOSrelease 6.5 (Final) Linux 2.6.32-431.el6.x86_64
软件版本：
pacemaker 1.1.10-14.el6_5.3
clusterlib 3.0.12.1-59.el6_5.2
corosync 1.4.1-17.el6_5.1
corosynclib 1.4.1-17.el6_5.1
libibverbs 1.1.7-1.el6
libqb 0.16.0-2.el6
librdmacm 1.0.17-1.el6
pacemaker-cli 1.1.10-14.el6_5.3
pacemaker-cluster-libs 1.1.10-14.el6_5.3
pacemaker-libs 1.1.10-14.el6_5.3
perl-TimeDate 1:1.16-11.1.el6
resource-agents 3.9.2-40.el6_5.7
安装pacemaker
pacemaker自动会依赖corosync
yum install -y pacemaker corosync cluster-glue resource-agents cman rgmanager
chkconfig pacemaker on
wget ftp://ftp.pbone.net/mirror/ftp5.gwdg.de/pub/opensuse/repositories/network:/ha-clustering:/Stable/CentOS_CentOS-6/x86_64/crmsh-1.2.6-0.rc2.2.1.x86_64.rpm
yum install -ycrmsh-1.2.6-0.rc2.2.1.x86_64.rpm
1.3 PaceMaker&CoroSync配置
1.3.1 CoroSync配置
sed-i.sed “s/.*CMAN_QUORUM_TIMEOUT=.*/CMAN_QUORUM_TIMEOUT=0/g” /etc/sysconfig/cman
cp /etc/corosync/corosync.conf.example/etc/corosync/corosync.conf
vim /etc/corosync/corosync.conf
编辑如下
# Please read the corosync.conf.5manual page
compatibility: whitetank
totem {
version:2 ##版本号，只能是2，不能修改
token: 10000
secauth:off ##安全认证，当使用aisexec时，开启会非常消耗CPU
threads:0 ##线程数，根据CPU个数和核心数确定
interface{
ringnumber:0 ##冗余环号，节点有多个网卡是可定义对应网卡在一个环内
bindnetaddr:192.168.40.0 ##绑定心跳网段
mcastaddr:226.94.40.1 ##心跳组播地址
mcastport:5405 ##心跳组播使用端口
ttl:1
}
}
logging {
fileline:off ##指定要打印的行
to_stderr:no ##是否发送到标准错误输出
to_logfile:yes ##记录到文件
to_syslog:yes ##记录到syslog
logfile:/var/log/cluster/corosync.log
debug:off
timestamp:on ##是否打印时间戳，利于定位错误，但会消耗CPU
logger_subsys{
subsys:AMF
debug:off
}
}
amf {
mode:disabled
}
service {
# Load the Pacemaker Cluster Resource Manager
ver: 1
name: pacemaker ##定义corosync启动时同时启动pacemaker
}
aisexec {
user: root
group: root
}
1.3.2 生成认证Key：
corosync-keygen
会在/etc/corosync/目录下生成authkey文件
1.3.3 拷贝配置到备节点
scp /etc/corosync/authkey /etc/corosync/corosync.confnode2:/etc/corosync/
1.3.4 启动corosync&pacemaker
/etc/init.d/corosync start
corosync-cfgtool -s
/etc/init.d/pacemaker start
crm_mon
可能存在的问题：iptables没有配置相关策略，导致两个节点无法通信。可关闭iptables或配置节点间的通信策略。
1.3.5 检查启动情况
查看corosync引擎是否正常启动：
grep -e "Corosync ClusterEngine" -e "configuration file" /var/log/messages
查看初始化成员节点通知是否正常发出：
grep TOTEM /var/log/messages
检查启动过程中是否有错误产生：
grep ERROR: /var/log/messages | grep -vunpack_resources
查看pacemaker是否正常启动：
grep pcmk_startup /var/log/messages
1.4 PaceMaker命令
1.4.1 PaceMaker常用命令
CRM CLI常用命令：
进入live模式：在shell输入crm，默认进入live模式
进入cib模式：在live模式中，输入cib new <res>，即进入cib的编辑模式
cib模式下创建、删除、提交资源：
crmcib new <cib>
crmcib delete <cib>
crmcib commit <cib>
列出指定标准的可用RA：
crmra list <class> [<provider>]
查看指定RA的配置选项：
crmra meta [<class>:[<provider>:]]<type>
node节点的常见操作：
删除节点：crm node delete <node>
停用节点：crm node standby [<node>]，默认停用操作命令所在的节点
启动节点：crm node online [<node>]
设置/删除节点的attribute：crmnode attribute <node> set <attr> <value>
crmnode attribute <node> delete <attr>
resource资源的常见操作：
显示资源状态：crm resource status [<src>]
启用/停用资源：crm resource start/stop <src>
重启资源：crm resource restart <src>
手动迁移资源：crm resource migrate (move) <rsc> [<node>]
设置/删除资源的property： crmresource param <rsc> set <param> <value>
crmresource param <rsc> delete <param>
设置/删除资源的attribute：crmresource meta <rsc> set <attr> <value>
crmresource meta <rsc> delete <attr>
configure层级的常用操作：
Commandsfor resources are:
primitive：创建原始资源，最最常用的命令
monitor：资源检视
group：创建组资源
ms/master (master-slave)：创建状态克隆
Thereare three types of constraints:
location：指定资源偏爱运行于某个节点
colocation：指定哪些资源启动在同一个节点
order：指定资源的启动顺序
删除已经配置的configure：
delete <id> 可以删除资源、约束
Thereare the cluster properties, resource meta attributes defaults, and operationsdefaults:
property：设置集群的属性，对集群内所有资源有效
rsc_defaults：设置集群资源的默认attribute，对所有集群内所有资源有效
op_defaults：设置集群资源的meta attribute的默认操作，对所有资源有效
1.4.2 PaceMaker配置
propertyno-quorum-policy="ignore" \
pe-warn-series-max="1000"\
pe-input-series-max="1000"\
pe-error-series-max="1000"\
cluster-recheck-interval="5min"stonith-enabled="false" \
rsc_defaultsresource-stickiness=100
crmconfigure rsc_defaults migration-threshold=1
crmconfigure rsc_defaults failure-timeout=1
2 Keystone高可用
2.1 VIP配置
pacemaker创建vip资源,资源名称为p_vip_keystone，IP为192.168.11.101，30s检查时间
crm configure
primitive p_vip_keystone ocf:heartbeat:IPaddr2\
paramsip="192.168.11.101" cidr_netmask="24" \
opmonitor interval="30s" timeout="30s"
commit
exit
2.2 PaceMaker配置
创建keystone资源并注册到PaceMaker，其中IP和参数须根据OpenStack的认证信息确定。
crm configure
primitive p_keystone ocf:openstack:keystone \
paramsconfig="/etc/keystone/keystone.conf" \
os_tenant_name="admin" \
os_username="admin" \
os_password="111111" \
os_auth_url="http://192.168.11.101:5000/v2.0/" \
user="root" \
op monitorinterval="30s" timeout="30s"
group g_keystone p_vip_keystone p_keystone
colocation col_p_keystone_on_vip inf:p_keystone p_vip_keystone
order ord_vip_before_p_keystone inf: p_vip_keystonep_keystone
commit
exit
删除keystone资源
crm resource stop p_keystone
crm resource cleanup p_keystone
crm configure delete p_keystone
2.3 修改使用KeystoneIP配置
3 Glance高可用
3.1 VIP配置
pacemaker创建vip资源,资源名称为p_vip_ glance，IP为192.168.11.102，30s检查时间
crm configure
primitive p_vip_glance ocf:heartbeat:IPaddr2 \
paramsip="192.168.11.102" cidr_netmask="24" \
opmonitor interval="30s" timeout="30s"
commit
exit
3.2 PaceMaker配置
创建glance资源并注册到PaceMaker
crm configure
primitive p_glance-api ocf:openstack:glance-api\
paramsconfig="/etc/glance/glance-api.conf" \
pid="/var/run/glance/glance-api.pid"\
os_password="111111" \
os_username="admin" \
os_tenant_name="admin" \
os_auth_url="http://192.168.11.101:5000/v2.0/"\
op monitor interval="30s"timeout="30s"
primitive p_glance-registryocf:openstack:glance-registry \
paramsconfig="/etc/glance/glance-registry.conf" \
pid="/var/run/glance/glance-registry.pid"\
os_tenant_name="admin" \
os_username="admin" \
os_password="111111" \
url="http://192.168.11.102:9191/images"\
keystone_get_token_url="http://192.168.11.101:5000/v2.0/tokens"\
op monitor interval="30s"timeout="30s"
group g_glance p_vip_glance p_glance-apip_glance-registry
order ord_p_glance-registry_before_p_glance-apiinf: p_glance-registry p_glance-api
colocation col_p_glance-api_on_vip inf:p_glance-api p_vip_glance
order ord_vip_before_p_glance-api inf:p_vip_glance p_glance-api
colocation col_p_glance-registry_on_vip inf:p_glance-registry p_vip_glance
order ord_vip_before_p_glance-registry inf:p_vip_glance p_glance-registry
commit
exit
删除glance资源
crm resource stop p_glance-api
crm resource stop p_glance-registry
crm resource cleanup p_glance-api
crm resource cleanup p_glance-registry
crm configure delete p_glance-api
crm configure delete p_glance-registry
3.3 修改使用GlanceIP配置
4 Nova高可用
4.1 VIP配置
pacemaker创建vip资源,资源名称为p_vip_ nova，IP为192.168.11.103，30s检查时间
crm configure
primitive p_vip_nova ocf:heartbeat:IPaddr2 \
paramsip="192.168.11.103" cidr_netmask="24" \
opmonitor interval="30s" timeout="30s"
commit
exit
4.2 PaceMaker配置
创建nova资源并注册到PaceMaker
crm configure
primitive p_nova-api ocf:openstack:nova-api \
params\
os_tenant_name="admin" \
os_username="admin" \
os_password="111111" \
config="/etc/nova/nova.conf" \
pid="/var/run/nova/nova-api.pid" \
url="http://192.168.11.103:8774/v2/" \
keystone_get_token_url="http://192.168.11.101:5000/v2.0/tokens"\
opstart timeout="120s" \
opstop timeout="120s" \
opmonitor interval="60s" timeout="120s"
primitive p_nova-schedulerocf:openstack:nova-scheduler \
params\
config="/etc/nova/nova.conf" \
pid="/var/run/nova/nova-scheduler.pid" \
database_server_port="3306" \
amqp_server_port="5672" \
op starttimeout="120s" \
opstop timeout="120s" \
opmonitor interval="60s" timeout="120s"
primitive p_nova-cert ocf:openstack:nova-cert \
params\
config="/etc/nova/nova.conf" \
pid="/var/run/nova/nova-cert.pid" \
database_server_port="3306" \
amqp_server_port="5672" \
opstart timeout="120s" \
opstop timeout="120s" \
opmonitor interval="60s" timeout="120s"
primitive p_nova-conductorocf:openstack:nova-conductor \
params\
config="/etc/nova/nova.conf" \
pid="/var/run/nova/nova-conductor.pid" \
database_server_port="3306" \
amqp_server_port="5672" \
opstart timeout="120s" \
opstop timeout="120s" \
opmonitor interval="60s" timeout="120s"
primitive p_nova-consoleauth ocf:openstack:nova-consoleauth\
params\
config="/etc/nova/nova.conf" \
pid="/var/run/nova/nova-consoleauth.pid" \
database_server_port="3306" \
amqp_server_port="5672" \
opstart timeout="120s" \
opstop timeout="120s" \
opmonitor interval="60s" timeout="120s"
primitive p_nova-novncocf:openstack:nova-novnc \
params\
config="/etc/nova/nova.conf" \
pid="/var/run/nova/nova-novncproxy.pid" \
console_port="6080" \
web="/usr/share/novnc/" \
opstart timeout="120s" \
op stop timeout="120s" \
opmonitor interval="120s" timeout="120s"
group g_nova p_vip_nova p_nova-apip_nova-scheduler p_nova-cert p_nova-conductor p_nova-consoleauth p_nova-novnc
colocation col_p_nova-api_on_vip inf:p_nova-api p_vip_nova
order ord_vip_before_p_nova-api inf: p_vip_novap_nova-api
colocation col_p_nova-cert_on_vip inf:p_nova-cert p_vip_nova
order ord_vip_before_p_nova-cert inf:p_vip_nova p_nova-cert
colocation col_p_nova-conductor_on_vip inf:p_nova-conductor p_vip_nova
order ord_vip_before_p_nova-conductor inf:p_vip_nova p_nova-conductor
colocation col_p_nova-scheduler_on_vip inf:p_nova-scheduler p_vip_nova
order ord_vip_before_p_nova-scheduler inf:p_vip_nova p_nova-scheduler
colocation col_p_nova-consoleauth_on_vip inf:p_nova-consoleauth p_vip_nova
order ord_vip_before_p_nova-consoleauth inf:p_vip_nova p_nova-consoleauth
colocation col_p_nova-novnc_on_vip inf:p_nova-novnc p_vip_nova
order ord_vip_before_p_nova-novnc inf:p_vip_nova p_nova-novnc
commit
exit
删除nova资源
crm resourcestop p_nova-api
crm resourcestop p_nova-scheduler
crm resourcestop p_nova-cert
crm resourcestop p_nova-conductor
crm resourcestop p_nova-consoleauth
crm resourcestop p_nova-novnc
crm resourcecleanup p_nova-api
crm resourcecleanup p_nova-scheduler
crm resourcecleanup p_nova-cert
crm resourcecleanup p_nova-conductor
crm resourcecleanup p_nova-consoleauth
crm resourcecleanup p_nova-novnc
crmconfigure delete p_nova-api
crmconfigure delete p_nova-scheduler
crmconfigure delete p_nova-cert
crmconfigure delete p_nova-conductor
crmconfigure delete p_nova-consoleauth
crmconfigure delete p_nova-novnc
4.3 修改使用NovaIP配置
5 Cinder高可用
5.1 VIP配置
pacemaker创建vip资源,资源名称为p_vip_ cinder，IP为192.168.11.104，30s检查时间
crm configure
primitive p_vip_cinder ocf:heartbeat:IPaddr2 \
paramsip="192.168.11.104" cidr_netmask="24" \
opmonitor interval="30s" timeout="30s"
commit
exit
5.2 PaceMaker配置
创建cinder资源并注册到PaceMaker
crm configure
primitive p_cinder-api ocf:openstack:cinder-api\
paramsconfig="/etc/cinder/cinder.conf" \
pid="/var/run/cinder/cinder-api.pid" \
os_tenant_name="admin" \
os_username="admin" \
os_password="111111" \
url="http://192.168.11.104:8776/v1/" \
keystone_get_token_url="http://192.168.11.101:5000/v2.0/tokens"\
opmonitor interval="60s" timeout="60s"
primitive p_cinder-scheduleocf:openstack:cinder-schedule \
paramsconfig="/etc/cinder/cinder.conf" \
pid="/var/run/cinder/cinder-scheduler.pid" \
amqp_server_port="5672" \
opmonitor interval="60s" timeout="60s"
primitive p_cinder-volumeocf:openstack:cinder-volume \
paramsconfig="/etc/cinder/cinder.conf" \
additional_parameters="--config-file/usr/share/cinder/cinder-dist.conf" \
pid="/var/run/cinder/cinder-volume.pid" \
multibackend="true"\
amqp_server_port="5672" \
opmonitor interval="60s" timeout="60s"
primitive p_cinder-backupocf:openstack:cinder-backup \
paramsconfig="/etc/cinder/cinder.conf" \
additional_parameters="--config-file/usr/share/cinder/cinder-dist.conf" \
pid="/var/run/cinder/cinder-backup.pid" \
multibackend="true"\
amqp_server_port="5672" \
opmonitor interval="60s" timeout="60s"
group g_cinder p_vip_cinder p_cinder-apip_cinder-schedule p_cinder-volume p_cinder-backup
colocation col_p_cinder-api_on_vip inf:p_cinder-api p_vip_cinder
order ord_vip_before_p_cinder-api inf:p_vip_cinder p_cinder-api
colocation col_p_cinder-schedule_on_vip inf:p_cinder-schedule p_vip_cinder
order ord_vip_before_p_cinder-schedule inf:p_vip_cinder p_cinder-schedule
colocation col_p_cinder-volume_on_vip inf:p_cinder-volume p_vip_cinder
order ord_vip_before_p_cinder-volume inf:p_vip_cinder p_cinder-volume
colocation col_p_cinder-backup_on_vip inf:p_cinder-backup p_vip_cinder
order ord_vip_before_p_cinder-backup inf:p_vip_cinder p_cinder-backup
commit
exit
删除cinder资源
crm resource stop p_cinder-api
crm resource stop p_cinder-schedule
crm resource stop p_cinder-volume
crm resource stop p_cinder-backup
crm resource cleanup p_cinder-api
crm resource cleanup p_cinder-schedule
crm resource cleanup p_cinder-volume
crm resource cleanup p_cinder-backup
crm configure delete p_cinder-api
crm configure delete p_cinder-schedule
crm configure delete p_cinder-volume
crm configure delete p_cinder-backup
5.3 修改CinderIP配置
6 Neutron高可用
6.1 VIP配置
pacemaker创建vip资源,资源名称为p_vip_ neutron，IP为192.168.11.105，30s检查时间
crm configure
primitive p_vip_neutron ocf:heartbeat:IPaddr2 \
paramsip="192.168.11.105" cidr_netmask="24" \
opmonitor interval="30s" timeout="30s"
commit
exit
6.2 PaceMaker配置
创建Neutron资源并注册到PaceMaker
crmconfigure
primitivep_neutron-server ocf:openstack:neutron-server \
paramsconfig="/etc/neutron/neutron.conf" \
pid="/var/run/neutron/neutron.pid" \
os_tenant_name="admin" \
os_username="admin" \
os_password="111111" \
url="http://192.168.11.105:9696"\
keystone_get_token_url="http://192.168.11.101:5000/v2.0/tokens"\
op monitor interval="60s"timeout="60s"
primitivep_neutron-l3-agent ocf:openstack:neutron-l3-agent \
paramsconfig="/etc/neutron/neutron.conf" \
pid="/var/run/neutron/neutron-l3-agent.pid" \
neutron_server_port="5672" \
op monitor interval="60s"timeout="60s"
primitivep_neutron-dhcp-agent ocf:openstack:neutron-dhcp-agent \
paramsconfig="/etc/neutron/neutron.conf" \
pid="/var/run/neutron/neutron-dhcp-agent.pid" \
amqp_server_port="5672" \
op monitor interval="60s"timeout="60s"
primitivep_neutron-metadata-agent ocf:openstack:neutron-metadata-agent \
paramsconfig="/etc/neutron/neutron.conf" \
agent_config="/etc/neutron/metadata_agent.ini" \
pid="/var/run/neutron/neutron-metadata-agent.pid" \
op monitor interval="60s" timeout="60s"
primitivep_neutron-openvswitch-agent ocf:openstack:neutron-openvswitch-agent \
paramsconfig="/etc/neutron/neutron.conf" \
plugin_config="/etc/neutron/plugin.ini" \
pid="/var/run/neutron/neutron-openvswitch-agent.pid" \
segmentation_bridge="br-tun" \
op monitor interval="60s"timeout="60s"
groupg_neutron p_vip_neutron p_neutron-server p_neutron-l3-agentp_neutron-dhcp-agent p_neutron-metadata-agent p_neutron-openvswitch-agent
colocationcol_p_neutron-server_on_vip inf: p_neutron-server p_vip_neutron
orderord_vip_before_p_neutron-server inf: p_vip_neutron p_neutron-server
colocationcol_p_neutron-l3-agent_on_vip inf: p_neutron-l3-agent p_vip_neutron
orderord_vip_before_p_neutron-l3-agent inf: p_vip_neutron p_neutron-l3-agent
colocationcol_p_neutron-dhcp-agent_on_vip inf: p_neutron-dhcp-agent p_vip_neutron
orderord_vip_before_p_neutron-dhcp-agent inf: p_vip_neutron p_neutron-dhcp-agent
colocationcol_p_neutron-metadata-agent_on_vip inf: p_neutron-metadata-agent p_vip_neutron
order ord_vip_before_p_neutron-metadata-agentinf: p_vip_neutron p_neutron-metadata-agent
colocationcol_p_neutron-openvswitch-agent_on_vip inf: p_neutron-openvswitch-agentp_vip_neutron
orderord_vip_before_p_neutron-openvswitch-agent inf: p_vip_neutron p_neutron-openvswitch-agent
commit
exit
删除Neutron资源
crm resource stop p_neutron-server
crm resource stop p_neutron-l3-agent
crm resource stop p_neutron-dhcp-agent
crm resource stop p_neutron-metadata-agent
crm resource stop p_neutron-openvswitch-agent
crm resource cleanup p_neutron-server
crm resource cleanup p_neutron-l3-agent
crm resource cleanup p_neutron-dhcp-agent
crm resource cleanup p_neutron-metadata-agent
crm resource cleanupp_neutron-openvswitch-agent
crm configure delete p_neutron-server
crm configure delete p_neutron-l3-agent
crm configure delete p_neutron-dhcp-agent
crm configure delete p_neutron-metadata-agent
crm configure deletep_neutron-openvswitch-agent
6.3 修改NeutronIP配置
7 Horizone高可用
7.1 VIP配置
pacemaker创建vip资源,资源名称为p_vip_httpd，IP为192.168.11.100，30s检查时间
crm configure
primitive p_vip_httpd ocf:heartbeat:IPaddr2 \
paramsip="192.168.11.100" cidr_netmask="24" \
opmonitor interval="30s" timeout="30s"
commit
exit
7.2 PaceMaker配置
创建httpd资源并注册到PaceMaker。
crm configure
primitive p_httpd lsb:httpd
op monitor interval="30s"timeout="30s"
group g_httpd p_vip_httpd p_httpd
colocation col_p_httpd_on_vip inf: p_httpdp_vip_httpd
order ord_vip_before_p_httpd inf: p_vip_httpdp_httpd
commit
exit
删除httpd资源
crm resource stop g_httpd
crm resource cleanup g_httpd
crm resource delete g_httpd
8 active/passive配置
OpenStack组件HA测试
| 组件 |
Active/Passive |
Active/Active |
备注 |
| MySQL |
√ |
||
| Qpid / RabbitMQ |
√ |
√ |
|
| memcached |
√ |
√ |
|
| mongodb |
√ |
√ |
|
| Keystone |
√ |
Ⅹ |
|
| Glance |
Ⅹ |
||
| Horizone |
√ |
√ |
|
| Nova |
|||
| nova-api |
√ |
√ |
|
| nova-cert |
√ |
Ⅹ |
|
| nova-conductor |
√ |
Ⅹ |
|
| nova-console |
√ |
Ⅹ |
|
| nova-consoleauth |
√ |
Ⅹ |
|
| nova-metadata-api |
√ |
Ⅹ |
|
| nova-novncproxy |
√ |
Ⅹ |
|
| nova-scheduler |
√ |
Ⅹ |
|
| nova-spicehtml5proxy |
√ |
Ⅹ |
|
| nova-xvpvncproxy |
√ |
Ⅹ |
|
| nova-compute |
Ⅹ |
Ⅹ |
|
| Cinder |
|||
| cinder-api |
√ |
√ |
|
| cinder-schedule |
√ |
Ⅹ |
|
| cinder-volume |
√ |
Ⅹ |
|
| cinder-backup |
√ |
Ⅹ |
|
| Neutron |
|||
| neutron-server |
√ |
Ⅹ |
|
| neutron-l3-agent |
√ |
Ⅹ |
|
| neutron-dhcp-agent |
√ |
Ⅹ |
|
| neutron-openvswitch-agent |
√ |
Ⅹ |
|
| neutron-metadata-agent |
√ |
Ⅹ |
|
| Ceilometer |
|||
| ceilometer-alarm-evaluator |
|||
| ceilometer-alarm-notifier |
|||
| ceilometer-api |
|||
| ceilometer-central |
|||
| ceilometer-collector |
|||
| ceilometer-compute |
|
|
|
物理拓扑环境如下:
| 主机描述 |
网卡 |
主机IP |
主机安装的服务 |
服务VIP |
| 控制节点主节点 |
bond0 |
192.168.11.22 |
Keystone |
192.168.11.101 |
| Glance |
192.168.11.102 |
|||
| Nova |
192.168.11.103 |
|||
| Cinder |
192.168.11.104 |
|||
| 控制节点副节点 |
eth0 |
192.168.11.23 |
Keystone、Glance、Nova、Cinder |
|
| eth1 |
192.168.12.22 |
|||
| 网络节点主节点 |
eth0 |
192.168.11.20 |
Neutron |
192.168.11.105 |
| br-tun |
192.168.12.10 |
|||
| br-ex |
192.168.13.11 |
|||
| 网络节点副节点 |
br-ex |
192.168.11.15 |
Neutron |
|
| br-tun |
192.168.12.51 |
|||
| 计算节点 |
eth0 |
192.168.11.21 |
Nova Compute、MySQL、Qpid |
|
| br-tun |
192.168.12.11 |
系统版本：
CentOS release 6.5 (Final) Linux2.6.32-431.el6.x86_64
基本前提配置：
节点配置好源、时间同步、ssh互相通信、hosts名称解析、关闭iptables或者开启相应策略、关闭SELinux。
8.1 配置ssh互相通信
scp ~/.ssh/authorized_keys ~/.ssh/id_rsa192.168.11.20:/root/.ssh/
scp ~/.ssh/authorized_keys ~/.ssh/id_rsa192.168.11.21:/root/.ssh/
scp ~/.ssh/authorized_keys ~/.ssh/id_rsa192.168.11.22:/root/.ssh/
scp ~/.ssh/authorized_keys ~/.ssh/id_rsa192.168.11.23:/root/.ssh/
scp ~/.ssh/authorized_keys ~/.ssh/id_rsa192.168.11.15:/root/.ssh/
8.2 配置时间同步
scp /etc/ntp.conf 192.168.11.20:/etc/
scp /etc/ntp.conf 192.168.11.21:/etc/
scp /etc/ntp.conf 192.168.11.22:/etc/
scp /etc/ntp.conf 192.168.11.23:/etc/
scp /etc/ntp.conf 192.168.11.15:/etc/
scp /etc/sysconfig/clock192.168.11.20:/etc/sysconfig
scp /etc/sysconfig/clock192.168.11.21:/etc/sysconfig
scp /etc/sysconfig/clock192.168.11.22:/etc/sysconfig
scp /etc/sysconfig/clock192.168.11.23:/etc/sysconfig
scp /etc/sysconfig/clock192.168.11.15:/etc/sysconfig
重启所有节点ntp服务
/etc/init.d/ntpd restart
8.3 配置源
cd /etc/yum.repos.d/
scp * 192.168.11.20:/etc/yum.repos.d/
scp * 192.168.11.21:/etc/yum.repos.d/
scp * 192.168.11.22:/etc/yum.repos.d/
scp * 192.168.11.23:/etc/yum.repos.d/
scp * 192.168.11.15:/etc/yum.repos.d/
yum clean all
8.4 配置hosts
scp /etc/hosts 192.168.11.20:/etc/
scp /etc/hosts 192.168.11.21:/etc/
scp /etc/hosts 192.168.11.22:/etc/
scp /etc/hosts 192.168.11.23:/etc/
scp /etc/hosts 192.168.11.15:/etc/
8.5 关闭Iptables/ SELinux
/etc/init.d/iptables stop &&chkconfigiptables off
sed -i '/^SELINUX=/ cSELINUX=disabled'/etc/selinux/config
setenforce 0
ssh 192.168.11.21 -C “sed -i '/^SELINUX=/cSELINUX=disabled' /etc/selinux/config && setenforce 0 &&/etc/init.d/iptables stop && chkconfig iptables off”
ssh 192.168.11.22 -C “sed -i '/^SELINUX=/cSELINUX=disabled' /etc/selinux/config && setenforce 0 &&/etc/init.d/iptables stop && chkconfig iptables off”
ssh 192.168.11.23 -C “sed -i '/^SELINUX=/cSELINUX=disabled' /etc/selinux/config && setenforce 0 &&/etc/init.d/iptables stop && chkconfig iptables off”
ssh 192.168.11.15 -C “sed -i '/^SELINUX=/cSELINUX=disabled' /etc/selinux/config && setenforce 0 &&/etc/init.d/iptables stop && chkconfig iptables off”
8.6 控制节点A/P组件高可用配置
本环境中OpenStack控制主节点为192.168.11.22
OpenStack控制备节点为192.168.11.23
注：为保证切换过程中消息的可持续性，建议将主备节点的HOSTNAME名称改一致。
8.6.1 安装PaceMaker&CoroSync
a) 安装服务
[[email protected]~]# yum install -y pacemaker corosync cluster-glue resource-agents openais cmanrgmanager ccs
b) 设置cman 和pacemaker 开机自启动
[[email protected]~]# chkconfig pacemaker on
[[email protected]~]# chkconfig cman on
c) 安装crmsh 客户端工具
[[email protected]~]# wgetftp://ftp.pbone.net/mirror/ftp5.gwdg.de/pub/opensuse/repositories/network:/ha-clustering:/Stable/CentOS_CentOS-6/x86_64/crmsh-1.2.6-0.rc2.2.1.x86_64.rpm&& yum install -y crmsh-1.2.6-0.rc2.2.1.x86_64.rpm
d) 远程在192.168.11.23执行安装,并修改hostname为主节点
[[email protected]~]# ssh 192.168.11.23 -C "yum install -y pacemaker corosync cluster-glueresource-agents openais cman rgmanager ccs && chkconfig pacemaker on &&chkconfig cman on && wgetftp://ftp.pbone.net/mirror/ftp5.gwdg.de/pub/opensuse/repositories/network:/ha-clustering:/Stable/CentOS_CentOS-6/x86_64/crmsh-1.2.6-0.rc2.2.1.x86_64.rpm&& yum install -y crmsh-1.2.6-0.rc2.2.1.x86_64.rpm"
[[email protected]~]# name=`echo hostname` && ssh 192.168.11.23 -C "hostname `$name`&& sed -i '/^HOSTNAME=/ cHOSTNAME=`$name`' /etc/sysconfig/network"
e) PaceMaker&CoroSync配置
CoroSync配置
sed -i.sed “s/.*CMAN_QUORUM_TIMEOUT=.*/CMAN_QUORUM_TIMEOUT=0/g” /etc/sysconfig/cman
[[email protected] ~]# cp/etc/corosync/corosync.conf.example /etc/corosync/corosync.conf
[[email protected] ~]# vim /etc/corosync/corosync.conf
编辑如下
# Please read the corosync.conf.5 manual page
compatibility: whitetank
totem {
version:2 ##版本号，只能是2，不能修改
token: 10000
secauth:off ##安全认证，当使用aisexec时，开启会非常消耗CPU
threads:0 ##线程数，根据CPU个数和核心数确定
interface{
ringnumber:0 ##冗余环号，节点有多个网卡是可定义对应网卡在一个环内
bindnetaddr:192.168.11.0 ##绑定心跳网段
mcastaddr:226.94.11.1 ##心跳组播地址，用于集群内机器心跳监听
mcastport:5405 ##心跳组播使用端口
ttl:1
}
}
logging {
fileline:off ##指定要打印的行
to_stderr:no ##是否发送到标准错误输出
to_logfile:yes ##记录到文件
to_syslog:yes ##记录到syslog
logfile:/var/log/cluster/corosync.log
debug:off
timestamp:on ##是否打印时间戳，利于定位错误，但会消耗CPU
logger_subsys{
subsys:AMF
debug:off
}
}
amf {
mode:disabled
}
service {
ver: 1
name: pacemaker ##定义corosync启动时同时启动pacemaker
}
aisexec {
user: root
group: root
}
f) 生成认证Key：
[[email protected] ~]# corosync-keygen
等待一会儿后在/etc/corosync/目录下会生成authkey文件
g) 拷贝配置到备节点
[[email protected]~]# scp /etc/corosync/authkey/etc/corosync/corosync.conf 192.168.11.23:/etc/corosync/
authkey 100% 128 0.1KB/s 00:00
corosync.conf 100% 606 0.6KB/s 00:00
h) 启动cman&pacemaker
[[email protected]~]# vim /etc/cluster/cluster.conf
编辑如下
<?xmlversion="1.0"?>
<clusterconfig_version="3" name="cluster">
<clusternodes>
<clusternodename="192.168.11.22" nodeid="1"/>
<clusternodename="192.168.11.23" nodeid="2"/>
</clusternodes>
<cman expected_votes="1"two_node="1"/>
<fencedevices/>
<rm/>
</cluster>
[[email protected]~]# scp /etc/cluster/cluster.conf 192.168.11.23:/etc/cluster/
[[email protected]~]# /etc/init.d/cman start
[[email protected]~]# /etc/init.d/pacemaker status
[[email protected]~]# /etc/init.d/pacemaker start
[[email protected]~]# ssh 192.168.11.23 -C “/etc/init.d/cman start && /etc/init.d/pacemakerstatus &&/etc/init.d/pacemaker start ”
i) 查看cman集群状态
Lastupdated: Tue Jul 15 13:12:38 2014
Last change:Tue Jul 15 13:01:25 2014 via crmd on 192.168.11.20
Stack: cman
Current DC: 192.168.11.20- partition with quorum
Version:1.1.10-14.el6-368c726
2 Nodesconfigured
0 Resourcesconfigured
Online: [192.168.11.22 192.168.11.23 ]
注：可能存在的问题：iptables没有配置相关策略，导致两个节点无法通信。可关闭iptables或配置节点间的通信策略。
j) 1.3.5 检查启动情况
查看corosync引擎是否正常启动：
grep -e "Corosync Cluster Engine" -e"configuration file" /var/log/messages
查看初始化成员节点通知是否正常发出：
grep TOTEM /var/log/messages
检查启动过程中是否有错误产生：
grep ERROR: /var/log/messages | grep -vunpack_resources
查看pacemaker是否正常启动：
grep pcmk_startup /var/log/messages
k) 初始化pacemaker配置
crm configure \
property no-quorum-policy="ignore" \
pe-warn-series-max="1000" \
pe-input-series-max="1000" \
pe-error-series-max="1000" \
cluster-recheck-interval="5min"stonith-enabled="false"
crm configure rsc_defaultsresource-stickiness=100
crm configure rsc_defaultsmigration-threshold=1
crm configure rsc_defaults failure-timeout=1
l) 上传OCF脚本
mkdir /usr/lib/ocf/resource.d/openstack
cd /usr/lib/ocf/resource.d/openstack
上传ocf脚本（见附件OCF脚本.zip）
chmod a+rx /usr/lib/ocf/resource.d/openstack/*
scp -r /usr/lib/ocf/resource.d/openstack192.168.11.23:/usr/lib/ocf/resource.d/
8.6.2 Keystone组件高可用
Keystone服务的虚IP为192.168.11.101
修改Keystone 数据库中 endpoint
5000与35357端口的IP为虚IP 192.168.11.101
+++++++++++++++
9e9ecf182b8e46eabf4e73a1a740df22 388d9a686ecf43b58ad8180d4b0293fc admin RegionOne 81e514476f7e43309bf80a76e35eed28 http://192.168.11.101:35357/v2.0 {}
63c207cb645840aa9482548889a01855 388d9a686ecf43b58ad8180d4b0293fc public RegionOne 81e514476f7e43309bf80a76e35eed28 http://192.168.11.101:5000/v2.0 {}
7c22e8110500478f96073f5036b9497e 388d9a686ecf43b58ad8180d4b0293fc internal RegionOne 81e514476f7e43309bf80a76e35eed28 http://192.168.11.101:5000/v2.0 {}
+++++++++++++++
修改keystone配置
vim /etc/keystone/keystone.conf
去掉注释 config_file =/usr/share/keystone/keystone-dist-paste.ini
去掉注释 log_file =/var/log/keystone/keystone.log
修改配置中KeystoneIP 为KeystoneVIP
/etc/nova/nova.conf
/etc/glance/glance-api.conf
/etc/glance/glance-registry.conf
/etc/cinder/api-paste.ini
/etc/openstack-dashboard/local_settings
/etc/ceilometer/ceilometer.conf
/etc/neutron/api-paste.ini
/etc/neutron/metadata_agent.ini
/etc/neutron/neutron.conf
在备节点上安装keystone服务
ssh192.168.11.23 -C "yum install -y openstack-keystone"
复制keystone配置文件
scp -r/etc/keystone/ 192.168.11.23:/etc/
ssh192.168.11.23 -C "chown -R keystone /etc/keystone"
配置Keystone crm 脚本
crmconfigure
primitivep_vip_keystone ocf:heartbeat:IPaddr2 \
params ip="192.168.11.101"cidr_netmask="24" \
op monitor interval="30s" timeout="30s"
commit
exit
crmconfigure
primitivep_keystone ocf:openstack:keystone \
paramsconfig="/etc/keystone/keystone.conf" \
os_tenant_name="admin" \
os_username="admin" \
os_password="111111" \
os_auth_url="http://192.168.11.101:5000/v2.0/" \
op monitorinterval="30s" timeout="30s"
groupg_keystone p_vip_keystone p_keystone
colocationcol_p_keystone_on_vip inf: p_keystone p_vip_keystone
orderord_vip_before_p_keystone inf: p_vip_keystone p_keystone
commit
exit
#################
# 开启资源 #
#################
crmresource start g_keystone
#################
# 停止资源 #
#################
crm resourcestop g_keystone
#################
# 删除资源 #
#################
crm resourcestop g_keystone
crmresource cleanup g_keystone
crm configuredelete g_keystone
8.6.3 Glance组件高可用
由于Glance服务需要使用相同的镜像文件，必须提前在主备节点挂载共享目录
/var/lib/glance/images/
Glance服务的虚IP为192.168.11.102
修改Keystone 数据库中 endpoint
9292端口的IP为虚IP192.168.11.102
+++++++++++++++++++++
3a96184cf75a41d4bc6902f299dc3593 543a50f136f5426dad9c95b88e24a06d public RegionOne 12bed2112ea14b24a1c4f73e7a8d955f http://192.168.11.102:9292 {}
5a9b6e3010834a2ab48289a38f4097b5 543a50f136f5426dad9c95b88e24a06d internal RegionOne 12bed2112ea14b24a1c4f73e7a8d955f http://192.168.11.102:9292 {}
6c35f5cc277749afbf513a40205b2d14 543a50f136f5426dad9c95b88e24a06d admin RegionOne 12bed2112ea14b24a1c4f73e7a8d955f http://192.168.11.102:9292 {}
+++++++++++++++++++++
修改glance配置
vim /etc/glance/glance-api.conf
去掉注释config_file=/usr/share/glance/glance-api-dist-paste.ini
修改glance-registry配置
vim /etc/glance/glance-registry.conf
去掉注释 config_file=/usr/share/glance/glance-registry-dist-paste.ini
修改配置中GlanceIP 为GlanceVIP
/etc/nova/nova.conf
/etc/cinder/cinder.conf
在备节点上安装glance服务
ssh 192.168.11.23 -C "yum install -yopenstack-glance"
scp -r /etc/glance/ 192.168.11.23:/etc/
ssh 192.168.11.23 -C "chown -R glance/etc/glance"
配置Glance crm 脚本
crm configure
primitive p_vip_glance ocf:heartbeat:IPaddr2 \
paramsip="192.168.11.102" cidr_netmask="24" \
opmonitor interval="30s" timeout="30s"
commit
exit
crm configure
primitive p_glance-api ocf:openstack:glance-api\
paramsconfig="/etc/glance/glance-api.conf" \
pid="/var/run/glance/glance-api.pid"\
os_password="111111" \
os_username="admin" \
os_tenant_name="admin" \
os_auth_url="http://192.168.11.101:5000/v2.0/"\
op monitor interval="30s"timeout="30s"
primitive p_glance-registryocf:openstack:glance-registry \
paramsconfig="/etc/glance/glance-registry.conf" \
pid="/var/run/glance/glance-registry.pid"\
os_tenant_name="admin" \
os_username="admin" \
os_password="111111" \
url="http://192.168.11.102:9191/images"\
keystone_get_token_url="http://192.168.11.101:5000/v2.0/tokens"\
op monitor interval="30s"timeout="30s"
order ord_p_glance-registry_before_p_glance-apiinf: p_glance-registry p_glance-api
colocation col_p_glance-api_on_vip inf:p_glance-api p_vip_glance
order ord_vip_before_p_glance-api inf:p_vip_glance p_glance-api
colocation col_p_glance-registry_on_vip inf:p_glance-registry p_vip_glance
order ord_vip_before_p_glance-registry inf:p_vip_glance p_glance-registry
commit
exit
#################
# 开启资源 #
#################
crmresource start g_glance
#################
# 停止资源 #
#################
crmresource stop g_glance
#################
# 删除资源 #
#################
crmresource stop g_glance
crmresource cleanup g_glance
crmresource delete g_glance
8.6.4 Nova组件高可用
Nova服务的虚IP为192.168.11.103
修改Keystone 数据库中 endpoint
8774端口的IP为虚IP192.168.11.103
+++++++++++++++++++++
03955152e2e0411dafd0949475414332 49d6e75570cf448c9e835fd6d667c277 admin RegionOne 2ad20f405f4144738e486c0f3049f255 http://192.168.11.103:8774/v2/%(tenant_id)s {}
2745680abaf146b8b0c6ba67aabf563a 49d6e75570cf448c9e835fd6d667c277 public RegionOne 2ad20f405f4144738e486c0f3049f255 http://192.168.11.103:8774/v2/%(tenant_id)s {}
51e8876785f44862bb240b0da3507357 49d6e75570cf448c9e835fd6d667c277 internal RegionOne 2ad20f405f4144738e486c0f3049f255 http://192.168.11.103:8774/v2/%(tenant_id)s {}
+++++++++++++++++++++
修改配置vim /etc/nova/nova.conf
glance_api_servers=192.168.11.102:9292
metadata_host=192.168.11.103
neutron_url=http://192.168.11.20:9696
neutron_admin_auth_url=http://192.168.11.101:35357/v2.0
在备节点上安装nova服务
ssh192.168.11.23 -C "yum install -y openstack-nova-consoleopenstack-nova-novncproxy openstack-nova-api openstack-nova-conductoropenstack-nova-cert openstack-nova-scheduler"
复制nova配置
scp -r /etc/nova/ 192.168.11.23:/etc/
ssh 192.168.11.23 -C "chown -R nova/etc/nova"
配置Nova crm 脚本
crm configure
primitive p_vip_nova ocf:heartbeat:IPaddr2 \
paramsip="192.168.11.103" cidr_netmask="24" \
opmonitor interval="30s" timeout="30s"
commit
exit
crm configure
primitive p_nova-api ocf:openstack:nova-api \
params\
os_tenant_name="admin" \
os_username="admin" \
os_password="111111" \
config="/etc/nova/nova.conf" \
pid="/var/run/nova/nova-api.pid" \
url="http://192.168.11.103:8774/v2/" \
keystone_get_token_url="http://192.168.11.101:5000/v2.0/tokens"\
opstart timeout="120s" \
opstop timeout="120s" \
opmonitor interval="60s" timeout="120s"
primitive p_nova-schedulerocf:openstack:nova-scheduler \
params\
config="/etc/nova/nova.conf" \
pid="/var/run/nova/nova-scheduler.pid" \
database_server_port="3306" \
amqp_server_port="5672" \
opstart timeout="120s" \
opstop timeout="120s" \
opmonitor interval="60s" timeout="120s"
primitive p_nova-cert ocf:openstack:nova-cert \
params\
config="/etc/nova/nova.conf" \
pid="/var/run/nova/nova-cert.pid" \
database_server_port="3306" \
amqp_server_port="5672" \
opstart timeout="120s" \
opstop timeout="120s" \
opmonitor interval="60s" timeout="120s"
primitive p_nova-conductorocf:openstack:nova-conductor \
params\
config="/etc/nova/nova.conf" \
pid="/var/run/nova/nova-conductor.pid" \
database_server_port="3306" \
amqp_server_port="5672" \
opstart timeout="120s" \
opstop timeout="120s" \
opmonitor interval="60s" timeout="120s"
primitive p_nova-consoleauthocf:openstack:nova-consoleauth \
params\
config="/etc/nova/nova.conf" \
pid="/var/run/nova/nova-consoleauth.pid" \
database_server_port="3306" \
amqp_server_port="5672" \
opstart timeout="120s" \
opstop timeout="120s" \
opmonitor interval="60s" timeout="120s"
primitive p_nova-novncocf:openstack:nova-novnc \
params\
config="/etc/nova/nova.conf" \
pid="/var/run/nova/nova-novncproxy.pid" \
console_port="6080" \
web="/usr/share/novnc/" \
opstart timeout="120s" \
opstop timeout="120s" \
op monitorinterval="120s" timeout="120s"
group g_nova p_vip_nova p_nova-apip_nova-scheduler p_nova-cert p_nova-conductor p_nova-consoleauth p_nova-novnc
colocation col_p_nova-api_on_vip inf:p_nova-api p_vip_nova
order ord_vip_before_p_nova-api inf: p_vip_novap_nova-api
colocation col_p_nova-cert_on_vip inf:p_nova-cert p_vip_nova
order ord_vip_before_p_nova-cert inf:p_vip_nova p_nova-cert
colocation col_p_nova-conductor_on_vip inf:p_nova-conductor p_vip_nova
order ord_vip_before_p_nova-conductor inf:p_vip_nova p_nova-conductor
colocation col_p_nova-scheduler_on_vip inf:p_nova-scheduler p_vip_nova
order ord_vip_before_p_nova-scheduler inf:p_vip_nova p_nova-scheduler
colocation col_p_nova-consoleauth_on_vip inf:p_nova-consoleauth p_vip_nova
order ord_vip_before_p_nova-consoleauth inf:p_vip_nova p_nova-consoleauth
colocation col_p_nova-novnc_on_vip inf:p_nova-novnc p_vip_nova
order ord_vip_before_p_nova-novnc inf:p_vip_nova p_nova-novnc
commit
exit
#################
# 开启资源 #
#################
crm resource start g_nova
#################
# 停止资源 #
#################
crm resource stop g_nova
#################
# 删除资源 #
#################
crm resource stop g_nova
crm resource cleanup g_nova
crm resource delete g_nova
8.6.5 Cinder组件高可用
Cinder备机需要挂载与主机相同的iscsi设备
Cinder备机需要访问相同的cinder-volumesVG
Cinder服务的虚IP为192.168.11.104
修改Keystone 数据库中 endpoint
8776端口的IP为虚IP192.168.11.104
+++++++++++++++++++++
5844447f5d6c4ea490378eadc6e5d32b f7316baf8d6241fe95c46a7726865d08 admin RegionOne d4e5f5d61aa7422aa2a6ab9149ca5da6 http://192.168.11.104:8776/v1/%(tenant_id)s {}
7238aa8b01b14b1abd141c5046e84c35 f7316baf8d6241fe95c46a7726865d08 internal RegionOne d4e5f5d61aa7422aa2a6ab9149ca5da6 http://192.168.11.104:8776/v1/%(tenant_id)s {}
eae02ac984cf4cf38ce33dacbefb5587 f7316baf8d6241fe95c46a7726865d08 public RegionOne d4e5f5d61aa7422aa2a6ab9149ca5da6 http://192.168.11.104:8776/v1/%(tenant_id)s {}
+++++++++++++++++++++
修改配置/etc/cinder/cinder.conf
iscsi_ip_address=192.168.11.104
glance_host=192.168.11.102
在备节点上安装cinder服务
ssh 192.168.11.23 -C "yum install -yopenstack-cinder"
复制配置文件
scp -r /etc/cinder/ 192.168.11.23:/etc/
ssh 192.168.11.23 -C "chown -R cinder/etc/cinder"
配置Cinder crm 脚本
crm configure
primitive p_vip_cinder ocf:heartbeat:IPaddr2 \
paramsip="192.168.11.104" cidr_netmask="24" \
opmonitor interval="30s" timeout="30s"
commit
exit
crm configure
primitive p_cinder-api ocf:openstack:cinder-api\
paramsconfig="/etc/cinder/cinder.conf" \
pid="/var/run/cinder/cinder-api.pid" \
os_tenant_name="admin" \
os_username="admin" \
os_password="111111" \
url="http://192.168.11.104:8776/v1/" \
keystone_get_token_url="http://192.168.11.101:5000/v2.0/tokens"\
opmonitor interval="60s" timeout="60s"
primitive p_cinder-scheduleocf:openstack:cinder-schedule \
paramsconfig="/etc/cinder/cinder.conf" \
pid="/var/run/cinder/cinder-scheduler.pid" \
amqp_server_port="5672" \
opmonitor interval="60s" timeout="60s"
primitive p_cinder-volume ocf:openstack:cinder-volume\
paramsconfig="/etc/cinder/cinder.conf" \
additional_parameters="--config-file/usr/share/cinder/cinder-dist.conf" \
pid="/var/run/cinder/cinder-volume.pid" \
multibackend="true"\
amqp_server_port="5672" \
opmonitor interval="60s" timeout="60s"
primitive p_cinder-backupocf:openstack:cinder-backup \
paramsconfig="/etc/cinder/cinder.conf" \
additional_parameters="--config-file/usr/share/cinder/cinder-dist.conf" \
pid="/var/run/cinder/cinder-backup.pid" \
multibackend="true"\
amqp_server_port="5672" \
opmonitor interval="60s" timeout="60s"
group g_cinder p_vip_cinder p_cinder-apip_cinder-schedule p_cinder-volume p_cinder-backup
colocation col_p_cinder-api_on_vip inf:p_cinder-api p_vip_cinder
order ord_vip_before_p_cinder-api inf:p_vip_cinder p_cinder-api
colocation col_p_cinder-schedule_on_vip inf:p_cinder-schedule p_vip_cinder
order ord_vip_before_p_cinder-schedule inf:p_vip_cinder p_cinder-schedule
colocation col_p_cinder-volume_on_vip inf:p_cinder-volume p_vip_cinder
order ord_vip_before_p_cinder-volume inf:p_vip_cinder p_cinder-volume
colocation col_p_cinder-backup_on_vip inf:p_cinder-backup p_vip_cinder
order ord_vip_before_p_cinder-backup inf:p_vip_cinder p_cinder-backup
commit
exit
#################
# 开启资源 #
#################
crm resource start g_cinder
#################
# 停止资源 #
#################
crm resource stop g_cinder
#################
# 删除资源 #
#################
crm resource stop g_cinder
crm resource cleanup g_cinder
crm resource delete g_cinder
8.6.6 Horizone组件高可用
Horizone服务的虚IP为192.168.11.100
修改Horizone配置
vim /etc/openstack-dashboard/local_settings
修改
OPENSTACK_HOST = "192.168.11.101" IP为Keystone虚IP
ALLOWED_HOSTS = ['*', 'localhost.localdomain','localhost', ]
远程在horizone备机192.168.11.23上安装Horizone服务
ssh192.168.11.23 -C “yum install memcached python-memcached mod_wsgi openstack-dashboard”
ssh192.168.11.23 -C “/etc/init.d/memcached start“
ssh192.168.11.23 -C “chkconfig memcached on”
复制配置文件
scp -r/etc/openstack-dashboard/ 192.168.11.23:/etc/
scp -r/etc/httpd/ 192.168.11.23:/etc/
配置Horizone crm 脚本
crm configure
primitive p_vip_httpd ocf:heartbeat:IPaddr2 \
paramsip="192.168.11.100" cidr_netmask="24" \
opmonitor interval="30s" timeout="30s"
commit
exit
crm configure
primitive p_httpd lsb:httpd
op monitor interval="30s"timeout="30s"
group g_httpd p_vip_httpd p_httpd
colocation col_p_httpd_on_vip inf: p_httpdp_vip_httpd
order ord_vip_before_p_httpd inf: p_vip_httpdp_httpd
commit
exit
#################
# 开启资源 #
#################
crm resource start g_httpd
#################
# 停止资源 #
#################
crm resource stop g_httpd
#################
# 删除资源 #
#################
crm resource stop g_httpd
crm resource cleanup g_httpd
crm resource delete g_httpd
8.7 网络节点A/P组件高可用配置
本环境中OpenStack网络主节点为192.168.11.20
OpenStack网络备节点为192.168.11.15
注：为保证切换过程中消息的可持续性，建议将主备节点的HOSTNAME名称改一致。
8.7.1 安装PaceMaker&CoroSync
a) 安装服务
[[email protected]~]# yum install -y pacemaker corosync cluster-glue resource-agents openais cmanrgmanager ccs
b) 设置cman 和pacemaker 开机自启动
[[email protected]~]# chkconfig pacemaker on
[[email protected]~]# chkconfig cman on
c) 安装crmsh 客户端工具
[[email protected]~]# wget ftp://ftp.pbone.net/mirror/ftp5.gwdg.de/pub/opensuse/repositories/network:/ha-clustering:/Stable/CentOS_CentOS-6/x86_64/crmsh-1.2.6-0.rc2.2.1.x86_64.rpm&& yum install -y crmsh-1.2.6-0.rc2.2.1.x86_64.rpm
d) 远程在192.168.11.23执行安装
[[email protected]~]# ssh 192.168.11.15 -C "yum install -y pacemaker corosync cluster-glueresource-agents openais cman rgmanager ccs && chkconfig pacemaker on &&chkconfig cman on && wgetftp://ftp.pbone.net/mirror/ftp5.gwdg.de/pub/opensuse/repositories/network:/ha-clustering:/Stable/CentOS_CentOS-6/x86_64/crmsh-1.2.6-0.rc2.2.1.x86_64.rpm&& yum install -y crmsh-1.2.6-0.rc2.2.1.x86_64.rpm"
e) PaceMaker&CoroSync配置
CoroSync配置
sed -i.sed“s/.*CMAN_QUORUM_TIMEOUT=.*/CMAN_QUORUM_TIMEOUT=0/g” /etc/sysconfig/cman
CoroSync配置
[[email protected] ~]# cp /etc/corosync/corosync.conf.example/etc/corosync/corosync.conf
[[email protected] ~]# vim/etc/corosync/corosync.conf
编辑如下
# Please read the corosync.conf.5 manual page
compatibility: whitetank
totem {
version:2 ##版本号，只能是2，不能修改
token: 10000
secauth:off ##安全认证，当使用aisexec时，开启会非常消耗CPU
threads:0 ##线程数，根据CPU个数和核心数确定
interface{
ringnumber:0 ##冗余环号，节点有多个网卡是可定义对应网卡在一个环内
bindnetaddr:192.168.11.0 ##绑定心跳网段
mcastaddr:227.94.11.1 ##心跳组播地址，用于集群内机器心跳监听
mcastport:5405 ##心跳组播使用端口
ttl:1
}
}
logging {
fileline:off ##指定要打印的行
to_stderr:no ##是否发送到标准错误输出
to_logfile:yes ##记录到文件
to_syslog:yes ##记录到syslog
logfile:/var/log/cluster/corosync.log
debug:off
timestamp:on ##是否打印时间戳，利于定位错误，但会消耗CPU
logger_subsys{
subsys:AMF
debug:off
}
}
amf {
mode:disabled
}
service {
ver: 1
name: pacemaker ##定义corosync启动时同时启动pacemaker
}
aisexec {
user: root
group: root
}
f) 生成认证Key：
[[email protected] ~]# corosync-keygen
等待一会儿后在/etc/corosync/目录下会生成authkey文件
g) 拷贝配置到备节点
[[email protected]~]# scp /etc/corosync/authkey /etc/corosync/corosync.conf 192.168.11.15:/etc/corosync/
authkey 100% 128 0.1KB/s 00:00
corosync.conf 100% 606 0.6KB/s 00:00
h) 启动cman&pacemaker
[[email protected]~]# vim /etc/cluster/cluster.conf
编辑如下
<?xmlversion="1.0"?>
<clusterconfig_version="3" name="cluster">
<clusternodes>
<clusternode name="192.168.11.20"nodeid="1"/>
<clusternode name="192.168.11.15"nodeid="2"/>
</clusternodes>
<cman expected_votes="1"two_node="1"/>
<fencedevices/>
<rm/>
</cluster>
[[email protected]~]# scp /etc/cluster/cluster.conf 192.168.11.15:/etc/cluster/
[[email protected]~]# /etc/init.d/cman start
[[email protected]~]# /etc/init.d/pacemaker status
[[email protected]~]# /etc/init.d/pacemaker start
[[email protected]~]# ssh 192.168.11.15 -C “/etc/init.d/cman start && /etc/init.d/pacemakerstatus &&/etc/init.d/pacemaker start ”
i) 查看cman集群状态
Lastupdated: Tue Jul 15 13:12:38 2014
Last change:Tue Jul 15 13:01:25 2014 via crmd on 192.168.11.20
Stack: cman
Current DC:192.168.11.20 - partition with quorum
Version:1.1.10-14.el6-368c726
2 Nodesconfigured
0 Resourcesconfigured
Online: [ 192.168.11.20192.168.11.15 ]
注：可能存在的问题：iptables没有配置相关策略，导致两个节点无法通信。可关闭iptables或配置节点间的通信策略。
j) 1.3.5 检查启动情况
查看corosync引擎是否正常启动：
grep -e "Corosync Cluster Engine" -e"configuration file" /var/log/messages
查看初始化成员节点通知是否正常发出：
grep TOTEM /var/log/messages
检查启动过程中是否有错误产生：
grep ERROR: /var/log/messages | grep -vunpack_resources
查看pacemaker是否正常启动：
grep pcmk_startup /var/log/messages
k) 初始化pacemaker配置
crm configure \
property no-quorum-policy="ignore" \
pe-warn-series-max="1000" \
pe-input-series-max="1000" \
pe-error-series-max="1000" \
cluster-recheck-interval="5min"stonith-enabled="false"
crm configure rsc_defaultsresource-stickiness=100
crm configure rsc_defaultsmigration-threshold=1
crm configure rsc_defaults failure-timeout=1
l) 上传OCF脚本
mkdir /usr/lib/ocf/resource.d/openstack
cd /usr/lib/ocf/resource.d/openstack
上传ocf脚本（见附件OCF脚本.zip）
chmod a+rx /usr/lib/ocf/resource.d/openstack/*
scp -r /usr/lib/ocf/resource.d/openstack192.168.11.15:/usr/lib/ocf/resource.d/
8.7.2 Neutron组件高可用
注：Neutron 主备节点需要配置相同的hostname
Neutron服务的虚IP为192.168.11.105
修改Keystone 数据库中 endpoint
9696端口的IP改为虚IP192.168.11.105
+++++++++++++++++++++
363ae64ff91e46fb83a9a7cfd1d62c20 fcd3d0b5b034482c84f478e0e9a2b681 admin RegionOne 177b590f8b9940b3ada281242b39aba8 http://192.168.11.105:9696/ {}
892380a5d69a428e823ab65e68a2f774 fcd3d0b5b034482c84f478e0e9a2b681 internal RegionOne 177b590f8b9940b3ada281242b39aba8 http://192.168.11.105:9696/ {}
c0acb9c5baa44faa87baf2d4175c2e78 fcd3d0b5b034482c84f478e0e9a2b681 public RegionOne 177b590f8b9940b3ada281242b39aba8 http://192.168.11.105:9696/ {}
+++++++++++++++++++++
修改控制节点与所有计算节点nova配置中neutronIP 为NeutronVIP
/etc/nova/nova.conf
neutron_url=http://192.168.11.105:9696
在主节点关闭Neutron服务自启动
chkconfigneutron-dhcp-agent off
chkconfigneutron-l3-agent off
chkconfigneutron-lbaas-agent off
chkconfigneutron-metadata-agent off
chkconfigneutron-openvswitch-agent off
chkconfigneutron-server off
chkconfig|grep neutron
在备节点安装Neutron服务，并修改hostname为主节点
ssh192.168.11.15 -C “yum install openstack-neutron openstack-neutron-openvswitchpython-neutronclient”
name=`echohostname` && ssh 192.168.11.15 -C "hostname `$name` && sed-i '/^HOSTNAME=/ cHOSTNAME=`$name`' /etc/sysconfig/network"
复制配置文件到备节点
scp -r/etc/neutron/ [email protected]:/etc/
ssh192.168.11.15 -C “chown -R neutron:neutron /etc/neutron/ ”
登录Neutron备节点配置neutron plugin参数
vim /etc/neutron/plugin.ini
local_ip=192.168.12.51
vim /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini
local_ip=192.168.12.51
修改/etc/sysctl.conf
net.ipv4.ip_forward=1
net.ipv4.conf.all.rp_filter=0
net.ipv4.conf.default.rp_filter=0
sysctl –p
开启openvswitch服务
serviceopenvswitch start
chkconfigopenvswitch on
添加OVS网桥设备
ovs-vsctladd-br br-int
ovs-vsctladd-br br-ex
添加br-ex 并修改eth0配置文件
ethtool -K eth0gro off
配置Neutron crm 脚本
crmconfigure
primitivep_vip_neutron ocf:heartbeat:IPaddr2 \
params ip="192.168.11.105"cidr_netmask="24" \
op monitor interval="30s"timeout="30s"
commit
exit
crmconfigure
primitivep_neutron-server ocf:openstack:neutron-server \
paramsconfig="/etc/neutron/neutron.conf" \
pid="/var/run/neutron/neutron.pid" \
os_tenant_name="admin" \
os_username="admin" \
os_password="111111" \
url="http://192.168.11.105:9696"\
keystone_get_token_url="http://192.168.11.101:5000/v2.0/tokens"\
op monitor interval="60s"timeout="60s"
primitivep_neutron-l3-agent ocf:openstack:neutron-l3-agent \
paramsconfig="/etc/neutron/neutron.conf" \
pid="/var/run/neutron/neutron-l3-agent.pid"\
neutron_server_port="5672" \
op monitor interval="60s"timeout="60s"
primitivep_neutron-dhcp-agent ocf:openstack:neutron-dhcp-agent \
paramsconfig="/etc/neutron/neutron.conf" \
pid="/var/run/neutron/neutron-dhcp-agent.pid" \
amqp_server_port="5672" \
op monitor interval="60s"timeout="60s"
primitivep_neutron-metadata-agent ocf:openstack:neutron-metadata-agent \
paramsconfig="/etc/neutron/neutron.conf" \
agent_config="/etc/neutron/metadata_agent.ini"\
pid="/var/run/neutron/neutron-metadata-agent.pid" \
op monitor interval="60s"timeout="60s"
primitivep_neutron-openvswitch-agent ocf:openstack:neutron-openvswitch-agent \
params config="/etc/neutron/neutron.conf"\
plugin_config="/etc/neutron/plugin.ini" \
pid="/var/run/neutron/neutron-openvswitch-agent.pid" \
segmentation_bridge="br-tun" \
op monitor interval="60s"timeout="60s"
groupg_neutron p_vip_neutron p_neutron-server p_neutron-l3-agentp_neutron-dhcp-agent p_neutron-metadata-agent p_neutron-openvswitch-agent
colocationcol_p_neutron-server_on_vip inf: p_neutron-server p_vip_neutron
orderord_vip_before_p_neutron-server inf: p_vip_neutron p_neutron-server
colocationcol_p_neutron-l3-agent_on_vip inf: p_neutron-l3-agent p_vip_neutron
orderord_vip_before_p_neutron-l3-agent inf: p_vip_neutron p_neutron-l3-agent
colocationcol_p_neutron-dhcp-agent_on_vip inf: p_neutron-dhcp-agent p_vip_neutron
orderord_vip_before_p_neutron-dhcp-agent inf: p_vip_neutron p_neutron-dhcp-agent
colocationcol_p_neutron-metadata-agent_on_vip inf: p_neutron-metadata-agent p_vip_neutron
orderord_vip_before_p_neutron-metadata-agent inf: p_vip_neutronp_neutron-metadata-agent
colocationcol_p_neutron-openvswitch-agent_on_vip inf: p_neutron-openvswitch-agentp_vip_neutron
orderord_vip_before_p_neutron-openvswitch-agent inf: p_vip_neutronp_neutron-openvswitch-agent
commit
exit
#################
# 开启资源 #
#################
crm resource start g_neutron
#################
# 停止资源 #
#################
crm resource stop g_neutron
#################
# 删除资源 #
#################
crm resource stop g_neutron
crm resource cleanup g_neutron
crm resource delete g_neutron
9 问题汇总
nova-novnc经常切换
vim/usr/lib/ocf/resource.d/openstack/nova-novnc
修改如下211行
202nova_vnc_console_monitor() {
203 local rc
204 local vnc_list_check
205
206 nova_vnc_console_status
207 rc=$?
208
209 # If status returned anything but success,return that immediately
210 if [ $rc -ne $OCF_SUCCESS ]; then
211 ocf_logerr "Nova VNC Console nova_vnc_console_monitor seem to be no PID filefound: $rc"
212 # return $rc
213 fi
/usr/lib/ocf/resource.d/openstack/nova-conductor增加支持多线程