1、移除计算节点上的ceph osd
停止节点上所有的osd服务
systemctl stop ceph-osd@20
systemctl stop ceph-osd@21
systemctl stop ceph-osd@22
systemctl stop ceph-osd@23
在mon节点上移除osd
ceph osd out 23
ceph osd out 22
ceph osd out 21
ceph osd out 20
ceph osd crush remove osd.23
ceph osd crush remove osd.22
ceph osd crush remove osd.21
ceph osd crush remove osd.20
ceph auth del osd.23
ceph auth del osd.22
ceph auth del osd.21
ceph auth del osd.20
ceph osd rm 23
ceph osd rm 22
ceph osd rm 21
ceph osd rm 20
ceph osd crush remove bdc8
等待集群自动恢复即可 如果配置文件中有对应osd在某台节点上的配置，需要删掉
2、移除计算节点上openstack服务
2.1、计算节点上停止服务
systemctl stop neutron-openvswitch-agent.service
systemctl stop openstack-nova-compute.service
systemctl disable neutron-openvswitch-agent.service
systemctl disable openstack-nova-compute.service
2.2、移除计算服务
查看计算节点服务
. admin-openrc.sh
nova service-list
+----+------------------+------+----------+---------+-------+----------------------------+-----------------+
| Id | Binary | Host | Zone | Status | State | Updated_at | Disabled Reason |
+----+------------------+------+----------+---------+-------+----------------------------+-----------------+
| 1 | nova-cert | bdc2 | internal | enabled | up | 2018-03-22T02:54:42.000000 | - |
| 2 | nova-conductor | bdc2 | internal | enabled | up | 2018-03-22T02:54:43.000000 | - |
| 3 | nova-consoleauth | bdc2 | internal | enabled | up | 2018-03-22T02:54:42.000000 | - |
| 4 | nova-scheduler | bdc2 | internal | enabled | up | 2018-03-22T02:54:40.000000 | - |
| 5 | nova-compute | bdc3 | nova | enabled | up | 2018-03-22T02:54:41.000000 | - |
| 6 | nova-compute | bdc2 | nova | enabled | up | 2018-03-22T02:54:45.000000 | - |
| 7 | nova-compute | bdc4 | nova | enabled | up | 2018-03-22T02:54:42.000000 | - |
| 8 | nova-compute | bdc5 | nova | enabled | up | 2018-03-22T02:54:44.000000 | - |
| 9 | nova-compute | bdc6 | nova | enabled | up | 2018-03-22T02:54:44.000000 | - |
| 10 | nova-compute | bdc7 | nova | enabled | up | 2018-03-22T02:54:46.000000 | - |
| 11 | nova-compute | bdc8 | nova | enabled | up | 2018-03-22T02:54:39.000000 | - |
+----+------------------+------+----------+---------+-------+----------------------------+-----------------+
删除bdc8上的计算服务
nova service-delete 11
2.3、移除网络服务
查看网络服务
# neutron agent-list
+--------------------------------------+--------------------+------+-------+----------------+---------------------------+
| id | agent_type | host | alive | admin_state_up | binary |
+--------------------------------------+--------------------+------+-------+----------------+---------------------------+
| 08bc9d33-3e67-4a06-9223-74be1c969b8a | Open vSwitch agent | bdc2 | :-) | True | neutron-openvswitch-agent |
| 19a31ea2-280b-42b4-9f08-c1f99182e826 | Open vSwitch agent | bdc5 | :-) | True | neutron-openvswitch-agent |
| 286336ae-22fd-4391-8470-aae559ee5ccb | DHCP agent | bdc3 | :-) | True | neutron-dhcp-agent |
| 42ab605e-3d66-402f-b86a-4ef246df2ecf | Open vSwitch agent | bdc8 | :-) | True | neutron-openvswitch-agent |
| 4791844d-07f7-43be-8b27-bcfe8d8a7791 | Open vSwitch agent | bdc6 | :-) | True | neutron-openvswitch-agent |
| 63c99510-205a-4f94-ac18-61082c6edb44 | L3 agent | bdc3 | :-) | True | neutron-l3-agent |
| 830a75fb-c516-4f6f-ae74-e09bfd558291 | Open vSwitch agent | bdc7 | :-) | True | neutron-openvswitch-agent |
| 91df9210-0a27-4933-9f03-3ccce6192011 | Open vSwitch agent | bdc4 | :-) | True | neutron-openvswitch-agent |
| a9f7bc30-beda-437d-92a8-be64296c57d0 | Open vSwitch agent | bdc3 | :-) | True | neutron-openvswitch-agent |
| cf6443f7-ef8e-4738-bfc5-e491e4f89aeb | Metadata agent | bdc3 | :-) | True | neutron-metadata-agent |
+--------------------------------------+--------------------+------+-------+----------------+---------------------------+
删除bdc8上的网络服务
# neutron agent-delete 42ab605e-3d66-402f-b86a-4ef246df2ecf
Deleted agent: 42ab605e-3d66-402f-b86a-4ef246df2ecf
3、元数据查看
nova库
MariaDB [nova]> select hypervisor_hostname,deleted_at,deleted from compute_nodes;
+---------------------+---------------------+---------+
| hypervisor_hostname | deleted_at | deleted |
+---------------------+---------------------+---------+
| bdc3 | NULL | 0 |
| bdc2 | NULL | 0 |
| bdc4 | NULL | 0 |
| bdc5 | NULL | 0 |
| bdc6 | NULL | 0 |
| bdc7 | NULL | 0 |
| bdc8 | 2018-03-22 02:58:53 | 7 |
+---------------------+---------------------+---------+
MariaDB [nova]> select * from services where host = 'bdc8';
+---------------------+---------------------+---------------------+----+------+--------------+---------+--------------+----------+---------+-----------------+
| created_at | updated_at | deleted_at | id | host | binary | topic | report_count | disabled | deleted | disabled_reason |
+---------------------+---------------------+---------------------+----+------+--------------+---------+--------------+----------+---------+-----------------+
| 2016-10-13 09:20:41 | 2018-03-22 02:58:49 | 2018-03-22 02:58:53 | 11 | bdc8 | nova-compute | compute | 4442289 | 0 | 11 | NULL |
+---------------------+---------------------+---------------------+----+------+--------------+---------+--------------+----------+---------+-----------------+
4、删除其它节点与bdc8连通的隧道网络
先在元数据库中删除bdc8的gre信息
MariaDB [neutron]> select * from ml2_gre_endpoints;
+-------------+------+
| ip_address | host |
+-------------+------+
| 172.16.13.2 | bdc2 |
| 172.16.13.3 | bdc3 |
| 172.16.13.4 | bdc4 |
| 172.16.13.5 | bdc5 |
| 172.16.13.6 | bdc6 |
| 172.16.13.7 | bdc7 |
| 172.16.13.8 | bdc8 |
+-------------+------+
MariaDB [neutron]> delete from ml2_gre_endpoints where ip_address='172.16.13.8';
各个节点中删除连接bdc8的gre隧道
# ovs-vsctl show
8d6ea1e1-9b90-4e6b-94ae-d5a04601daf9
Bridge br-tun
fail_mode: secure
Port "gre-ac100d03"
Interface "gre-ac100d03"
type: gre
options: {df_default="true", in_key=flow, local_ip="172.16.13.2", out_key=flow, remote_ip="172.16.13.3"}
Port "gre-ac100d07"
Interface "gre-ac100d07"
type: gre
options: {df_default="true", in_key=flow, local_ip="172.16.13.2", out_key=flow, remote_ip="172.16.13.7"}
Port "gre-ac100d08"
Interface "gre-ac100d08"
type: gre
options: {df_default="true", in_key=flow, local_ip="172.16.13.2", out_key=flow, remote_ip="172.16.13.8"}
Port "gre-ac100d05"
Interface "gre-ac100d05"
type: gre
options: {df_default="true", in_key=flow, local_ip="172.16.13.2", out_key=flow, remote_ip="172.16.13.5"}
Port "gre-ac100d06"
Interface "gre-ac100d06"
type: gre
options: {df_default="true", in_key=flow, local_ip="172.16.13.2", out_key=flow, remote_ip="172.16.13.6"}
Port patch-int
Interface patch-int
type: patch
options: {peer=patch-tun}
Port "gre-ac100d04"
Interface "gre-ac100d04"
type: gre
options: {df_default="true", in_key=flow, local_ip="172.16.13.2", out_key=flow, remote_ip="172.16.13.4"}
Port br-tun
Interface br-tun
type: internal
Bridge br-int
fail_mode: secure
Port "qvo86bf0e09-3f"
tag: 2
Interface "qvo86bf0e09-3f"
Port "qvo73d3f4d9-72"
tag: 2
Interface "qvo73d3f4d9-72"
Port br-int
Interface br-int
type: internal
Port "qvo7c5fbeca-ab"
tag: 2
Interface "qvo7c5fbeca-ab"
Port "qvo2364aa3b-98"
tag: 4
Interface "qvo2364aa3b-98"
Port "qvo0194c615-74"
tag: 5
Interface "qvo0194c615-74"
Port patch-tun
Interface patch-tun
type: patch
options: {peer=patch-int}
Port "qvo4a30d4d8-e3"
tag: 3
Interface "qvo4a30d4d8-e3"
Port "qvo1d912409-ed"
tag: 2
Interface "qvo1d912409-ed"
Port "qvoe79c563d-50"
tag: 1
Interface "qvoe79c563d-50"
Port "qvo4b146bb0-52"
tag: 3
Interface "qvo4b146bb0-52"
ovs_version: "2.5.0"
所有节点上连接bdc8的port一样，全都需要删除
# ovs-vsctl del-port br-tun gre-ac100d08