OpenStack(四)——使用Kolla部署OpenStack多节点云平台(失败存根)

(1).实验环境

主机名 IP地址 角色 内存 网卡 CPU 磁盘
OpenStack-con 192.168.128.110 controller(控制) 8G 桥接网卡ens32和ens33 4核 200G

OpenStack-com

192.168.128.111 compute(计算) 4G 桥接网卡ens32 4核 20G
OpenStack-sto 192.168.128.112 storage(存储) 4G 桥接网卡ens32 4核 2个20G

  注意:OpenStack-con主机充当两个角色:kolla-ansible部署节点和controller节点。

(2).准备工作

  三台主机都安装<Tab>键补全软件包和vim软件包

yum -y install bash-completion.noarch vim

  三台主机都关闭firewalld和SELinux

# vim /etc/selinux/config
SELINUX=disabled
# setenforce 0
# getenforce
Permissive
# systemctl disable firewalld && systemctl stop firewalld
Removed symlink /etc/systemd/system/multi-user.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.
[root@OpenStack-con ~]# systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
   Active: inactive (dead)
     Docs: man:firewalld(1)

12月 27 10:43:55 OpenStack-con systemd[1]: Starting firewalld - dynamic firewall daemon...
12月 27 10:44:01 OpenStack-con systemd[1]: Started firewalld - dynamic firewall daemon.
12月 27 11:03:53 OpenStack-con systemd[1]: Stopping firewalld - dynamic firewall daemon...
12月 27 11:03:54 OpenStack-con systemd[1]: Stopped firewalld - dynamic firewall daemon.

  在OpenStack-con(110)上配置hosts文件,并复制到另两台主机上

[root@OpenStack-con ~]# vim /etc/hosts
192.168.128.110 OpenStack-con
192.168.128.111 OpenStack-com
192.168.128.112 OpenStack-sto
[root@OpenStack-con ~]# scp /etc/hosts 192.168.128.111:/etc/
[root@OpenStack-con ~]# scp /etc/hosts 192.168.128.112:/etc/

  修改OpenStack-con(110)的ens33配置文件

[root@OpenStack-con ~]# vim /etc/sysconfig/network-scripts/ifcfg-ens33
//只放以下内容
TYPE=Ethernet
BOOTPROTO=none  //静态获取IP
ONBOOT=yes  //启用网卡
PROXY_METHOD=none
NAME=ens33
DEVICE=ens33
[root@OpenStack-con ~]# systemctl restart network

(3).安装ansible

 1)在OpenStack-con(110)主机上安装并更新pip工具

//安装epel源
[root@OpenStack-con ~]# yum -y install epel-release
[root@OpenStack-con ~]# yum -y install python-pip
[root@OpenStack-con ~]# mkdir .pip
//配置pip软件包源
[root@OpenStack-con ~]# tee /root/.pip/pip.conf << 'EOF'
[global]
index-url = http://mirrors.aliyun.com/pypi/simple/
[install]
trusted-host=mirrors.aliyun.com
EOF
//更新pip
[root@OpenStack-con ~]# pip install -U pip

 2)配置OpenStack-com(111)和OpenStack-sto(112)pip软件包源

  注意:这一步是为了后期安装

//OpenStack-com(111)执行一次
[root@OpenStack-com ~]# mkdir .pip
[root@OpenStack-com ~]# tee /root/.pip/pip.conf << 'EOF'
[global]
index-url = http://mirrors.aliyun.com/pypi/simple/
[install]
trusted-host=mirrors.aliyun.com
EOF
//OpenStack-sto(112)执行一次
[root@OpenStack-sto ~]# mkdir .pip
[root@OpenStack-sto ~]# tee /root/.pip/pip.conf << 'EOF'
[global]
index-url = http://mirrors.aliyun.com/pypi/simple/
[install]
trusted-host=mirrors.aliyun.com
EOF

 3)在OpenStack-con(110)上安装ansible

//安装依赖包
[root@OpenStack-con ~]# yum -y install python-devel libffi-devel gcc openssl-devel libselinux-python
[root@OpenStack-con ~]# pip install ansible

 4)在OpenStack-con(110)上配置ansible参数

  注意:pip安装的ansible是没有配置文件的,此时需要前往github将默认配置文件拷贝下来。网址:https://github.com/ansible/ansible/blob/devel/examples/ansible.cfg

[root@OpenStack-con ~]# ansible --version  //可以看到此时是没有配置文件的
ansible 2.9.2
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, Aug  7 2019, 00:51:29) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]
//生成配置文件
[root@OpenStack-con ~]# mkdir /etc/ansible
[root@OpenStack-con ~]# vim /etc/ansible/ansible.cfg
[root@OpenStack-con ~]# ansible --version  //已经自动加载了配置文件
ansible 2.9.2
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/site-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.5 (default, Aug  7 2019, 00:51:29) [GCC 4.8.5 20150623 (Red Hat 4.8.5-39)]

  然后开始优化

[root@OpenStack-con ~]# vim /etc/ansible/ansible.cfg
forks           = 10  //第19行,设置并行进程数。如果要管理的主机很多,可以优先尝试增加该值
host_key_checking = False  //第67行,跳过ssh首次连接提示验证部分
pipelining = True  //第403行,开启管道输送。ansible在执行一个模块需要ssh到目的主机多次,开启该模式减少ssh连接次数,缩短ansible执行时间。
//在部署大规模服务器或引用模块非常多时,开启pipelining会给ansible带来显著的性能提升

(4).OpenStack-sto(112)配置cinder(块存储)信息

[root@OpenStack-sto ~]# yum -y install yum-utils device-mapper-persistent-data lvm2  //安装相关软件包
[root@OpenStack-sto ~]# pvs  //查看已经存在的pv
  PV         VG     Fmt  Attr PSize   PFree
  /dev/sda2  centos lvm2 a--  <19.00g    0 
[root@OpenStack-sto ~]# pvcreate /dev/sdb
  Physical volume "/dev/sdb" successfully created.
[root@OpenStack-sto ~]# vgcreate cinder /dev/sdb
  Volume group "cinder" successfully created
[root@OpenStack-sto ~]# systemctl status lvm2-lvmetad.service  //保证开机自启
● lvm2-lvmetad.service - LVM2 metadata daemon
   Loaded: loaded (/usr/lib/systemd/system/lvm2-lvmetad.service; static; vendor preset: enabled)
   Active: active (running) since 二 2019-12-17 15:26:11 CST; 20min ago
     Docs: man:lvmetad(8)
 Main PID: 11954 (lvmetad)
   CGroup: /system.slice/lvm2-lvmetad.service
           └─11954 /usr/sbin/lvmetad -f

12月 17 15:26:11 OpenStack-sto systemd[1]: Started LVM2 metadata daemon.
12月 17 15:26:11 OpenStack-sto systemd[1]: Starting LVM2 metadata daemon...

(5).在OpenStack-con(110)上安装kolla-ansible并自定义OpenStack的相关配置文件

 1)安装kolla-ansible

[root@OpenStack-con ~]# pip install kolla-ansible

 2)复制kolla-ansible的相关配置文件

[root@OpenStack-con ~]# cp -r /usr/share/kolla-ansible/etc_examples/kolla /etc/
[root@OpenStack-con ~]# cp /usr/share/kolla-ansible/ansible/inventory/* /etc/kolla/
[root@OpenStack-con ~]# ls /etc/kolla/
all-in-one  globals.yml  multinode  passwords.yml

  文件说明:all-in-one是安装单节点OpenStack的ansible自动安装配置文件;multinode是安装多节点OpenStack的ansible自动安装配置文件;globals.yml是OpenStack部署的自定义配置文件;passwords.yml是OpenStack中各个服务的密码文件。

 3)生成密钥,并授权三台主机

[root@OpenStack-con ~]# ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:MlEHvdjHadF+ydFC80Gg0u/sKcP+hvC8gDpvHTOGuL4 root@OpenStack-con
The key's randomart image is:
+---[RSA 2048]----+
|        oo.  +=o.|
|       . .o o o+o|
|      .  + = +..+|
|       .. + * .o.|
|      + S  o . . |
|     . +.*  o    |
|      ..o.O .o   |
|     o.. ..B...  |
|    .E=.  .o*+   |
+----[SHA256]-----+
[root@OpenStack-con ~]# ssh-copy-id -i .ssh/id_rsa.pub root@OpenStack-con
[root@OpenStack-con ~]# ssh-copy-id -i .ssh/id_rsa.pub root@OpenStack-com
[root@OpenStack-con ~]# ssh-copy-id -i .ssh/id_rsa.pub root@OpenStack-sto

 4)配置multinode多节点主机清单文件

[root@OpenStack-con ~]# vim /etc/kolla/multinode  //修改模块名不带:chilldren的部分
# These initial groups are the only groups required to be modified. The
# additional groups are for more control of the environment.
[control]  #控制模块
# These hostname must be resolvable from your deployment host
OpenStack-con  #给110主机

# The above can also be specified as follows:
#control[01:03]     ansible_user=kolla

# The network nodes are where your l3-agent and loadbalancers will run
# This can be the same as a host in the control group
[network]  #网络模块
OpenStack-con  #给110主机

[compute]  #计算模块
OpenStack-com  #给111主机

[monitoring]  #监控模块
OpenStack-con  #给110主机

# When compute nodes and control nodes use different interfaces,
# you need to comment out "api_interface" and other interfaces from the globals.yml
# and specify like below:
#compute01 neutron_external_interface=eth0 api_interface=em1 storage_interface=em1 tunnel_interface=em1

[storage]  #存储模块
OpenStack-sto  #给112主机

[deployment]  #部署模块
OpenStack-con  #给110主机

 5)检测所有主机是否正常通信

[root@OpenStack-con ~]# ansible -i /etc/kolla/multinode all -m ping
[DEPRECATION WARNING]: The TRANSFORM_INVALID_GROUP_CHARS settings is set to 
allow bad characters in group names by default, this will change, but still be 
user configurable on deprecation. This feature will be removed in version 2.10.
 Deprecation warnings can be disabled by setting deprecation_warnings=False in 
ansible.cfg.
[WARNING]: Invalid characters were found in group names but not replaced, use
-vvvv to see details

OpenStack-con | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python"
    }, 
    "changed": false, 
    "ping": "pong"
}
OpenStack-com | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python"
    }, 
    "changed": false, 
    "ping": "pong"
}
OpenStack-sto | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python"
    }, 
    "changed": false, 
    "ping": "pong"
}

 6)自动生成OpenStack各服务的密码文件

[root@OpenStack-con ~]# kolla-genpwd
[root@OpenStack-con ~]# vim /etc/kolla/passwords.yml
keystone_admin_password: 123456    //第164行,修改网页登录密码

 7)编辑/etc/kolla/global.yml自定义OpenStack中的部署事项

[root@OpenStack-con ~]# vim /etc/kolla/globals.yml
//第14行和第15行,选择下载的基础镜像,5选1
# Valid options are ['centos', 'debian', 'oraclelinux', 'rhel', 'ubuntu']
kolla_base_distro: "centos"
 
//第17行和第18行,选择的安装方法,2选1。binary二进制安装,source源码安装
# Valid options are [ binary, source ]
kolla_install_type: "source"
 
//第20行和第21行,选择OpenStack的版本标签,详细请看:https://releases.openstack.org/
# Valid option is Docker repository tag
openstack_release: "stein"  //注意版本必须小写,后期下载的OpenStack相关的docker镜像标签也为stein。我是train版本失败,才换成stein
 
//第23行和第24行,存放配置文件的位置
# Location of configuration overrides
#node_custom_config: "/etc/kolla/config"  //默认存放地址
 
//第31行,OpenStack内部管理网络地址,通过该IP访问OpenStack Web页面进行管理。如果启用了高可用,需要设置为VIP(漂移IP)
kolla_internal_vip_address: "192.168.128.110"
 
//第88行,OpenStack内部管理网络地址的网卡接口
network_interface: "ens32"

//第93~95、98~99行去除注释,使内部通信网络都走ens32
api_interface: "{{ network_interface }}"
storage_interface: "{{ network_interface }}"
cluster_interface: "{{ network_interface }}"
tunnel_interface: "{{ network_interface }}"
dns_interface: "{{ network_interface }}"
 
//第120行,OpenStack外部(或公共)网络的网卡接口,可以是vlan模式或flat模式。
//此网卡应该在没有IP地址的情况下处于活动,如果不是,那么OpenStack云平台中的云主机实例将无法访问外部网络。(存在IP时br-ex桥接就不成功)
neutron_external_interface: "ens33"
 
//第208行,关闭高可用
enable_haproxy: "no"

//第234行,启用cinder(块存储)
enable_cinder: "yes"

//第238行,cinder(块存储)后端启用lvm
enable_cinder_backend_lvm: "yes"

//第462行,cinder(块存储)的卷组名称,需要和OpenStack-sto主机上的一致
cinder_volume_group: "cinder"

//第497行和第498行,指定nova-compute守护进程使用的虚拟化技术。(kvm好像有点问题,大家可以试试,看看你们能不能过nova下载)
//nova-compute是一个非常重要的守护进程,负责创建和终止虚拟机实例,即管理虚拟机实例的生命周期
# Valid options are [ qemu, kvm, vmware, xenapi ]
nova_compute_virt_type: "qemu"

 8)通过kolla-ansible安装OpenStack所需依赖包

  注意:此时会对三台主机都进行操作,请保持网络畅通

[root@OpenStack-con ~]# kolla-ansible -i /etc/kolla/multinode bootstrap-servers
PLAY RECAP *********************************************************************
OpenStack-com              : ok=40   changed=23   unreachable=0    failed=0    skipped=32   rescued=0    ignored=0   
OpenStack-con              : ok=40   changed=21   unreachable=0    failed=0    skipped=32   rescued=0    ignored=0   
OpenStack-sto              : ok=40   changed=23   unreachable=0    failed=0    skipped=32   rescued=0    ignored=0   

 9)对主机进行预部署检查

[root@OpenStack-con ~]# kolla-ansible -i /etc/kolla/multinode prechecks
PLAY RECAP *********************************************************************
OpenStack-com              : ok=30   changed=0    unreachable=0    failed=0    skipped=22   rescued=0    ignored=0   
OpenStack-con              : ok=70   changed=0    unreachable=0    failed=0    skipped=77   rescued=0    ignored=0   
OpenStack-sto              : ok=23   changed=0    unreachable=0    failed=0    skipped=11   rescued=0    ignored=0

 10)编辑docker volume卷挂载方式,并指定docker加速器

  注意:三台主机都需要进行设置,设置方法一样

//由于不存在docker卷挂载配置文件,所以需要手动生成
# mkdir -p /etc/systemd/system/docker.service.d/
# vim /etc/systemd/system/docker.service.d/kolla.conf
[Service]
MountFlags=shared
//指定加速器,这里使用阿里云的加速器
# tee /etc/docker/daemon.json << 'EOF'
{
  "registry-mirrors": ["https://8mkqrctt.mirror.aliyuncs.com"]
}
EOF
# systemctl daemon-reload
# systemctl restart docker && systemctl enable docker

 11)拉取OpenStack的镜像

[root@OpenStack-con ~]# kolla-ansible -i /etc/kolla/multinode pull
PLAY RECAP *********************************************************************
OpenStack-com              : ok=18   changed=6    unreachable=0    failed=0    skipped=11   rescued=0    ignored=0   
OpenStack-con              : ok=37   changed=15   unreachable=0    failed=0    skipped=52   rescued=0    ignored=0   
OpenStack-sto              : ok=14   changed=4    unreachable=0    failed=0    skipped=4    rescued=0    ignored=0
[root@OpenStack-con ~]# docker images
REPOSITORY                                      TAG                 IMAGE ID            CREATED             SIZE
kolla/centos-source-horizon                     stein               e1251effe3cb        2 hours ago         1.04GB
kolla/centos-source-heat-engine                 stein               5c71a1370cb6        2 hours ago         895MB
kolla/centos-source-heat-api-cfn                stein               af03c22f8f12        2 hours ago         895MB
kolla/centos-source-heat-api                    stein               413ecad34b48        2 hours ago         895MB
kolla/centos-source-neutron-server              stein               fcee7f0b408d        2 hours ago         1.03GB
kolla/centos-source-neutron-l3-agent            stein               6067e9aaefe7        2 hours ago         1.04GB
kolla/centos-source-neutron-openvswitch-agent   stein               af9a9abe078f        2 hours ago         1GB
kolla/centos-source-neutron-dhcp-agent          stein               0cdb73c0c93a        2 hours ago         1GB
kolla/centos-source-neutron-metadata-agent      stein               81917e943f01        2 hours ago         1GB
kolla/centos-source-nova-api                    stein               c68ebcfb98aa        2 days ago          1.09GB
kolla/centos-source-nova-novncproxy             stein               a026c0d1309e        2 days ago          1.06GB
kolla/centos-source-glance-api                  stein               f8b487bd7770        2 days ago          911MB
kolla/centos-source-nova-scheduler              stein               5e98a6ff422d        2 days ago          1.03GB
kolla/centos-source-nova-conductor              stein               da65aeb3c40f        2 days ago          1.03GB
kolla/centos-source-placement-api               stein               f15badd1d2fc        2 days ago          920MB
kolla/centos-source-cinder-api                  stein               993d5361e1b6        2 days ago          1.09GB
kolla/centos-source-keystone-fernet             stein               83ba4f35ee57        2 days ago          921MB
kolla/centos-source-keystone-ssh                stein               d616e4544f1d        2 days ago          922MB
kolla/centos-source-cinder-scheduler            stein               3f06dc4ddb70        2 days ago          1.02GB
kolla/centos-source-keystone                    stein               bb43ae11542a        2 days ago          921MB
kolla/centos-source-openvswitch-db-server       stein               893e4aa4c7fa        2 days ago          424MB
kolla/centos-source-openvswitch-vswitchd        stein               c4fc759fbd81        2 days ago          424MB
kolla/centos-source-memcached                   stein               eff3e9a80876        2 days ago          407MB
kolla/centos-source-rabbitmq                    stein               7163a6f28c55        2 days ago          486MB
kolla/centos-source-kolla-toolbox               stein               6dd5d80b8505        2 days ago          688MB
kolla/centos-source-mariadb                     stein               cf42fce94264        2 days ago          594MB
kolla/centos-source-fluentd                     stein               de5a46efdcff        2 days ago          540MB
kolla/centos-source-chrony                      stein               6a81938d2b3b        2 days ago          407MB
kolla/centos-source-cron                        stein               5b83db419abb        2 days ago          406MB

[root@OpenStack-com ~]# docker images
REPOSITORY                                      TAG                 IMAGE ID            CREATED             SIZE
kolla/centos-source-nova-compute                stein               aac534e5f463        2 hours ago         1.85GB
kolla/centos-source-nova-ssh                    stein               095013bc0a50        2 hours ago         1.06GB
kolla/centos-source-neutron-openvswitch-agent   stein               af9a9abe078f        2 hours ago         1GB
kolla/centos-source-openvswitch-db-server       stein               893e4aa4c7fa        2 days ago          424MB
kolla/centos-source-openvswitch-vswitchd        stein               c4fc759fbd81        2 days ago          424MB
kolla/centos-source-nova-libvirt                stein               a8e165c353f5        2 days ago          1.2GB
kolla/centos-source-kolla-toolbox               stein               6dd5d80b8505        2 days ago          688MB
kolla/centos-source-fluentd                     stein               de5a46efdcff        2 days ago          540MB
kolla/centos-source-iscsid                      stein               7e9884caedc9        2 days ago          411MB
kolla/centos-source-chrony                      stein               6a81938d2b3b        2 days ago          407MB
kolla/centos-source-cron                        stein               5b83db419abb        2 days ago          406MB

[root@OpenStack-sto ~]# docker images
REPOSITORY                          TAG                 IMAGE ID            CREATED             SIZE
kolla/centos-source-cinder-volume   stein               a4bd8f7c0633        2 days ago          1.1GB
kolla/centos-source-cinder-backup   stein               6c69c3f19602        2 days ago          1.09GB
kolla/centos-source-kolla-toolbox   stein               6dd5d80b8505        2 days ago          688MB
kolla/centos-source-fluentd         stein               de5a46efdcff        2 days ago          540MB
kolla/centos-source-iscsid          stein               7e9884caedc9        2 days ago          411MB
kolla/centos-source-tgtd            stein               d9c831e2b3a7        2 days ago          406MB
kolla/centos-source-chrony          stein               6a81938d2b3b        2 days ago          407MB
kolla/centos-source-cron            stein               5b83db419abb        2 days ago          406MB

  拉取时,如果报错,可以尝试重新拉取。检查配置文件是否有问题,没有问题的前提下可以尝试换版本进行操作。

 12)部署OpenStack

  部署OpenStack时遇到了报错,暂时不知道怎么处理

[root@OpenStack-con ~]# kolla-ansible -i /etc/kolla/multinode deploy
TASK [mariadb : Creating haproxy mysql user] ********************************************************************
fatal: [OpenStack-con]: FAILED! => {
  "changed": false,
  "msg": "Can not parse the inner module output: localhost | SUCCESS => {\n    \"changed\": false, \n    \"user\": \"haproxy\"\n}\n"
}
[root@OpenStack-con ~]# docker ps -a
CONTAINER ID        IMAGE                                     COMMAND                  CREATED             STATUS              PORTS               NAMES
dbf2e410563a        kolla/centos-source-mariadb:stein         "dumb-init -- kolla_…"   4 days ago          Up About an hour                        mariadb
a4b2668bff0d        kolla/centos-source-chrony:stein          "dumb-init --single-…"   4 days ago          Up About an hour                        chrony
0e072ba208f9        kolla/centos-source-cron:stein            "dumb-init --single-…"   4 days ago          Up About an hour                        cron
36cd35ad03fb        kolla/centos-source-kolla-toolbox:stein   "dumb-init --single-…"   4 days ago          Up About an hour                        kolla_toolbox
0463ab847e1c        kolla/centos-source-fluentd:stein         "dumb-init --single-…"   4 days ago          Up About an hour                        fluentd
[root@OpenStack-con ~]# docker stop mariadb
mariadb
[root@OpenStack-con ~]# docker rm mariadb
mariadb
[root@OpenStack-con ~]# rm -rf /var/lib/docker/volumes/mariadb/_data/*

 13)验证部署

[root@OpenStack-con ~]# kolla-ansible -i /etc/kolla/multinode post-deploy
PLAY RECAP *********************************************************************
localhost : ok=2 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
[root@OpenStack-con ~]# cat /etc/kolla/admin-openrc.sh  //会生成该文件,查看一下
# Clear any old environment that may conflict.
for key in $( set | awk '{FS="="}  /^OS_/ {print $1}' ); do unset $key ; done
export OS_PROJECT_DOMAIN_NAME=Default
export OS_USER_DOMAIN_NAME=Default
export OS_PROJECT_NAME=admin
export OS_TENANT_NAME=admin
export OS_USERNAME=admin  //网页的账号
export OS_PASSWORD=123456  //网页的密码
export OS_AUTH_URL=http://192.168.128.110:35357/v3
export OS_INTERFACE=internal
export OS_ENDPOINT_TYPE=internalURL
export OS_IDENTITY_API_VERSION=3
export OS_REGION_NAME=RegionOne
export OS_AUTH_PLUGIN=password

  理论上这样就可以使用了,但是卡住了,暂时不知道怎么解决。

猜你喜欢

转载自www.cnblogs.com/diantong/p/12010193.html