- 现象：Volume Type lvmdriver-1 already exists. (HTTP 409)
问题原因： 控制节点在执行devstack时已经创建了一个名为 `lvmdriver-1` 的cinder type。
问题解决：修改`/opt/stack/devstack/lib/cinder`文件551行，改动如下：
# 修改前
if is_service_enabled c-api && [[ -n "$CINDER_ENABLED_BACKENDS" ]]; then
local be be_name
for be in ${CINDER_ENABLED_BACKENDS//,/ }; do
be_name=${be##*:}
openstack --os-region-name="$REGION_NAME" volume type create --property volume_backend_name="${be_name}" ${be_name}
done
fi
# 修改后
if is_service_enabled c-api && [[ -n "$CINDER_ENABLED_BACKENDS" ]]; then
local be be_name
for be in ${CINDER_ENABLED_BACKENDS//,/ }; do
be_name=${be##*:}
openstack volume type show $be_name || \
openstack --os-region-name="$REGION_NAME" volume type create \
--property volume_backend_name="${be_name}" ${be_name}
done
fi
- 现象：Didn't find service registered by hostname after 120 seconds
# 问题原因：配置文件local.conf中rabbit密码错误导致rabbit连接失败
# 问题解决：修改local.conf文件，如下：
# Credentials
ADMIN_PASSWORD=devstack
MYSQL_PASSWORD=root
RABBIT_PASSWORD=password
SERVICE_PASSWORD=devstack
SERVICE_TOKEN=devstack
- 现象：DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on u'127.0.0.1' ([Errno 111] ECONNREFUSED)")
# 具体日志信息：
2018-05-03 09:30:06.828 | /usr/local/lib/python2.7/dist-packages/pymysql/cursors.py:166: Warning: (1287, u"'@@tx_isolation' is deprecated and will be removed in a future release. Please use '@@transaction_isolation' instead")
2018-05-03 09:30:06.828 | result = self._query(query)
2018-05-03 09:30:07.085 | Found 2 cell mappings.
2018-05-03 09:30:07.085 | Skipping cell0 since it does not contain hosts.
2018-05-03 09:30:07.085 | Getting computes from cell 'cell1': 09b55e68-82af-4049-87e0-5036a4c0809b
2018-05-03 09:30:07.085 | WARNING oslo_db.sqlalchemy.engines [None req-07242c5d-e19b-4809-96aa-a7efda12d21a None None] SQL connection failed. 10 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on u'127.0.0.1' ([Errno 111] ECONNREFUSED)")
2018-05-03 09:30:17.087 | WARNING oslo_db.sqlalchemy.engines [None req-07242c5d-e19b-4809-96aa-a7efda12d21a None None] SQL connection failed. 9 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on u'127.0.0.1' ([Errno 111] ECONNREFUSED)")
2018-05-03 09:30:27.089 | WARNING oslo_db.sqlalchemy.engines [None req-07242c5d-e19b-4809-96aa-a7efda12d21a None None] SQL connection failed. 8 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on u'127.0.0.1' ([Errno 111] ECONNREFUSED)")
2018-05-03 09:30:37.100 | WARNING oslo_db.sqlalchemy.engines [None req-07242c5d-e19b-4809-96aa-a7efda12d21a None None] SQL connection failed. 7 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on u'127.0.0.1' ([Errno 111] ECONNREFUSED)")
2018-05-03 09:30:47.110 | WARNING oslo_db.sqlalchemy.engines [None req-07242c5d-e19b-4809-96aa-a7efda12d21a None None] SQL connection failed. 6 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on u'127.0.0.1' ([Errno 111] ECONNREFUSED)")
2018-05-03 09:30:57.118 | WARNING oslo_db.sqlalchemy.engines [None req-07242c5d-e19b-4809-96aa-a7efda12d21a None None] SQL connection failed. 5 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on u'127.0.0.1' ([Errno 111] ECONNREFUSED)")
2018-05-03 09:31:07.123 | WARNING oslo_db.sqlalchemy.engines [None req-07242c5d-e19b-4809-96aa-a7efda12d21a None None] SQL connection failed. 4 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on u'127.0.0.1' ([Errno 111] ECONNREFUSED)")
2018-05-03 09:31:17.129 | WARNING oslo_db.sqlalchemy.engines [None req-07242c5d-e19b-4809-96aa-a7efda12d21a None None] SQL connection failed. 3 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on u'127.0.0.1' ([Errno 111] ECONNREFUSED)")
2018-05-03 09:31:27.139 | WARNING oslo_db.sqlalchemy.engines [None req-07242c5d-e19b-4809-96aa-a7efda12d21a None None] SQL connection failed. 2 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on u'127.0.0.1' ([Errno 111] ECONNREFUSED)")
2018-05-03 09:31:37.150 | WARNING oslo_db.sqlalchemy.engines [None req-07242c5d-e19b-4809-96aa-a7efda12d21a None None] SQL connection failed. 1 attempts left.: DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on u'127.0.0.1' ([Errno 111] ECONNREFUSED)")
2018-05-03 09:31:47.162 | An error has occurred:
2018-05-03 09:31:47.162 | Traceback (most recent call last):
2018-05-03 09:31:47.162 | File "/opt/stack/nova/nova/cmd/manage.py", line 1868, in main
2018-05-03 09:31:47.162 | ret = fn(*fn_args, **fn_kwargs)
2018-05-03 09:31:47.162 | File "/opt/stack/nova/nova/cmd/manage.py", line 1588, in discover_hosts
2018-05-03 09:31:47.163 | by_service)
2018-05-03 09:31:47.163 | File "/opt/stack/nova/nova/objects/host_mapping.py", line 257, in discover_hosts
2018-05-03 09:31:47.163 | by_service)
2018-05-03 09:31:47.163 | File "/opt/stack/nova/nova/objects/host_mapping.py", line 219, in _check_and_create_host_mappings
2018-05-03 09:31:47.163 | ctxt, 1)
2018-05-03 09:31:47.163 | File "/usr/local/lib/python2.7/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper
2018-05-03 09:31:47.163 | result = fn(cls, context, *args, **kwargs)
2018-05-03 09:31:47.163 | File "/opt/stack/nova/nova/objects/compute_node.py", line 397, in get_all_by_not_mapped
2018-05-03 09:31:47.163 | context, mapped_less_than)
2018-05-03 09:31:47.163 | File "/opt/stack/nova/nova/db/api.py", line 273, in compute_node_get_all_mapped_less_than
2018-05-03 09:31:47.163 | mapped_less_than)
2018-05-03 09:31:47.163 | File "/opt/stack/nova/nova/db/sqlalchemy/api.py", line 264, in wrapped
2018-05-03 09:31:47.163 | with ctxt_mgr.reader.using(context):
2018-05-03 09:31:47.163 | File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
2018-05-03 09:31:47.163 | return self.gen.next()
2018-05-03 09:31:47.163 | File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 1028, in _transaction_scope
2018-05-03 09:31:47.163 | context=context) as resource:
2018-05-03 09:31:47.163 | File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
2018-05-03 09:31:47.163 | return self.gen.next()
2018-05-03 09:31:47.163 | File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 633, in _session
2018-05-03 09:31:47.163 | bind=self.connection, mode=self.mode)
2018-05-03 09:31:47.163 | File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 398, in _create_session
2018-05-03 09:31:47.163 | self._start()
2018-05-03 09:31:47.164 | File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 484, in _start
2018-05-03 09:31:47.164 | engine_args, maker_args)
2018-05-03 09:31:47.164 | File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 508, in _setup_for_connection
2018-05-03 09:31:47.164 | sql_connection=sql_connection, **engine_kwargs)
2018-05-03 09:31:47.164 | File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py", line 179, in create_engine
2018-05-03 09:31:47.164 | test_conn = _test_connection(engine, max_retries, retry_interval)
2018-05-03 09:31:47.164 | File "/usr/local/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py", line 365, in _test_connection
2018-05-03 09:31:47.164 | six.reraise(type(de_ref), de_ref)
2018-05-03 09:31:47.164 | File "<string>", line 2, in reraise
2018-05-03 09:31:47.164 | DBConnectionError: (pymysql.err.OperationalError) (2003, "Can't connect to MySQL server on u'127.0.0.1' ([Errno 111] ECONNREFUSED)")
2018-05-03 09:31:47.164 |
2018-05-03 09:31:47.268 | ++./stack.sh:main:1461 err_trap
2018-05-03 09:31:47.274 | ++./stack.sh:err_trap:562 local r=1
2018-05-03 09:31:47.280 | stack.sh failed: full log in /opt/stack/logs/stack.sh.log.2018-05-03-052802
2018-05-03 09:31:47.283 | Error on exit
# 问题原因：
计算节点应该连接的是控制节点的数据库，但是这里出现"Can't connect to MySQL server on u'127.0.0.1'"表明计算节点在连接本地数据库，然而本地又没有安装mysql数据库，所以本质原因是nova配置的数据库连接选项有问题。具体哪里配置有问题不清楚，网上发现有相同的bug，https://lists.launchpad.net/yahoo-eng-team/msg70399.html 和 https://bugzilla.redhat.com/show_bug.cgi?id=1443214
参见上述网页，在控制节点执行下列命令：
stack@controller:/opt/stack/devstack$ nova-manage cell_v2 list_cells
结果如下：
/usr/local/lib/python2.7/dist-packages/pymysql/cursors.py:166: Warning: (1287, u"'@@tx_isolation' is deprecated and will be removed in a future release. Please use '@@transaction_isolation' instead")
result = self._query(query)
+-------+--------------------------------------+------------------------------------------------+------------------------------------------------------------------+
| Name | UUID | Transport URL | Database Connection |
+-------+--------------------------------------+------------------------------------------------+------------------------------------------------------------------+
| cell0 | 00000000-0000-0000-0000-000000000000 | none:/ | mysql+pymysql://root:****@127.0.0.1/nova_cell0?charset=utf8 |
| cell1 | 220faf2d-fab8-4c47-bf02-4270f1e380be | rabbit://stackrabbit:****@192.168.100.30:5672/ | mysql+pymysql://root:****@127.0.0.1/nova_cell1?charset=utf8 |
+-------+--------------------------------------+------------------------------------------------+------------------------------------------------------------------+
可以发现控制节点cell1的数据库的接入ip变成127.0.0.1，正常情况下应该是控制节点的管理网ip，也即192.168.100.30。
# 解决办法：控制节点执行下列命令
stack@controller:/opt/stack/devstack$ nova-manage cell_v2 update_cell --cell_uuid 220faf2d-fab8-4c47-bf02-4270f1e380be --database_connection mysql+pymysql://root:****@192.168.100.30/nova_cell1?charset=utf8
stack@controller:/opt/stack/devstack$ nova-manage cell_v2 update_cell --cell_uuid 00000000-0000-0000-0000-000000000000 --database_connection mysql+pymysql://root:****@192.168.100.30/nova_cell0?charset=utf8
- 现象：OperationalError: (pymysql.err.OperationalError) (1045, u"Access denied for user 'root'@'192.168.100.30' (using password: YES)")
# 问题原因：出现"Access denied for user XXXX" 一般可能是两方面的原因：1）mysql数据库接入权限的问题；2）用户名和密码错误；
这里出现问题的本质原因是用户名和密码错误，具体是在上一个解决办法中将数据库接入的密码改成了****
通过在控制节点执行下列命令可以发现：
stack@controller:/opt/stack/devstack$ nova-manage cell_v2 list_cells --verbose
/usr/local/lib/python2.7/dist-packages/pymysql/cursors.py:166: Warning: (1287, u"'@@tx_isolation' is deprecated and will be removed in a future release. Please use '@@transaction_isolation' instead")
result = self._query(query)
+-------+--------------------------------------+----------------------------------------------------+------------------------------------------------------------------+
| Name | UUID | Transport URL | Database Connection |
+-------+--------------------------------------+----------------------------------------------------+------------------------------------------------------------------+
| cell0 | 00000000-0000-0000-0000-000000000000 | none:/// | mysql+pymysql://root:[email protected]/nova_cell0?charset=utf8 |
| cell1 | 220faf2d-fab8-4c47-bf02-4270f1e380be | rabbit://stackrabbit:[email protected]:5672/ | mysql+pymysql://root:****@192.168.100.30/nova_cell1?charset=utf8 |
+-------+--------------------------------------+----------------------------------------------------+------------------------------------------------------------------+
# 问题解决：控制节点执行下列命令
stack@controller:/opt/stack/devstack$ nova-manage cell_v2 update_cell --cell_uuid 220faf2d-fab8-4c47-bf02-4270f1e380be --database_connection mysql+pymysql://root:[email protected]/nova_cell1?charset=utf8
stack@controller:/opt/stack/devstack$ nova-manage cell_v2 update_cell --cell_uuid 00000000-0000-0000-0000-000000000000 --database_connection mysql+pymysql://root:[email protected]/nova_cell0?charset=utf8
- 现象： 创建成功后的虚拟机无法进入控制台，failed to connect to server (code: 1006)
# 问题原因：
1. 在devstack安装过程后可以通过如下命令查看各个服务：
sinet@compute1:~$ systemctl list-units | grep devstack
[email protected] loaded active running Devstack [email protected]
[email protected] loaded active running Devstack [email protected]
[email protected] loaded active running Devstack [email protected]
[email protected] loaded active running Devstack [email protected]
[email protected] loaded active running Devstack [email protected]
● [email protected] not-found failed failed [email protected]
[email protected] loaded active running Devstack [email protected]
[email protected] loaded active running Devstack [email protected]
system-devstack.slice loaded active active system-devstack.slice
2. 可以通过命令 sudo journalctl -u devstack@n-novnc | grep ERROR 查看日志报错选项：
sinet@controller:~$ sudo journalctl -u devstack@n-novnc | grep ERROR
[sudo] password for sinet:
May 04 22:35:55 controller nova-novncproxy[5933]: ERROR nova.console.websocketproxy Traceback (most recent call last):
May 04 22:35:55 controller nova-novncproxy[5933]: ERROR nova.console.websocketproxy File "/usr/local/lib/python2.7/dist-packages/websockify/websocket.py", line 930, in top_new_client
May 04 22:35:55 controller nova-novncproxy[5933]: ERROR nova.console.websocketproxy client = self.do_handshake(startsock, address)
May 04 22:35:55 controller nova-novncproxy[5933]: ERROR nova.console.websocketproxy File "/usr/local/lib/python2.7/dist-packages/websockify/websocket.py", line 860, in do_handshake
May 04 22:35:55 controller nova-novncproxy[5933]: ERROR nova.console.websocketproxy self.RequestHandlerClass(retsock, address, self)
May 04 22:35:55 controller nova-novncproxy[5933]: ERROR nova.console.websocketproxy File "/opt/stack/nova/nova/console/websocketproxy.py", line 176, in __init__
May 04 22:35:55 controller nova-novncproxy[5933]: ERROR nova.console.websocketproxy websockify.ProxyRequestHandler.__init__(self, *args, **kwargs)
May 04 22:35:55 controller nova-novncproxy[5933]: ERROR nova.console.websocketproxy File "/usr/local/lib/python2.7/dist-packages/websockify/websocket.py", line 114, in __init__
May 04 22:35:55 controller nova-novncproxy[5933]: ERROR nova.console.websocketproxy SimpleHTTPRequestHandler.__init__(self, req, addr, server)
May 04 22:35:55 controller nova-novncproxy[5933]: ERROR nova.console.websocketproxy File "/usr/lib/python2.7/SocketServer.py", line 652, in __init__
May 04 22:35:55 controller nova-novncproxy[5933]: ERROR nova.console.websocketproxy self.handle()
May 04 22:35:55 controller nova-novncproxy[5933]: ERROR nova.console.websocketproxy File "/usr/local/lib/python2.7/dist-packages/websockify/websocket.py", line 581, in handle
May 04 22:35:55 controller nova-novncproxy[5933]: ERROR nova.console.websocketproxy SimpleHTTPRequestHandler.handle(self)
May 04 22:35:55 controller nova-novncproxy[5933]: ERROR nova.console.websocketproxy File "/usr/lib/python2.7/BaseHTTPServer.py", line 340, in handle
May 04 22:35:55 controller nova-novncproxy[5933]: ERROR nova.console.websocketproxy self.handle_one_request()
May 04 22:35:55 controller nova-novncproxy[5933]: ERROR nova.console.websocketproxy File "/usr/lib/python2.7/BaseHTTPServer.py", line 328, in handle_one_request
May 04 22:35:55 controller nova-novncproxy[5933]: ERROR nova.console.websocketproxy method()
May 04 22:35:55 controller nova-novncproxy[5933]: ERROR nova.console.websocketproxy File "/usr/local/lib/python2.7/dist-packages/websockify/websocket.py", line 543, in do_GET
May 04 22:35:55 controller nova-novncproxy[5933]: ERROR nova.console.websocketproxy if not self.handle_websocket():
May 04 22:35:55 controller nova-novncproxy[5933]: ERROR nova.console.websocketproxy File "/usr/local/lib/python2.7/dist-packages/websockify/websocket.py", line 531, in handle_websocket
May 04 22:35:55 controller nova-novncproxy[5933]: ERROR nova.console.websocketproxy self.new_websocket_client()
May 04 22:35:55 controller nova-novncproxy[5933]: ERROR nova.console.websocketproxy File "/opt/stack/nova/nova/console/websocketproxy.py", line 105, in new_websocket_client
May 04 22:35:55 controller nova-novncproxy[5933]: ERROR nova.console.websocketproxy connect_info = rpcapi.check_token(ctxt, token=token)
May 04 22:35:55 controller nova-novncproxy[5933]: ERROR nova.console.websocketproxy File "/opt/stack/nova/nova/consoleauth/rpcapi.py", line 97, in check_token
May 04 22:35:55 controller nova-novncproxy[5933]: ERROR nova.console.websocketproxy return cctxt.call(ctxt, 'check_token', token=token)
May 04 22:35:55 controller nova-novncproxy[5933]: ERROR nova.console.websocketproxy File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 169, in call
May 04 22:35:55 controller nova-novncproxy[5933]: ERROR nova.console.websocketproxy retry=self.retry)
May 04 22:35:55 controller nova-novncproxy[5933]: ERROR nova.console.websocketproxy File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 123, in _send
May 04 22:35:55 controller nova-novncproxy[5933]: ERROR nova.console.websocketproxy timeout=timeout, retry=retry)
May 04 22:35:55 controller nova-novncproxy[5933]: ERROR nova.console.websocketproxy File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 566, in send
May 04 22:35:55 controller nova-novncproxy[5933]: ERROR nova.console.websocketproxy retry=retry)
May 04 22:35:55 controller nova-novncproxy[5933]: ERROR nova.console.websocketproxy File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 555, in _send
May 04 22:35:55 controller nova-novncproxy[5933]: ERROR nova.console.websocketproxy result = self._waiter.wait(msg_id, timeout)
May 04 22:35:55 controller nova-novncproxy[5933]: ERROR nova.console.websocketproxy File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 447, in wait
May 04 22:35:55 controller nova-novncproxy[5933]: ERROR nova.console.websocketproxy message = self.waiters.get(msg_id, timeout=timeout)
May 04 22:35:55 controller nova-novncproxy[5933]: ERROR nova.console.websocketproxy File "/usr/local/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py", line 335, in get
May 04 22:35:55 controller nova-novncproxy[5933]: ERROR nova.console.websocketproxy 'to message ID %s' % msg_id)
May 04 22:35:55 controller nova-novncproxy[5933]: ERROR nova.console.websocketproxy MessagingTimeout: Timed out waiting for a reply to message ID f6b67778a9e942f2adeaad068a408ece
May 04 22:35:55 controller nova-novncproxy[5933]: ERROR nova.console.websocketproxy
查看日志发现应该是配置文件的问题，具体是etc/nova/nova.conf配置文件中“vncserver_proxyclient_address”参数是本地IP，而不是主机的IP；
# 解决办法：
1. 修改控制节点的local.conf 配置文件，增加如下内容：
# vnc config
NOVA_VNC_ENABLED=True
NOVNCPROXY_URL="http://$SERVICE_HOST:6080/vnc_auto.html"
VNCSERVER_LISTEN=$HOST_IP
VNCSERVER_PROXYCLIENT_ADDRESS=$VNCSERVER_LISTEN
2. 修改计算节点的local.conf 配置文件如下：
# vnc config
NOVA_VNC_ENABLED=True
NOVNCPROXY_URL="http://192.168.100.30:6080/vnc_auto.html"
VNCSERVER_LISTEN=0.0.0.0
VNCSERVER_PROXYCLIENT_ADDRESS=$HOST_IP