Openstack平台虚拟机疏散失败提示(pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query')问题

其他 2020-05-15 10:30:47 阅读次数: 0

一.问题的表象

过程:创建完windowserver2012r2版本虚拟机，挂载cinder卷，写入一定的数据
问题:关闭虚拟机所在的宿主机，执行疏散操作，虚拟机疏散失败

二.问题日志

  • 迁移的目的服务器nova-compute的日志信息如下：
2018-11-28 10:12:01.120 7 ERROR nova.volume.cinder [req-8d60395f-5c21-4699-bd5e-31b921acb378 90dea4aefe0c44e19cea9b3cbf0a37c2 3c4c8987959d465285c6e5e1db72e5a5 - default default] Delete attachment failed for attachment e6b83198-8286-4925-b1d8-1b09b0ae9f79. Error: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-9f348d0a-6afd-4ab9-a11f-de2aae655ca6) Code: 500: ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-9f348d0a-6afd-4ab9-a11f-de2aae655ca6)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server [req-8d60395f-5c21-4699-bd5e-31b921acb378 90dea4aefe0c44e19cea9b3cbf0a37c2 3c4c8987959d465285c6e5e1db72e5a5 - default default] Exception during message handling: ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-9f348d0a-6afd-4ab9-a11f-de2aae655ca6)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 7829, in detach_volume
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     attachment_id)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/exception_wrapper.py", line 76, in wrapped
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     function_name, call_dict, binary)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     self.force_reraise()
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/exception_wrapper.py", line 67, in wrapped
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     return f(self, context, *args, **kw)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/utils.py", line 976, in decorated_function
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 214, in decorated_function
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     kwargs['instance'], e, sys.exc_info())
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     self.force_reraise()
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 202, in decorated_function
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     return function(self, context, *args, **kwargs)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 5436, in detach_volume
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     attachment_id=attachment_id)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/compute/manager.py", line 5389, in _detach_volume
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     attachment_id=attachment_id, destroy_bdm=destroy_bdm)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/block_device.py", line 415, in detach
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     attachment_id, destroy_bdm)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_concurrency/lockutils.py", line 274, in inner
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/block_device.py", line 412, in _do_locked_detach
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     self._do_detach(*args, **_kwargs)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/virt/block_device.py", line 400, in _do_detach
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     volume_api.attachment_delete(context, self['attachment_id'])
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/volume/cinder.py", line 379, in wrapper
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     res = method(self, ctx, *args, **kwargs)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/volume/cinder.py", line 415, in wrapper
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     res = method(self, ctx, attachment_id, *args, **kwargs)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/volume/cinder.py", line 838, in attachment_delete
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     'code': getattr(ex, 'code', None)})
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     self.force_reraise()
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     six.reraise(self.type_, self.value, self.tb)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/volume/cinder.py", line 831, in attachment_delete
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     attachment_id)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinderclient/v3/attachments.py", line 39, in delete
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     return self._delete("/attachments/%s" % base.getid(attachment))
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinderclient/base.py", line 339, in _delete
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     resp, body = self.api.client.delete(url)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinderclient/client.py", line 209, in delete
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     return self._cs_request(url, 'DELETE', **kwargs)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinderclient/client.py", line 191, in _cs_request
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     return self.request(url, method, **kwargs)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinderclient/client.py", line 177, in request
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server     raise exceptions.from_response(resp, body)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-9f348d0a-6afd-4ab9-a11f-de2aae655ca6)
2018-11-28 10:12:01.190 7 ERROR oslo_messaging.rpc.server
  • 迁移的目的服务器cinder-volume的日志信息如下：
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server [req-9f348d0a-6afd-4ab9-a11f-de2aae655ca6 90dea4aefe0c44e19cea9b3cbf0a37c2 3c4c8987959d465285c6e5e1db72e5a5 - - -] Exception during message handling: DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server Traceback (most recent call last):
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     res = self.dispatcher.dispatch(message)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 220, in dispatch
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return self._do_dispatch(endpoint, method, ctxt, args)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 190, in _do_dispatch
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     result = func(ctxt, **new_args)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/manager.py", line 4503, in attachment_delete
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     attachment_id)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/objects/base.py", line 314, in get_by_id
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     orm_obj = db.get_by_id(context, cls.model, id, *args, **kwargs)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/db/api.py", line 1777, in get_by_id
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return IMPL.get_by_id(context, model, id, *args, **kwargs)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 182, in wrapper
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 6980, in get_by_id
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return _GET_METHODS[model](context, id, *args, **kwargs)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 182, in wrapper
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return f(*args, **kwargs)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 2050, in volume_attachment_get
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return _attachment_get(context, attachment_id)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py", line 1982, in _attachment_get
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     .options(joinedload('volume'))
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2778, in first
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     ret = list(self[0:1])
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2570, in __getitem__
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return list(res)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return self._execute_and_instances(context)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2899, in _execute_and_instances
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     close_with_result=True)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2908, in _get_bind_args
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     **kw
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2890, in _connection_from_session
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     conn = self.session.connection(**kw)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 1035, in connection
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     execution_options=execution_options)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 1042, in _connection_for_bind
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     conn = engine.contextual_connect(**kw)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2123, in contextual_connect
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     self._wrap_pool_connect(self.pool.connect, None),
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2162, in _wrap_pool_connect
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     e, dialect, self)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1472, in _handle_dbapi_exception_noconnection
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     util.raise_from_cause(newraise, exc_info)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     reraise(type(exception), exception, tb=exc_tb, cause=cause)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2158, in _wrap_pool_connect
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return fn()
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/pool.py", line 403, in connect
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return _ConnectionFairy._checkout(self)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/pool.py", line 782, in _checkout
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     fairy = _ConnectionRecord.checkout(pool)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/pool.py", line 532, in checkout
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     rec = pool._do_get()
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/pool.py", line 1186, in _do_get
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     self._dec_overflow()
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     compat.reraise(exc_type, exc_value, exc_tb)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/pool.py", line 1183, in _do_get
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return self._create_connection()
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/pool.py", line 350, in _create_connection
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return _ConnectionRecord(self)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/pool.py", line 477, in __init__
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     self.__connect(first_connect_check=True)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/pool.py", line 667, in __connect
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     connection = pool._invoke_creator(self)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 105, in connect
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return dialect.connect(*cargs, **cparams)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 410, in connect
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return self.dbapi.connect(*cargs, **cparams)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/__init__.py", line 90, in Connect
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     return Connection(*args, **kwargs)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 699, in __init__
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     self.connect()
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 936, in connect
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     self._request_authentication()
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 1156, in _request_authentication
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     auth_packet = self._read_packet()
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 991, in _read_packet
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     packet_header = self._read_bytes(4)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server   File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 1037, in _read_bytes
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server     CR.CR_SERVER_LOST, "Lost connection to MySQL server during query")
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server DBConnectionError: (pymysql.err.OperationalError) (2013, 'Lost connection to MySQL server during query') (Background on this error at: http://sqlalche.me/e/e3q8)
2018-11-28 10:12:01.080 41 ERROR oslo_messaging.rpc.server

日志分析:
1.我们模拟的是虚拟机的疏散过程，在疏散的过程中，我们查看到目的服务器的nova-compute.log和cinder-volume.log的日志存在错误

2.进一步分析得知，我们发现出现问题的原因在cinder-volume，对比虚拟机能够正常疏散时的log日志，我们发现正常疏散时，cinder-volume.log的疏散显示的日志如下：

2018-11-28 14:37:54.772 34 INFO cinder.volume.manager [req-e287af5e-c5c3-4dc3-9cba-ab003ad12704 90dea4aefe0c44e19cea9b3cbf0a37c2 3c4c8987959d465285c6e5e1db72e5a5 - - -] Terminate volume connection completed successfully.
2018-11-28 14:37:54.773 34 DEBUG cinder.volume.manager [req-e287af5e-c5c3-4dc3-9cba-ab003ad12704 90dea4aefe0c44e19cea9b3cbf0a37c2 3c4c8987959d465285c6e5e1db72e5a5 - - -] Deleting attachment 6e32750e-6134-45cc-942f-c1065826f28b. _do_attachment_delete /var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/manager.py:4519

3.对比cinder-volume.log问题日志和正常日志，我们分析，在虚拟机的疏散过程，涉及卷的卸载过程，怀疑的cinder卷的卸载存在问题，对此，我模拟windows server2012的虚拟机挂载cinder卷，写大量数据，然后分离卷连接，模拟了大概50次左右，复现过一次cinder卷分离失败的错误，对比log进行分析对比，发现和上面的虚拟机疏散失败问题一致。

4.分析cinder卷的挂载流程，我们发现根据log提示和实际cinder卷的卸载流程存在中断卷连接的操作，也就是第三章节中1.4.6.2步骤。这里我们第三章节从cinder-volume的日志来看，函数在/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 1037, in _read_bytes此处提示丢失数据库连接错误，导致了中断卷连接时报错。

三.卷的卸载流程


上述流程图显示，卷在detach()->get vol info by id->terminate connection(call)时报错。

代码分析：

我们看到函数在_read_bytes处抛出错误：

进入cinder-volume容器，查看/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py代码的此函数如下:

def _read_bytes(self, num_bytes):
        self._sock.settimeout(self._read_timeout)
        while True:
            try:
                data = self._rfile.read(num_bytes)
                break
            except (IOError, OSError) as e:
                if e.errno == errno.EINTR:
                    continue
                self._force_close()
                raise err.OperationalError(
                    CR.CR_SERVER_LOST,
                    "Lost connection to MySQL server during query (%s)" % (e,))
        #如果MySQL服务器没有根据协议所述发送足够的数据，就会抛出如下错误
        if len(data) < num_bytes:
            self._force_close()
            raise err.OperationalError(
                CR.CR_SERVER_LOST, "Lost connection to MySQL server during queryy
")
        return data
        
#当服务器正在关闭(但显然仍在运行，因为这是从服务器发送的数据)时，将在专门为MariaDB记录的情况下抛出     
if packet_number == 0:
# MariaDB sends error packet with seqno==0 when shutdown
    raise err.OperationalError(
    CR.CR_SERVER_LOST,
    "Lost connection to MySQL server during query")
raise err.InternalError(
    "Packet sequence number wrong - got %d expected %d"
    % (packet_number, self._next_seq_id))

四.解决办法

MySQL数据库中，max_allowed_packet允许的server段接受的数据包最大为4M，有的时候大的插入和更新可能会被max_allowed_packet参数限制，导致失败，这里我们修改了max_allowed_packet参数。

[root@hci-test01 templates]# vi /usr/share/kolla-ansible/ansible/roles/mariadb/templates/galera.cnf.j2
[server]
#增加如下行
max_allowed_packet= '32M'

相关知识链接:

数据库知识: https://blog.csdn.net/hxpjava1/article/details/80522201
数据库问题分析：https://bugs.launchpad.net/tripleo/+bug/1691951
cinder的流程图：https://blog.csdn.net/inodes/article/details/24729883

猜你喜欢

转载自www.cnblogs.com/yuhaohao/p/10038301.html
今日推荐
《美国对全球网络空间安全与发展的威胁和破坏》报告发布
火速冲上 GitHub 热榜 —— 开源编程语言、框架哪有这么可爱？
北京人形机器人创新中心发布全球首个纯电驱拟人奔跑的全尺寸人形机器人“天工”
LFOSSA 源来如此公开课 | 掌握云原生未来：CNCF 认证全面攻略与备考秘籍
国产云输入法——仅华为无云端数据上传安全问题
周排行
Python环境安装与基础语法（1）——计算机基础知识
IMU预积分
ADAS中的LDW、FCW、BSD、LCA、ACC、AEB、APA、DMS代表的含义
B站笔试两道题
skyeye arm 硬件虚拟机环境的搭建
Web前端静态页面示例
数组-合并排序数组 II-简单
springcloud之版本问题启动报错
面向对象-------------匿名对象(六)
输入URL到页面呈现中间发生了什么？
每日归档
更多
2024-04-30(1)
2024-04-29(40)
2024-04-28(0)
2024-04-27(56)
2024-04-26(39)
2024-04-25(22)
2024-04-24(36)
2024-04-23(26)
2024-04-22(39)
2024-04-21(0)

代码天地 © 2018 codetd.com

境外服务器   最新电视剧2022