RHEV VM has been paused due to lack of storage space after RHEV-H upgrade to version 20140112.0.el6
问题
环境
Red Hat Enterprise Virtualization Hypervisor release 6.5 (20140112.0.el6)
vdsm 4.13.2-0.6
问题
VM has been paused due to lack of storage space.
VM has been paused due to no Storage space error.
Error message received in audit_log:
2014-02-07 13:39:42.781 | Failed to start VM vmname, because exist snapshot for stateless state. Snapshot will be deleted. | c1102bb
Since upgrading to RHEVH-6.5 v.20140112.0.el6 /var/log/messages are being flooded by these messages:
由于两个rhevh升级至6.5 v.20140112.0.el6 /是/日志/信息是被flooded这些信息：
Jan 29 05:17:39 rhevh04 vdsm vm.Vm ERROR vmId=`ed1061e9-a91d-45f2-a62e-e90a57bcd32a`::Stats function failed: <AdvancedStatsFunction _highWrite at 0x25fa650>
#012Traceback (most recent call last):#012
File "/usr/share/vdsm/sampling.py", line 351, in collect#012
File "/usr/share/vdsm/sampling.py", line 226, in __call__#012
File "/usr/share/vdsm/vm.py", line 529, in _highWrite#012
File "/usr/share/vdsm/vm.py", line 2316, in extendDrivesIfNeeded#012
File "/usr/share/vdsm/vm.py", line 842, in f#012
File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 76, in wrapper#012
File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1814, in blockInfo#012
libvirtError: invalid argument: invalid path /rhev/data-center/mnt/blockSD/589e5d96-84f0-412f-a6f5-3524e12e7606/images/aec90018-7195-4306-b3bb-b4a334f315a2/5f5e48e5-b4fe-4030-923d-cec014cc21b5 not assigned to domain
/var/log/vdsm/vdsm.log contains these logs:
/var/log/vdsm/vdsm.log包含以下日志：
Thread-5949::DEBUG::2014-01-29 05:14:33,027::sampling::285::vm.Vm::(start) vmId=`ed1061e9-a91d-45f2-a62e-e90a57bcd32a`::Start statistics collection
Thread-5969::DEBUG::2014-01-29 05:14:33,027::sampling::314::vm.Vm::(run) vmId=`ed1061e9-a91d-45f2-a62e-e90a57bcd32a`::Stats thread started
Thread-5969::DEBUG::2014-01-29 05:14:33,033::libvirtconnection::108::libvirtconnection::(wrapper) Unknown libvirterror: ecode: 8 edom: 10 level: 2 message: invalid argument: invalid path /rhev/data-center/mnt/blockSD/589e5d96-84f0-412f-a6f5-3524e12e7606/images/aec90018-7195-4306-b3bb-b4a334f315a2/5f5e48e5-b4fe-4030-923d-cec014cc21b5 not assigned to domain
Thread-5969::ERROR::2014-01-29 05:14:33,034::sampling::355::vm.Vm::(collect) vmId=`ed1061e9-a91d-45f2-a62e-e90a57bcd32a`::Stats function failed: <AdvancedStatsFunction _highWrite at 0x25fa650>
Traceback (most recent call last):
File "/usr/share/vdsm/sampling.py", line 351, in collect
File "/usr/share/vdsm/sampling.py", line 226, in __call__
File "/usr/share/vdsm/vm.py", line 529, in _highWrite
File "/usr/share/vdsm/vm.py", line 2316, in extendDrivesIfNeeded
File "/usr/share/vdsm/vm.py", line 842, in f
File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 76, in wrapper
File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1814, in blockInfo
libvirtError: invalid argument: invalid path /rhev/data-center/mnt/blockSD/589e5d96-84f0-412f-a6f5-3524e12e7606/images/aec90018-7195-4306-b3bb-b4a334f315a2/5f5e48e5-b4fe-4030-923d-cec014cc21b5 not assigned to domain
决议
If this error is received, an error relating to the VM being paused due to lack of storage space. For information on the advisory, and where to find the updated files, follow the link: vdsm 3.3.0-3 bug fix update
如果收到此错误，则与由于缺少存储空间而暂停的虚拟机相关的错误。有关建议的信息以及在何处查找更新的文件，请访问以下链接：vdsm 3.3.0-3 bug fix update
For more troubleshooting, for upgrades and roll-back issues, see the following guide: RHEV 3.2 to 3.3 Upgrade Guide: Tips, Considerations and Roll-back Issues
有关更多故障排除，有关升级和回滚问题，请参阅以下指南：RHEV 3.2到3.3升级指南：提示、注意事项和回滚问题
根源
This seems to occur if a VM is migrated to a hypervisor running vdsm 4.13.2-0.6 from a hypervisor running an earlier version of vdsm.
如果从运行早期版本vdsm的管理程序迁移到运行vdsm 4.13.2-0.6的管理程序，则似乎会发生这种情况。
After the VM is migrated libvirt and vdsm will end up having different path information about the VMs disks.
在迁移VM之后，libvirt和vdsm最终将拥有关于vms磁盘的不同路径信息。
During the ExtendDrivesIfNeeded method vdsm asks libvirt for block info about the the disk and passes in a path to the disk, however libvirt has a different path to the disk and reports the invalid path error.
Note that if the VM is started on the vdsm 4.13.2-0.6 hypervisor then the error doesn't occur.
在extenddrivesifrequired方法期间，vdsm要求libvirt提供关于磁盘的块信息，并将路径传递到磁盘，但是libvirt有不同的磁盘路径，并报告无效的路径错误。
注意，如果VM是在vdsm 4.13.2-0.6 hypervisor上启动的，则不会发生错误。
诊断步骤
After migrating the VM to hypervisor running vdsm 4.13.2-0.6, one can see it maintains path information in the format /rhev/data-center/mnt/blockSD/[sduuid]/images/[imguuid]/[voluuid], like so:
将VM迁移到运行vdsm 4.13.2-0.6的hypervisor后，可以看到它以/rhev/data center/mnt/blockSD/[sduuid]/images/[imguuid]/[voluuid]的格式维护路径信息，如下:
# vdsClient -s 0 list
ed1061e9-a91d-45f2-a62e-e90a57bcd32a
Status = Up
...
'path': '/rhev/data-center/mnt/blockSD/589e5d96-84f0-412f-a6f5-3524e12e7606/images/aec90018-7195-4306-b3bb-b4a334f315a2/5f5e48e5-b4fe-4030-923d-cec014cc21b5'}, 'format': 'cow', 'deviceId': 'aec90018-7195-4306-b3bb-b4a334f315a2', 'poolID': '5849b030-626e-47cb-ad90-3ce782d831b3', 'device': 'disk', 'path': '/rhev/data-center/mnt/blockSD/589e5d96-84f0-412f-a6f5-3524e12e7606/images/aec90018-7195-4306-b3bb-b4a334f315a2/5f5e48e5-b4fe-4030-923d-cec014cc21b5'
...
But looking at the info libvirt maintains about the VM, the path is specified as /rhev/data-center/[spuuid]/[sduuid]/images/[imguuid]/[voluuid], like so:
但查看libvirt维护的有关虚拟机的信息，路径被指定为/rhev/data center/[spuid]/[sduuid]/images/[imguuid]/[voluuid]，如下所示：
# virsh -r domblklist testvm
Target Source
------------------------------------------------
hdc -
vda /rhev/data-center/5849b030-626e-47cb-ad90-3ce782d831b3/589e5d96-84f0-412f-a6f5-3524e12e7606/images/aec90018-7195-4306-b3bb-b4a334f315a2/5f5e48e5-b4fe-4030-923d-cec014cc21b5
And then the /var/log/messages is flooded with these errors:
然后' /var/log/messages '被这些错误淹没:
Jan 29 07:15:36 rhevh04 vdsm vm.Vm ERROR vmId=`ed1061e9-a91d-45f2-a62e-e90a57bcd32a`::Stats function failed: <AdvancedStatsFunction _highWrite at 0x25fa650>#012Traceback (most recent call last):#012 File "/usr/share/vdsm/sampling.py", line 351, in collect#012 File "/usr/share/vdsm/sampling.py", line 226, in __call__#012 File "/usr/share/vdsm/vm.py", line 529, in _highWrite#012 File "/usr/share/vdsm/vm.py", line 2316, in extendDrivesIfNeeded#012 File "/usr/share/vdsm/vm.py", line 842, in f#012 File "/usr/lib64/python2.6/site-packages/vdsm/libvirtconnection.py", line 76, in wrapper#012 File "/usr/lib64/python2.6/site-packages/libvirt.py", line 1814, in blockInfo#012libvirtError: invalid argument: invalid path /rhev/data-center/mnt/blockSD/589e5d96-84f0-412f-a6f5-3524e12e7606/images/aec90018-7195-4306-b3bb-b4a334f315a2/5f5e48e5-b4fe-4030-923d-cec014cc21b5 not assigned to domain
But if the VM is started directly on the vdsm 4.13.2-0.6 hypervisor then vsdm and libvirt maintain the same path information:
但是，如果虚拟机直接在vdsm 4.13.2-0.6管理程序上启动，那么vsdm和libvirt会维护相同的路径信息：
# vdsClient -s 0 list
ed1061e9-a91d-45f2-a62e-e90a57bcd32a
Status = Up
...
'path': '/rhev/data-center/mnt/blockSD/589e5d96-84f0-412f-a6f5-3524e12e7606/images/aec90018-7195-4306-b3bb-b4a334f315a2/5f5e48e5-b4fe-4030-923d-cec014cc21b5'}, 'format': 'cow', 'deviceId': 'aec90018-7195-4306-b3bb-b4a334f315a2', 'poolID': '5849b030-626e-47cb-ad90-3ce782d831b3', 'device': 'disk', 'path': '/rhev/data-center/mnt/blockSD/589e5d96-84f0-412f-a6f5-3524e12e7606/images/aec90018-7195-4306-b3bb-b4a334f315a2/5f5e48e5-b4fe-4030-923d-cec014cc21b5'
...
# virsh -r domblklist testvm
Target Source
------------------------------------------------
hdc -
vda /rhev/data-center/mnt/blockSD/589e5d96-84f0-412f-a6f5-3524e12e7606/images/aec90018-7195-4306-b3bb-b4a334f315a2/5f5e48e5-b4fe-4030-923d-cec014cc21b5