VMware vSphere vMotion functionality is one of today's virtual infrastructure is the most important function. Since its establishment in 2002 and released in 2003 since it allows us to be active virtual machines migrate from one physical ESXi host to another. Today, the ability to seamlessly migrate a virtual machine is an important part of almost every virtualization deployments. Workload portability is the real basis of hybrid cloud experience, be able to use VMware hybrid cloud extension (HCX) to move them between internal and public clouds. vSphere vMotion IT industry remains one of the most important game changer.
Over the years, we developed a number of internal vMotion technology to support the new technology.
This blog this article will focus on the standard vMotion, which is the active state transition from the source to calculate the standard vMotion target ESXi host. We can also perform Storage vMotion, when used in conjunction with standard vMotion, it is considered to be enhanced vMotion. Other types of long distance and cross-cluster vMotion vMotion, they are mainly above the vCenter vMotion end ESXi Server operating processes.
vMotion process
After starting the virtual machine migration, vCenter Server instance will perform the so-called long-running migration tasks to deal with migration. The first step is to perform a compatibility check. Can you run a virtual machine on the destination host? Consider possible constraints may hinder live migration. The next is to tell the source and target ESXi hosts what happened. To create a migration task with the following information:
- Whether it is being migrated virtual machine
- Whether in the configuration of the virtual machines (virtual hardware, VM options, etc.)
- Source ESXi hosts for compliance
- Target ESXi hosts for compliance
- vMotion network-related issues
vCenter Server instance with the source and target ESXi host ESXi hosts share migration practices to ensure the exchange of all necessary information in order to start the migration process. Use vCenter Server Virtual Provisioning X Daemon (VPXD) communicates with ESXi host, which calls Virtual Provisioning X Agent (VPXA) running on the ESXi host. VPXA VPXD message from the listener, which receives hostd migration specification and by passing it to VMX process. Host daemon (hostd) maintain specific access to information and management of the host, including VMstate and other virtual machine telemetry. When you start the migration, hostd virtual machine will be placed in the middle of the state, so it can not change the virtual machine configuration during the migration.
Virtual machine monitor (VMM) is responsible for managing the process of virtual machine memory and virtual machine storage and network IO request to the VMkernel. All other performance-independent IO requests are forwarded to the VMM VMX. VM extension (VMX) process running in the VMkernel, responsible for the performance of IO devices unimportant. Please note, VMM is only used on the source ESXi host during the migration, because this is the location of the active virtual machine's memory is located.
Once this is done, the VMkernel ESXi migration source module opens a socket on vMotion enable the network to communicate with the target set ESXi host.
Preparation stage to the pre-copy phase
So far, all processes and communication paths are ready to ticket line real-time migration. The purpose is to ensure the preparation phase for the target ESXi host virtual machines to migrate computing resources in advance. In addition, the virtual machine has been created on the target host, but it is being masked.
After completion of the preparation phase, the process will enter the pre-copy phase, at this stage the memory transfer from source to target ESXi host. All you need to track the source virtual machine memory pages on the ESXi host. By this way, vMotion process know which pages of memory source virtual machine is covered or modified (known as a dirty page) during the migration, because it requires these memory page re-sent to the target host.
Page tracking
在预复制阶段，虚拟机正在使用的vCPU会被短暂停顿，以安装页面跟踪器。VMkernel迁移模块现在要求VMM启动页面跟踪，因为VMM拥有虚拟机的内存页表状态。下图显示了guest虚拟机操作系统在vMotion期间将数据写入内存时发生的情况：
迭代内存预复制
页面跟踪是一个连续的循环。它将通过使用多次迭代来实现内存预复制收敛。第一次迭代（预拷贝阶段-1）复制虚拟机内存。以下迭代（预拷贝阶段0到n）用于复制脏内存页面。举个例子，这就是我们实时迁移具有24GB内存的虚拟机时迭代的样子：
阶段-1：复制24GB的虚拟机内存和跟踪页面。当我们发送内存时，它会带来8GB的污染。
阶段0：重新传输脏污的8GB。在这个过程中，内存污染另外3GB。
阶段1：发送3GB。当转移发生时，虚拟机又会污染1GB。
阶段2：发送剩余的1GB。
当内存页面从源复制到目标ESXi主机时，我们需要确定何时能够完成预复制，所以VMM会在每次迭代复制后询问VMkernel是否已完成预复制。当只有将所有内存更改（脏页）复制到目标主机时，才可以执行后续操作。迭代内存预拷贝算法的一部分是将所有目标内存页面与其源匹配。从第0页开始一直到最大或最后一个内存页码，依次检查所有内存页以查看目标页是否与源页同步。
要确定我们是否可以终止预复制，我们需要验证是否可以在<500ms的窗口中完成最后一次内存页面复制。我们可以使用迁移开销中的信息来计算：
- 迁移传输速率; 以什么速度（GbE）我们在主机之间复制内存数据？
- 脏页率（GB / s）; 客户操作系统覆盖了多少内存页面？
- 我们还有多少页要传输到目标主机？
如果不是，则发生下一次迭代。如果结果为是，则VMkernel迁移模块将终止预复制过程。
现在，如果脏页率高于迁移传输速率会发生什么？如果是这种情况，那么进行另一次迭代是没有意义的，因为我们永远无法实现内存预复制的收敛，并且迁移将停止。这就是我们在vSphere 5.0中引入Stun页面发送（SDPS）的原因。基本上，SDPS是VMkernel告诉VMM不运行预定指令但是引入非常短的“睡眠”的一种方式。这可能听起来像是对工作负载性能的影响，但这种情况发生在细粒度级别。正是由于这些非常小的微秒级别的时间窗口，我们可以将vMotion预复制收敛，并完成vMotion工作。
如果脏页面速率>传输速率，则每次迭代执行SDPS。后续迭代仅复制在上一次迭代期间修改的脏内存页。迭代的持续时间越短，客户OS就越不能修改或弄脏其存储页面，从而缩短了下一次预复制迭代。虽然产生一些性能开销，但SDPS通常不会对工作负载造成影响。这些开销对客户操作系统来说是可以忽略不计的。
切换
由VMM终止内存预复制后，所有内存页都驻留在目标ESXi主机上。VMM现在向VMX发送远程过程调用（RPC），它可以挂起源虚拟机。VMX将进入检查点阶段，暂停虚拟机并将检查点数据发送到目标ESXi主机。
在此过程中，目标ESXi主机上的虚拟机将被解除屏蔽，并使用源虚拟机的检查点数据恢复状态。基本流程是：启动目标虚拟机、中断启动过程、再把状态指向迁移过来的源虚拟机内存页，完成启动。所有这些通常发生在100-200ms，这是虚拟机处于不可访问的一个时间，这取决于主机硬件性能、动态的访问负载等各种因素。
到此，虚拟机的vMotion完成。
原文：https://blogs.vmware.com/vsphere/2019/07/the-vmotion-process-under-the-hood.html