Cloud computing and virtualization through knowledge entry

Friends often ask me, what do you do it?

I replied that cloud computing.

Fortunately, no answer, a reply, they are more puzzled. Listen on big pricey, very cattle b, just do not know what is. In most Pythonista's cognitive, learned Python, do not engage in reptiles, it is to do Web development, or else the data analysis / mining, artificial intelligence.

Think about it, this is reasonable, most training courses for you to make you pay tuition, usually take your career to the planned route, what Web developers, what reptiles engineers, data analysts, and so what, nobody tells you learned Python, you can also do a cloud computing engineer (to be exact OpenStack engineer, because the cloud covering the broader needs of more technology stack, rather than just a programming language).

To give you a better understanding of the content of some of the cloud, I thought to write this article, I explain in the field, as well as cloud computing and virtualization make this a sort of entry-level knowledge, if you just want to get into this field, this Getting Started Guide Liberal, should be quite right for you.

1. What is cloud computing?

Wikipedia definition of cloud computing (English: Cloud Computing ), is based on the Internet method of calculation, in this way, to share hardware and software resources and information can be provided on request to the computer terminals and a variety of other devices.

Boring definitions, still sounds very good understanding, I try next to explain in their own language.

Calculation that computing resources, including the familiar CPU, memory, disk, bandwidth, and so on.

Cloud, is to these resources become fragmented entity with a huge resource pool, with the pool, as individuals, you no longer need to own you buy a computer at home, as a small company, you do not need to own the whole a room, spend a lot of manpower and equipment costs to operate the infrastructure. Once you need, you can apply to pond owners. This greatly improves the utilization of resources and the flexibility of distribution.

Others say clouds are like the clouds, focus more water vapor will rain, rain fell on the ground will evaporate into the sky, the next time it rains continue to wait. Cloud computing in the cloud, as in cloud, like nature, can achieve recycling of resources. In a public cloud provider where you buy a year of cloud hosting, one year after the resources are recycled, it can be reassigned to someone else to use.

Cloud computing model, based on service-oriented. Depending on the service level can be divided into three categories:

IaaS (Infrastructure as a Service): infrastructure as a service, simply put, it is the provision of infrastructure, you bought on Ali cloud, AWS cloud hosting was part of this.
PaaS (Platform as a Service): Platform as a Service, simply put, it is to provide a platform Typical applications are, GAE (Google App Engine), you provide directly to the runtime environment of an application.
SaaS (Software as a Service): software as a service, that you are familiar with it, you are on the phone APP fall into this category.

These three models for different groups, from top to bottom, the user autonomy getting smaller and smaller, need less and less attention to the details.

Also depending on the deployment of the cloud can be divided into three categories: public cloud, private and hybrid clouds.

Private cloud of a business or organization dedicated to the use of cloud computing resources constitution. Private cloud can be organized on-site data center located in the physical, but also by third-party service provider hosting. The hybrid cloud is both.

2. What virtualization is?

Cloud computing, in order to improve resource utilization, flexibility allocation proposed a solution.

And underlying this solution, the need for technical support, the key is virtualization and container technology.

This is mainly about virtualization.

You are not going to Mongolia circle, what is virtualization?

It is a physical resource may be a computer (CPU, memory, storage, network, etc.) abstraction conversion, and provides segmentation, re-combination, a technique to achieve maximum utilization of resources.

Virtual machine, right?

When you only have a computer installed windows system, and you want to experience the linux system, if you do not want to toss installing a dual system, the easiest way is to use VMWare or VirtualBox to create virtual machines with linux mirrored in your computer .

Have you ever wondered how this virtual machine is created out how such a magical, one computer can actually running two operating systems simultaneously.

In fact, after you create the virtual machine is just a physical process only on the original machine. But it looks and the original host system you use is no different from the appearance. The virtual machine has its own memory, cpu, disk, network cards, which are dependent on the virtualization technology was able to achieve.

In the virtual machine, if to be used on a physical device virtualization technology allowed except indirectly using a physical device, the device may also be used so that the virtual machines through the use of a physical device directly, through this technique does not require the VMM (Virtual machine monitor, will be introduced later), so the performance will be better than virtualization. Common are GPU pass-through. There virtualization through binding, such as SR-IOV, i.e., a single IO Virtualization (Single-root I / O virtualization), to a physical NIC (PF) into a plurality of virtual virtual NICs (VF), then the virtual NIC to hang straight to the virtual machine.

What 3. VMM that?

VMM, usually called Hypervisor (Hypervisor Now we will also refer VMM), Chinese name: the virtual machine monitor, the English name: Virtual Machine Monitor.

Hypervisor is interposed between a virtual machine operating system and the physical resources in order to achieve virtualization software layer introduced.

Note that, Hypervisor is not a specific software, collectively, but a kind of software.

When a virtual machine is operating on physical resources, Hypervisor will be intercepted and redirected to their instructions, so no-aware virtual machines to use the same operating system as the physical physical resources.

Common Hypervisor, there

KVM
Xen
Hyper-V
VMWare

4. virtualization technology

1. KVM

KVM (Kernel-based Virtual Machine), means that the kernel-based virtual machine.

KVM is integrated into the Linux kernel Hypervisor, it is the X86 architecture and hardware support for virtualization technology (Intel VT or AMD-V) is a full Linux virtualization solutions. It is a small module for Linux, use Linux to do a lot of things, such as task scheduling, memory management hardware to interact with the equipment.

2. Xen

Xen is a first class runs on bare metal hypervisor. It supports full virtualization and paravirtualization, Xen support hypervisor and virtual machines communicate with each other, and available on all versions of Linux free products, including Red Hat Enterprise Linux and SUSE Linux Enterprise Server. The most important advantage of Xen paravirtualization, in addition to unmodified operating system can be run directly on xen (such as Windows), allows a virtual machine to run efficiently without the need for emulation, virtual function perceived hypervisor, without We need to simulate virtual hardware, which can achieve high performance.

3. QEMU

QEMU is a set of free software simulation processor by Fabrice Bellard written. Qemu, where the key emu, full name emulator, simulator, so simple to use fully virtualized mode qemu is used.

That QEMU what's the use? KVM and it is what does it matter?

Precisely, KVM is a Linux kernel module is. You can use the command modprobe to load the KVM modules. After loading the module, in order to further create a virtual machine by other tools.

But only KVM module is not enough, KVM is the bottom of the hypervisor, it is only used to simulate the operation of the CPU, the lack of support for the network and peripheral I / O, so we can not use it directly.

The QEMU-KVM is a complete simulator, which is based on KVM built, provides a complete network and I / O support.

Speaking of QEMU, in fact, it is also a virtualization software. What role is it, it is equivalent to a router, when the Guest OS kernel want to manipulate the physical hardware must be forwarded via Qemu, operating instructions transferred to real hardware. Since all instructions pass through Qemu hand from the inside, and therefore relatively poor performance.

to sum up

KVM and Xen are free.
KVM requires hardware support (Intel VT or AMD-V), integrated in the kernel, but Xen can run on all Linux, may not require hardware support.

4. libvirt

To explain what libvirt, as long as know why libvirt needs just fine.

虚拟化的底层可能是KVM，也可能是Xen，或者是其他市面上的Hypervisor，种类之繁多，如果没有一个统一的接口来管理它们，就太乱了，移植性非常差。
Hypervisor ，以 qemu-kvm 为例，它的命令行虚拟机管理工具参数众多，难于使用。需要有一个工具将这些参数进行封装。

这些都是在内核空间层做的事情，而我们用户创建、销毁虚拟机都是在用户空间层操作，这就尴尬了，我们没有权限。

这下该 libvirt 出场了，libvirt分为服务端各客户端。

服务端是libvirtd，而你所熟悉的virt，virt-install，virt-manager，virt-viewer 等都是libvirt的客户端。

目前，libvirt 已经成为使用最为广泛的对各种虚拟机进行管理的工具和应用程序接口（API），而且一些常用的虚拟机管理工具（如virsh、virt-install、virt-manager等）和云计算框架平台（如OpenStack、OpenNebula、Eucalyptus等）都在底层使用libvirt的应用程序接口。

5. 虚拟化分类

全虚拟化和半虚拟化

根据客户机系统是否需要修改定制可以分为 全虚拟化 和 半虚拟化。

1. 全虚拟化

全虚拟化（英语：Full virtualization），是需要依托于硬件虚拟化的。

在全虚拟化模式下，虚拟机的所有操作（CPU，内存，网络等）都需要经过一个运行在物理机上的虚拟化软件转发给物理机内核。而这个虚拟化软件，在windows上你常见且熟悉的有vmware，virtualbox。

允许未经修改的客操作系统（英语：Guest OS）隔离运行。在全虚拟化环境中，任何可以运行在裸机上的软件（通常是操作系统）都可以未经修改地运行在虚拟机中。

代表：VMWare(1998年)，KVM

2. 半虚拟化

半虚拟化(英语：Paravirtualization)是另一种类似于全虚拟化的热门技术。

半虚拟化对比全虚拟化，就是有一些可以直接操作物理内核空间，而不需要全部经过虚拟化软件。这就大大提高了虚拟机的性能。

它在HOST上使用Hpervisor（虚拟机管理程序）提供便利的接口，使得Guest OS能够调用接口访问虚拟硬件。而条件是，Guest OS 内部需要部署安装相应的驱动和软件逻辑，需要对操作系统进行修改。

代表：Xen（2006）

Xen是一款虚拟化软件，支持半虚拟化和完全虚拟化。它在不支持VT技术的cpu上也能使用，但是只能以半虚拟化模式运行。
半虚拟化的意思是需要修改被虚拟系统的内核，以实现系统能被完美的虚拟在Xen上面。完全虚拟化则是不需要修改系统内核则可以直接运行在Xen上面。
VMware是一款完全虚拟化软件。完全虚拟的弱点是效率不如半虚拟化的高。半虚拟化系统性能可以接近在裸机上的性能。

1型虚拟化和2型虚拟化

根据虚拟化层是直接位于硬件之上还是位于操作系统之上，可以分为 Type 1 虚拟化和 Type 2 虚拟化。

Type 1：Xen，VMWare ESX

Type 2：KVM，WMWare Workstation

硬件虚拟化和软件虚拟化

1. 软件虚拟化

在硬件虚拟化出现之前，市场上都是使用的软件虚拟化。

软件虚拟化，就是通过软件来实现虚拟化，原理是把从虚拟机传来的操作指令进行截取翻译，并传递给真实的物理硬件。

由于每条指令都需要经过“截取” -> “翻译” -> “转发”，所以其虚拟化性能会差一点。

哪些属于虚拟化软件呢？

KVM：负责cpu和内存的虚拟化，但cpu必须支持硬件虚拟化。
QEMU：负责IO设备（网卡、磁盘）的虚拟化

2. 硬件虚拟化

硬件虚拟化，是指计算机硬件本身提供能力让客户机指令独立运行，而不需要Hypervisor 截获重定向。直接从硬件层面开始支持虚拟化。由硬件支持并提供多个虚拟硬件设备接口，这些设备由虚拟机内核驱动传递给虚拟机使用。使用这种方式，虚拟机能获得和宿主机一样的硬件功能，性能也和宿主机相近，同时原生操作系统本来就支持这项技术，因此无需对操作系统进行修改。

Intel 从2005年开始在 x86 cpu 上支持硬件虚拟化，大大推进了虚拟化的发展。

缺点就是，硬件要支持虚拟化功能，在以前这可能是缺点，但是现在随着虚拟化技术的发展，越来越多的硬件都已经支持虚拟化，成本也越来越低，所以硬件辅助虚拟化是目前最流行，使用最广泛的虚拟化技术。

KVM这种流行的虚拟化技术里，既有软件虚拟化，也有硬件虚拟化，软件虚拟化要基于硬件的虚拟化，二者是相辅的关系，而不是互斥。

6. KVM工具

有了虚拟化，就有了虚拟机，那如何对这些虚拟机进行管理呢。

在 Linux 下有许多的工具可以使用：

Virsh：基于 libvirt 的命令行工具（CLI）
Virt-Manager：基于 libvirt 的 GUI 工具
virt-v2v：虚机格式迁移工具
virt-* 工具：包括 Virt-install （创建KVM虚机的命令行工具）， Virt-viewer （连接到虚机屏幕的工具），Virt-clone（虚机克隆工具），virt-top 等
libguestfs-tools：一组 Linux 下的 C 语言的 API ，用来访问/修改虚拟机的磁盘映像文件。

7. 创建虚拟机

手工创建

虚拟机的本质是宿主机上的一个进程，当你用OpenStack在界面，或者使用virsg 创建了一个虚拟机时。你可以使用ps -ef|grep kvm 看下这个虚拟机的进程，是下面这样子的。

参数多得让人头皮发麻。意思是，你可以使用这样一串命令才能创建一台虚拟机。

$ /usr/libexec/qemu-kvm \
-name guest=instance-00000035, debug-threads=on \
-S -object secret,id=masterKey0,format=raw,file=/var/lib/libvirt/qemu/domain-216-instance-00000035/master-key.aes \
-machine pc-i440fx-rhel7.5.0,accel=kvm,usb=off,dump-guest-core=off \
-cpu host \
-m 16384 \
-realtime mlock=off \
-smp 2,maxcpus=32,sockets=2,cores=16,threads=1 \
-uuid 31d70882-194f-469b-855e-fcfa6736550d \
-smbios type=1,manufacturer=RDO,product=OpenStack Compute,version=0.0.1-1.el7.centos,serial=bc147bfe8a204d06a09f98387e46b890,uuid=31d70882-194f-469b-855e-fcfa6736550d,family=Virtual Machine \
-display none \
-no-user-config -nodefaults \
-chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/domain-216-instance-00000035/monitor.sock,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=utc,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-hpet -no-shutdown -boot strict=on \

-device piix3-usb-uhci,id=usb,bus=pci.0,addr=0x1.0x2 -drive file=/dev/hdd-volumes/31d70882-194f-469b-855e-fcfa6736550d_disk,format=raw,if=none,id=drive-virtio-disk0,cache=none,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x4,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/var/lib/nova/instances/31d70882-194f-469b-855e-fcfa6736550d/disk.config,format=raw,if=none,id=drive-ide0-0-0,readonly=on,cache=writeback \
-device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0 \
-netdev tap,fds=28:35,id=hostnet0,vhost=on,vhostfds=36:37 \
-device virtio-net-pci,mq=on,vectors=6,netdev=hostnet0,id=net0,mac=fa:16:3e:69:63:18,bus=pci.0,addr=0x3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x2 \
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x5 -msg timestamp=on

virsh 创建

前面我们看到，创建一台虚拟机需要诸多的参数。

如果一个一个去指定，非常不易于管理及复用。

如果可以在创建时，指定一个配置文件，这个配置文件里包含上述所有的参数，不就大大简化了虚拟机创建过程。

这时候就出现了virsh这个基于 libvirt 的命令行工具（CLI）。通过它我们可以指定一个 xml 配置文件来很轻松的创建一台虚拟机。

virsh define vm.xml
virsh start guest_vm

其中xml的内容如下

<domain type='kvm' id='200'>
  <name>guest_vm</name>
  <memory unit='KiB'>12582912</memory>
  <currentMemory unit='KiB'>12582912</currentMemory>
  <vcpu placement='static'>6</vcpu>
  <numatune>
    <memory mode='strict' nodeset='0-1'/>
    <memnode cellid='0' mode='strict' nodeset='0'/>
    <memnode cellid='1' mode='strict' nodeset='1'/>
  </numatune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-i440fx-rhel7.0.0'>hvm</type>
    <boot dev='hd'/>
    <boot dev='cdrom'/>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <pm>
    <suspend-to-mem enabled='no'/>
    <suspend-to-disk enabled='no'/>
  </pm>
  <devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/path/to/test.qcow2'/>
      <backingStore/>
      <target dev='hda' bus='virtio'/>
      <alias name='virtio-disk0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </disk>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <controller type='ide' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <controller type='usb' index='0' model='piix3-uhci'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:d2:81:b0'/>
      <source bridge='br0-ovs'/>
      <virtualport type='openvswitch'>
        <parameters interfaceid='abc10709-ebff-4d0f-8761-4b7fdaba0dc0'/>
      </virtualport>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <interface type='bridge'>
      <mac address='52:54:00:5e:91:38'/>
      <source bridge='br0-ovs'/>
      <virtualport type='openvswitch'>
        <parameters interfaceid='cceb5703-185a-4f6f-b2ce-a7e273e52bdc'/>
      </virtualport>
      <bandwidth>
        <inbound average='50000'/>
        <outbound average='50000'/>
      </bandwidth>
      <target dev='vnet1'/>
      <model type='virtio'/>
      <alias name='net1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </interface>
    <interface type='bridge'>
      <mac address='52:54:00:b2:77:07'/>
      <source bridge='br0-ovs'/>
      <virtualport type='openvswitch'>
        <parameters interfaceid='674716ab-243d-4137-bc01-aa2c33cca21a'/>
      </virtualport>
      <target dev='vnet6'/>
      <model type='virtio'/>
      <alias name='net2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </interface>
    <console type='pty' tty='/dev/pts/4'>
      <source path='/dev/pts/4'/>
      <target type='virtio' port='0'/>
      <alias name='console0'/>
    </console>
    <input type='mouse' bus='ps2'>
      <alias name='input0'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input1'/>
    </input>
    <sound model='ich6'>
      <alias name='sound0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </sound>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <alias name='video0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+0:+0</label>
    <imagelabel>+0:+0</imagelabel>
  </seclabel>
</domain>

OpenStack

使用 virsh 来指定xml进行创建虽然能对虚拟机进行生命周期的管理，但是无法对成百上千台的机器进行集中式的管理。

这时候，OpenStack 这个开源的云计算管理平台就出现了。

有了OpenStack，你可以使用 Horizon提供的界面进行虚拟机的管理

来源网络，侵删

也可以使用nova 的 cli 命令进行创建。

nova boot <vm_name> \
--flavor <flavor_id> \
--nic net-id=<net_id>,v4-fixed-ip=<ip> \
--image <image_id/name> \
--config-drive True

说了半天，线于引出了OpenStack，我的工作基本60%的时间都是围绕着它转，OpenStack 是一个开源框架，是使用Python语言开发的最大的项目，具说有数百万行的代码量，是动态语言的一个优秀典范。

关于 OpenStck，你可能不太明白它是做什么的。这里引用我昨天看到的另一篇文章的一个说明：它有点像一个商店，负责管理所有的商品（计算资源、存储资源、网络资源等），卖给用户，但是它本身不制造商品（不具备虚拟化能力），它的商品来自KVM（当然也可以用Xen等其他Hypervisor）。

附录：参考文档

关注公众号，获取最新干货！