When it comes to virtual environments, VMware vSphere networking is perhaps one of the most critical components.
How do your ESXi hosts and virtual machines communicate?
Since virtual networking is the key to how everything else works, we need a good understanding of how ESXi networking works.
1. ESXi network analysis
When it comes to our ESXi hosts and their network communication, the first thing we need to understand is what they need to do, there are a few different ways to communicate:
- Mutual communication (think vMotion)
- Communicate with vCenter (think HA)
- Communicate with external resources they rely on, such as storage arrays or Active Directory
- Allow their resident VMs to communicate with other VMs and external resources they may depend on
Therefore, several different components of ESXi networking in VMware vSphere had to be configured in our environment.
- Virtual switches on ESXi hosts
- Port Groups on Virtual Switches
Within each of these categories, there are many things to consider.
2. Virtual switches on ESXi hosts
Think of a virtual switch on an ESXi host like any other switch you may have come across before, when it comes to the network layer, the virtual switch in ESXi is how our ESXi hosts provide connectivity, redundancy, and load balancing.
There are two types of switches, although you may hear them by several different names.
- vSphere Standard Switch
- vSphere Distributed Switch (vSphere Enterprise Plus license required)
Let's take a look at each virtual switch and its characteristics.
3. vSphere standard switch / standard vSwitch / vSwitch
The base virtual switch, also known as a vSwitch. It was the first type of switch available in VMware vSphere, and there's nothing wrong with using standard vSwitches, although they are more difficult to manage than vSphere Distributed Switches.
Standard vSwitches reside on ESXi hosts and must be configured individually on each host, there are several ways to simplify this process such as using PowerCLI or host profiles.
Network switches connect to NICs in ESXi hosts, and NICs in ESXi hosts connect to virtual switches in ESXi. That's all!
When we talk about the NICs of an ESXi host, we call them vmnics because that's what they're called in ESXi. vmincs always start at 0, so your first physical NIC port will correspond to vmnic0.
Depending on how you designed your virtual network, you may have multiple vSwitches on your ESXi host.
4. vSphere distributed switch/distributed virtual switch/dvSwitch
On the physical side there is not much difference between a standard vSwitch and a distributed vSwitch, we still redundantly connect the NICs in the ESXi hosts to the network switches, what changes is the virtualization layer, and the switches within VMware vSphere itself.
Here is a picture of the Distributed Virtual Switch:
As you can see, the switch configuration itself is on the vCenter server and not the ESXi host, of course, if something goes wrong with vCenter, that could be a problem, right?
There is a copy of the Distributed Virtual Switch configuration on each ESXi host, as you can see in the bright blue switch on each host, so that if something goes wrong with vCenter, our virtual machines won't be affected.
I remember when the Distributed Virtual Switch first came out in vSphere 4 it took everyone by surprise because there was such a drastic shift in the way we were used to doing things, so let's break down our vSphere Distributed Switch again:
Each vmnic is connected to a corresponding dvUplink port on the Distributed Virtual Switch. The Distributed Virtual Switch is then managed through the vCenter server. The important part about this is that the Distributed Virtual Switch configuration is designed to be the same on every ESXi host.
Similar to standard vSwitches, distributed vSwitches can be managed using the vSphere Client, PowerCLI, or host profiles. The difference here is that you configure a single distributed virtual switch on the vCenter server instead of a single virtual switch on the ESXi host.
5. Which VMware virtual switch is right for my environment?
This is a good question, unfortunately there is no single answer. Your choice of using a standard virtual switch or a distributed virtual switch will depend on the unique requirements of your environment.
When making your choice, it's important to keep infrastructure design quality in mind when determining what will meet your requirements. they are:
- availability
- Manageability
- performance
- Recoverability
- Safety
It may also be helpful to rank these qualities by their importance to the project.
Be sure to test your VMware vSphere network configuration extensively before declaring your VMware vSphere environment ready for production.
Remember, vSphere Distributed Switch does require a VMware Enterprise Plus license.
6. NIC teaming policy in VMware
Another important component of VMware networking is NIC teaming. NIC teaming is setting up our virtual switches in a way that facilitates load balancing and protects against component failures.
For example, if I have two physical NICs attached to my switch, ideally I'd like to ensure that traffic flows through both physical NICs, and if one of the NICs or the upstream switch fails, my switch will remain operational.
There are several ways to achieve this in VMware vSphere Networking.
Now assuming we have configured our switch, move on to the next step.
7. Configure the port group on the virtual switch
When we connect to our virtual switch, we connect to something called a port group. As the name suggests, a port group is a group of virtual ports on our virtual switch.
There are several different types of port groups:
- virtual machine port group
- VMkernel port group
Virtual machine port groups are how we connect virtual machines. For example, I might have a portgroup with a single VLAN, multiple VLANs (VLAN trunking), or in the case of a distributed virtual switch, private VLANs.
As for VMkernel portgroups, you might ask...
8. What is a VMkernel port?
good question! VMkernel ports are used for non-virtual machine traffic in VMware vSphere, and as you can see in the screenshots you took while configuring networking, there are many different types of VMkernel ports:
At a minimum, each ESXi host has one VMkernel port for host management. If it is a member of a vSphere cluster, it will also have a VMkernel port for vMotion. If the cluster is a vSAN cluster, there will be a VMkernel port for vSAN.
If you're taking advantage of that feature, there will also be VMkernel ports for IP-based storage such as iSCSI, NFS, or FCoE, and VMkernel ports for fault tolerance.
The biggest difference between a virtual machine port group and a VMkernel port group is the type of traffic it passes, a VMkernel port is passing VMware vSphere specific traffic, a virtual machine port group is just passing your various virtual machine traffic.
You can read more about VMkernel system traffic types in the official vSphere networking documentation.
Each VMkernel port has its own unique IP address, and you can also change the default MTU of 1500 on a VMkernel port.
9. VMware vSwitch security settings
Now that we've covered most of the basics of vSphere networking, I want to dig a little deeper into some areas involving VMware switch configuration. VMware vSwitch security settings are often misunderstood.
Each of these security settings has two options: Deny or Accept.
By default, these security settings are set to deny on this Distributed Virtual Switch portgroup I just created, so let's see what each of them really means.
9.1, Promiscuous mode VMware security policy
That sounds pretty much exactly what it means. If promiscuous mode is set to accept, the virtual machine's NIC will accept all frames sent to the active VLAN on that portgroup.
It's not secure, but it does have its use cases. If you are running a virtual machine that needs to inspect all packets, such as an intrusion detection system, you can set this option to Accept.
By default it is set to deny and should stay that way.
9.2. MAC address change VMware security policy
MAC address changes look at inbound traffic to a virtual machine.
When it comes to a virtual machine's network interface, there are several different MAC addresses.
First, there is the MAC address specified in the VMX file of the virtual machine, think of this as if your virtual machine had a physical NIC card, it is the MAC address of the NIC card specified by the manufacturer.
There is also the MAC address specified in the virtual machine, which is the same as the one in the VMX file...unless you need to change it for some reason.
In the properties of the virtual machine, the MAC address can be changed. In the past I have changed to ensure that the MAC address of the virtual machine is the same as the MAC address of the physical machine after P2V, because software licenses are tied to MAC addresses.
In some valid use cases, these MAC addresses may be different, and you must set MAC address changes to accept.
Otherwise, it should be set to Deny, as malicious actors can also wreak havoc on your environment using spoofed MAC addresses.
9.3. Forged transfers VMware security policies
Forged Transmission also looks at the MAC address of the virtual machine, but it handles outgoing traffic.
If the two MAC addresses do not match, it will drop frames, similar to a MAC address change.
Again, there are valid use cases for setting fake transports to accept, such as nested ESXi. In this case, we'll be sending all kinds of crazy packets!
Forged transmissions and MAC address changes are closely related.
10. Private VLAN in VMware
When we talk about port groups, we also talk about private VLANs.
Private VLANs are a more granular approach than isolating traffic at the VLAN level, which is why I brought it up after we started our security discussion.
Private VLANs are dedicated to the Distributed Virtual Switch , and the physical switch to which your ESXi hosts are connected must also support private VLANs.
When we use PVLANs, the VLANs we use break down in several ways:
- Primary PVLAN, which is promiscuous.
- There are two types of auxiliary PVLAN, community and isolated.
Let's look at the picture below, the Primary PVLAN is green and the secondary PVLAN is blue.
Virtual machine ports on an isolated PVLAN can only communicate with promiscuous ports on the primary PVLAN, and virtual machine ports cannot communicate with other ports on the isolated PVLAN.
Virtual machine ports on the community PVLAN and promiscuous ports on the primary PVLAN can communicate with each other.
11. VMware vSwitch Security Summary
When it comes to security policy settings for virtual switches, it's best to set them to deny unless you have a specific use case, and if you really have a reason to need to set promiscuous mode, change MAC addresses, or forge transmissions to accept, modify the security settings accordingly.
Private VLANs are another way to provide traffic segmentation in a virtual network environment, it is important to pay special attention to the secondary PVLAN type to ensure things work as expected.
The good news is that PVLAN settings are easy to change for troubleshooting purposes, as further described in the VMware vSphere networking documentation.
12. Basic knowledge of VMware vSphere network
Understanding VMware vSphere networking fundamentals is important for VMware administrators, network administrators, IT security analysts, and just about anyone who needs exposure to a VMware vSphere environment, and while many traditional networking concepts do apply to VMware vSphere, familiarity with virtualization How the layers are transformed is very important.