Networking

Runecast Academy Series 1 – Part 6. Networking

So, we have the virtual machine driven by the hypervisor with the data stored on the datastore. Now you would like to connect to that and share some data with it, right? Let's configure the network on the ESXi server.

vSphere Standard Switch

The heart of ESXi networking is the vSphere Standard Switch (vSwitch), a quite simple but very stable and powerful service that works like a basic Layer 2 switch, transferring data according to the MAC addresses. vSwitch doesn't need to learn MAC addresses from the traffic, because ESXi already knows what MAC address is configured on which virtual machine network card. This might cause a headache in case you want to run VM inside VM because the super-virtualized MAC is not known to the physical ESXi host. vSwitch doesn't have any separate management, it is an integral part of the ESXi server and we can use ESXi (or vCenter) management tools to configure it. The vSwitch is defined and configured separately for each ESXi server.

Port Groups

How about connecting some virtual machines to the vSwitch? We will need to define the port group(s). It may sound a bit improbable, but the port group (AKA portgroup) name is not just a label, it is the main identifier of the port group. The VM network card is connected to the port group by name and when we migrate the VM between ESXi hosts, it will be automatically connected to the port group with the same name as the origin, it doesn't matter how that is configured. If we are not careful enough, our VM may become easily offline after vMotion.

Each port group has configured VLAN policy. It can be 0 when native VLAN is configured on the physical switch port. That brings a slight labyrinth to the network design. It can be a real VLAN ID when VLANs are in a trunk on the physical switch port. And it can be 4095, meaning that the data are sent to the VM as is, and the OS itself is handling VLAN tagging. There needs to be a really good reason to use the last option.

From the virtual switch point of view, VMkernel network adapters are special kinds of port groups, the only difference is that we don't have connected VMs there, but ESXi management is communicating through that.

Virtual Machine Network Cards

VMs can be configured with up to 10 virtual network cards (vNICs). Usually, we are using VMXNET 3, or possibly E1000 vNIC type. E1000 is the emulation of the generic Intel E1000 network card. Most of the operating systems support that and we can start using it right away. But, it is an emulated physical card, so the OS has to handle it as a physical device with all the interruptions and calls, and the ESXi on the other hand has to emulate that. As Intel E1000 is a 1Gb card, such throughput might not be good enough. If we are not using a super-exotic OS, then we have VMXNET 3 supported by the OS itself, or the driver can be installed as part of the VMware Tools. How does VMXNET 3 look from the OS side? We can imagine it as a hole – one into which we are simply dropping data and which is spitting out replies from the network. The OS doesn't need to handle it anyhow, and the speed is limited only by the host CPU.

When we have two VMs running on the same ESXi host connected to the same subnet and try to ping from one to the other, data enters the vSwitch. vSwitch already knows the virtual port MAC address of the target VM, and when the port group setting allows it, the data is directly delivered to the target VM.

Physical Network Adapters

We will probably want to access our VM from the outer world. Then we need some uplink ports (physical network cards) assigned to the vSwitch. Physical NICs with speed up to 100 Gb are supported, but be very careful when selecting and installing firmware and drivers, and make sure that the used combination is aligned to the Hardware Compatibility List (HCL). The unsupported combination may lead to connectivity problems or even to a PSOD.

To keep high availability, we want to use at least two uplinks per virtual switch. How does the vSwitch decide which data goes through which physical cable? By default, for each virtual network card one physical network card is "randomly" (well, "randomly" according to the virtual port ID) selected, and that will change only in the case that the physical connection is broken. We can achieve slightly more balanced uplinks utilization by using Route Based on IP Hash method. Then the path is selected for each packet separately, costing us some CPU power, and it might not be expected by the physical switches. We can also set up a strict active-passive policy.

Security

We usually don't need to touch the vSwitch security settings. But the VM owner, typically from the security or network team, may appear beside our desk forcing us to fine-tune some of the security settings for his or her VM. They will most probably want to have allowed Promiscuous mode. If we change this setting to Accept, the VM will receive all the frames which are allowed on the portgroup by configured VLAN policy, it doesn't matter if they are addressed to the VM or not. Standard vSwitch doesn't support port mirroring and Promiscuous mode can be used as a workaround in this case. To get all the vSwitch traffic into the monitoring VM, use it in combination with VLAN configured to 4095.

"Physical" MAC address is configured on the VMware side, and this is typically reused by the operating system, but the OS can rewrite the setting. Without special reasons, we don't want to allow this, because such a virtual machine could then send requests instead of another server, or could receive data designated to something else. MAC address changes security setting switched to Accept causes that frames are delivered to the VM adapter, even if the MAC configured on the VMware side differs. To allow the VM to send packets with different MAC addresses, accept Forged transmits.

Conclusion

vSphere Standard Switch is a lightweight service independent of vCenter or any other centralized management. Using that might be slightly limiting, but it conversely brings independence on the vCenter and, as a bonus, it is included in the Standard vSphere license. Alternatively, we can use vSphere Distributed Switch (dvSwitch) which will be described in a future Runecast Academy article.