Understanding ESX/ESXi Load Balancing
Of Virtual Networking Concepts
Background
For better understanding VM ESX/ESXi vswitch load balancing method, I did some tests using an ESX server hosting two virtual servers and a Cisco3560G switch as access switch. Here are test scenarios and results.
Scenario One:
Load balancing method: Route based on the originating virtual switch port ID
“Choose an uplink based on the virtual port where the traffic entered the virtual switch. This is the default configuration and the one most commonly deployed.
When you use this setting, traffic from a given virtual Ethernet adapter is consistently sent to the same physical adapter unless there is a failover to another adapter in the NIC team.
Replies are received on the same physical adapter as the physical switch learns the port association.
This setting provides an even distribution of traffic if the number of virtual Ethernet adapters is greater than the number of physical adapters. ” - Virtual Networking Concepts, p8
AU-3560G-LAB#sh mac-address-table vlan 252
Mac Address Table
-------------------------------------------
Vlan Mac Address Type Ports
---- ----------- -------- -----
…
252 000c.29c6.d1ec DYNAMIC Gi0/11
252 000c.29c6.d1f6 DYNAMIC Gi0/5
252 000c.29dc.2883 DYNAMIC Gi0/11
…
Total Mac Addresses for this criterion: 28
AU-3560G-LAB#sh mac-address-table vlan 252
Mac Address Table
-------------------------------------------
Vlan Mac Address Type Ports
---- ----------- -------- -----
…
252 000c.29c6.d1ec DYNAMIC Gi0/11
252 000c.29c6.d1f6 DYNAMIC Gi0/5
252 000c.29dc.2883 DYNAMIC Gi0/11
…
AU-3560G-LAB#sh int g0/5
GigabitEthernet0/5 is up, line protocol is up (connected)
…
Output queue: 0/40 (size/max)
30 second input rate 0 bits/sec, 0 packets/sec
30 second output rate 0 bits/sec, 0 packets/sec
953540480 packets input, 3646569118 bytes, 0 no buffer
…
AU-3560G-LAB#sh int g0/11
GigabitEthernet0/11 is up, line protocol is up (connected)
…
Output queue: 0/40 (size/max)
30 second input rate 682973000 bits/sec, 59657 packets/sec
30 second output rate 3499000 bits/sec, 6432 packets/sec
777185120 packets input, 85576404 bytes, 0 no buffer
…
From switch statistics, we can see that “traffic from and to a given virtual Ethernet adapter is consistently sent to the same physical adapter”. But VM servers were only using one upstream link. There are some articles online also mentioned this problem. That “Route based on the originating virtual switch port ID” might over utilize one physical upstream link while other links are idle.
“There is no guarantee that the VMs will be evenly distributed across the uplinks on a vSwitch or port group. Although I’ve seen references in the VMware documentation to indicate that VMs are balanced in some fashion (I could not find those references, however), real-world experience seems to indicate otherwise. In my tests, I had instances where four VMs were all “linked” (via their virtual port ID) to the same uplink.”- [2]
Scenario Two:
Load balancing method: Route based on source MAC hash
Choose an uplink based on a hash of the source Ethernet MAC address.
“When you use this setting, traffic from a given virtual Ethernet adapter is consistently sent to the same physical adapter unless there is a failover to another adapter in the NIC team.
Replies are received on the same physical adapter as the physical switch learns the port association.
This setting provides an even distribution of traffic if the number of virtual Ethernet adapters is greater than the number of physical adapters. “- Virtual Networking Concepts, p8
AU-3560G-LAB#sh mac-address-table vlan 252
Mac Address Table
-------------------------------------------
Vlan Mac Address Type Ports
---- ----------- -------- -----
…
252 000c.29c6.d1ec DYNAMIC Gi0/11
252 000c.29c6.d1f6 DYNAMIC Gi0/11
252 000c.29dc.2883 DYNAMIC Gi0/5
…
AU-3560G-LAB#sh mac-address-table vlan 252
Mac Address Table
-------------------------------------------
Vlan Mac Address Type Ports
---- ----------- -------- -----
…
252 000c.29c6.d1ec DYNAMIC Gi0/11
252 000c.29c6.d1f6 DYNAMIC Gi0/11
252 000c.29dc.2883 DYNAMIC Gi0/5
…
AU-3560G-LAB#sh int g0/5
GigabitEthernet0/5 is up, line protocol is up (connected)
…
Output queue: 0/40 (size/max)
30 second input rate 312931000 bits/sec, 27229 packets/sec
30 second output rate 1730000 bits/sec, 3181 packets/sec
…
AU-3560G-LAB#sh int g0/11
GigabitEthernet0/11 is up, line protocol is up (connected)
…
Output queue: 0/40 (size/max)
30 second input rate 361261000 bits/sec, 31594 packets/sec
30 second output rate 1897000 bits/sec, 3486 packets/sec
788557468 packets input, 3465182387 bytes, 0 no buffer
…
From switch statistics, we can see that “traffic from and to a given virtual Ethernet adapter is consistently sent to the same physical adapter and even distribution of traffic”.
Scenario Three:
Load balancing method: Route based on IP hash
“Choose an uplink based on a hash of the source and destination IP addresses of each packet. (For non-IP packets, whatever is at those offsets is used to compute the hash.)
Evenness of traffic distribution depends on the number of TCP/IP sessions to unique destinations. There is no benefit for bulk transfer between a single pair of hosts.
You can use link aggregation — grouping multiple physical adapters to create a fast network pipe for a single virtual adapter in a virtual machine.
When you configure the system to use link aggregation, packet reflections are prevented because aggregated ports do not retransmit broadcast or multicast traffic.
The physical switch sees the client MAC address on multiple ports. There is no way to predict which physical Ethernet adapter will receive inbound traffic.
All adapters in the NIC team must be attached to the same physical switch or an appropriate set of stacked physical switches. “- Virtual Networking Concepts, p8
AU-3560G-LAB#sh mac-address-table vlan 252
Mac Address Table
-------------------------------------------
Vlan Mac Address Type Ports
---- ----------- -------- -----
…
252 000c.29c6.d1ec DYNAMIC Gi0/5
252 000c.29dc.2883 DYNAMIC Gi0/5
…
AU-3560G-LAB#sh mac-address-table vlan 252
Mac Address Table
-------------------------------------------
Vlan Mac Address Type Ports
---- ----------- -------- -----
…
252 000c.29c6.d1ec DYNAMIC Gi0/5
252 000c.29dc.2883 DYNAMIC Gi0/11
252 8071.1f75.9400 DYNAMIC Po1
…
AU-3560G-LAB#sh mac-address-table vlan 252
Mac Address Table
-------------------------------------------
Vlan Mac Address Type Ports
---- ----------- -------- -----
…
252 000c.29c6.d1ec DYNAMIC Gi0/5
252 000c.29dc.2883 DYNAMIC Gi0/11
…
AU-3560G-LAB#sh mac-address-table vlan 252
Mac Address Table
-------------------------------------------
Vlan Mac Address Type Ports
---- ----------- -------- -----
…
252 000c.29c6.d1ec DYNAMIC Gi0/5
252 000c.29dc.2883 DYNAMIC Gi0/5
…
AU-3560G-LAB#sh mac-address-table vlan 252
Mac Address Table
-------------------------------------------
Vlan Mac Address Type Ports
---- ----------- -------- -----
…
252 000c.29c6.d1ec DYNAMIC Gi0/5
252 000c.29dc.2883 DYNAMIC Gi0/11
…
…
AU-3560G-LAB#sh mac-address-table vlan 252
Mac Address Table
-------------------------------------------
Vlan Mac Address Type Ports
---- ----------- -------- -----
…
252 000c.29c6.d1ec DYNAMIC Gi0/11
252 000c.29dc.2883 DYNAMIC Gi0/11
…
AU-3560G-LAB#sh int g0/5
GigabitEthernet0/5 is up, line protocol is up (connected)
…
Output queue: 0/40 (size/max)
30 second input rate 376834000 bits/sec, 32902 packets/sec
30 second output rate 1993000 bits/sec, 3666 packets/sec
1039995654 packets input, 3092458811 bytes, 0 no buffer
…
AU-3560G-LAB#sh int g0/11
GigabitEthernet0/11 is up, line protocol is up (connected)
…
Output queue: 0/40 (size/max)
30 second input rate 317242000 bits/sec, 27656 packets/sec
30 second output rate 1699000 bits/sec, 3122 packets/sec
876193283 packets input, 4214465776 bytes, 0 no buffer
…
From switch statistics, we can see that “traffic from and to a given virtual Ethernet adapter is sent to different physical adapters and is unpredictable”.
Conclusions:
Here are conclusions when choosing VM ESX/ESXi vswitch load balancing methods during the network practice.
1. If you are using one server with two NICs and one upstream physical switch, you can use one of either three LB methods. But “Route based on the originating virtual switch port ID” is not recommended.
When you use “Route based on IP hash” method, you should configure link aggregation to prevent packet reflections - retransmit broadcast or multicast traffic.
Route based on the originating virtual switch port ID
Route based on source MAC hash
Route based on IP hash - Link Aggregation
2. If you are using one server with two NICs and two upstream physical switches, you can use one of either “Route based on the originating virtual switch port ID” or “Route based on source MAC hash” LB methods. But “Route based on the originating virtual switch port ID” is not recommended.
· When using “Route based on source MAC hash”, the port that VM will use is not predictable. This might not be preferable for some network setup.
· You should not use “Route based on IP hash” when you have multiple upstream switches.
Route based on the originating virtual switch port ID
Route based on source MAC hash
3. If you are using one server with multiple NICs and multiple upstream physical switches, you can use one of either “Route based on the originating virtual switch port ID” or “Route based on source MAC hash” LB methods. But “Route based on the originating virtual switch port ID” is not recommended.
· When using “Route based on source MAC hash”, the port that VM will use is not predictable. This might not be preferable for some network setup. Because you have multiple NICs to use, you can setup multiple vswitches and put different servers onto them to achieve network load balancing.
· You should not use “Route based on IP hash” when you have multiple upstream switches.
Route based on source MAC hash
Route based on the originating virtual switch port ID – With Multiple Vswtiches
Reference:
Virtual Networking Concepts - http://www.vmware.com/files/pdf/virtual_networking_concepts.pdf
[2] More on VMware ESX NIC Utilization - http://blog.scottlowe.org/2008/10/08/more-on-vmware-esx-nic-utilization/
