Seeing Jumbo Frames even though they are disabled on the NIC?

We have several servers running Windows Server 2012R2 on top of VMware. In trying to diagnose some slowdowns we noticed that many of our servers are sending out lots of jumbo frames. This seems odd since jumbo frames aren’t enabled on any of the vm nics.

Jumbo Frames

You can see that obviously we are sending out lots of jumbo frames and receiving a bunch too (albeit much much less).

Ping Example

…but the problem is that jumbo packets aren’t enabled in any of the virtual machines. How are they constructing the jumbo frames in the first place?

The ping screenshot is from the server with the most jumbo frames out to the server with the most jumbo frames in. You can see in the ping example that anything above 1500 bytes will fail. The 1472 byte payload (+ 20 bytes IP header + 8 bytes ICMP header) succeeds, but the 1473 byte ping fails.

Where are you capturing the packets? is it from a span or a tap or Mirror?
From what I have seen Extrahop classify’s anything over 1520 as Jumbo frames. I would suggest you look at ICMP messages from and to that machine as well. if your network only allows 1500 mtu you should see a bunch of ICMP type3 code 3 (I believe) being returned. is there any additional overhead i.e. NSX type of network encapsulation happening?

@mitchroberson correctly calls out encapsulation likely being the culprit here. Note that Jumbo frame identification is done at L2, so any sort of L2/L3 encapsulation that might be occurring on the network infrastructure upstream of the VM would cause these max MTU/MSS packets to be counted as Jumbo frames. Here are just a few encapsulation types that could potentially be in play:

Q-in-Q (802.1q)
Cisco VNTag

Try navigating to the device(s) associated with the Jumbo frames, then look at the device’s “Network” page, then scroll toward the bottom and take a look at the “Frame Types” section. Anything jump out at you there? Do you see “Other” frame types that correspond to these Jumbo frames?

1 Like

Seems like I’ve also seen Jumbo Frames show up with the use of either TOE (TCP Offload Engine) or TCP Large Send Offload… It’s been a good while since I chased that down, but I seem to recall something to the effect that TOE being enabled bypasses certain aspects of the OS’s TCP controls…