MTU, MSS, and why your VPN tunnels drop large packets

VPN tunnels have a hidden cost: encryption and encapsulation add overhead to every packet, shrinking the effective payload space. When a host sends a 1500-byte packet destined for a remote network over IPSec or SSL VPN, the tunnel endpoint wraps it in additional headers, creating a frame larger than the physical link's MTU. The result is silent packet loss, retransmission delays, and frustrated users reporting that 'the VPN is slow.' Understanding MTU, MSS, and how they interact across encrypted tunnels is essential for reliable network design.

What MTU and MSS actually do

Maximum Transmission Unit (MTU) is the largest IP packet a link can carry without fragmentation. On Ethernet, that is typically 1500 bytes. Maximum Segment Size (MSS) is the largest TCP payload the sender will place in a single segment, calculated as MTU minus IP header (20 bytes) minus TCP header (20 bytes), yielding 1460 bytes on standard Ethernet.

The relationship is simple: MSS constrains TCP data to fit inside an MTU-sized IP packet. Hosts negotiate MSS during the TCP three-way handshake using the MSS option. If both endpoints agree on MSS, TCP avoids sending packets that require IP-layer fragmentation.

Why VPN tunnels break this model

A VPN tunnel is an encapsulation layer. An IPSec tunnel adds at minimum 50-70 bytes of headers (ESP, ICV, outer IP). An SSL VPN adds record framing and encryption overhead. A GRE tunnel over IPSec can add 100+ bytes. When a 1500-byte packet enters the tunnel, the encapsulated result exceeds 1500 bytes and cannot be transmitted on a standard Ethernet link without fragmentation.

If the tunnel endpoint cannot fragment (or chooses not to), the packet is dropped. If fragmentation occurs inside the tunnel, reassembly overhead and retransmission on loss degrade throughput. The core problem: the host originating the traffic has no idea the tunnel exists and sends packets sized for a 1500-byte Ethernet link, not a 1430-byte effective tunnel payload.

Path MTU Discovery and the black hole problem

Path MTU Discovery (PMTUD) is the standard solution. A host sends packets with the Don't Fragment (DF) bit set. If a router cannot forward the packet without fragmenting, it sends an ICMP Fragmentation Needed message back to the source, reporting the link's MTU. The source then reduces its send size and retries.

VPN tunnels often break PMTUD. Firewalls may block ICMP messages for security reasons, or tunnel endpoints may not properly generate or relay them. Result: the host never learns the effective MTU is lower, keeps sending 1500-byte packets, and they silently drop inside the tunnel. This is the PMTUD black hole. Symptoms include sporadic connection failures, especially with large file transfers or video, while small requests work fine.

Practical solutions: MSS clamping and manual tuning

The most reliable fix is MSS clamping. The tunnel endpoint or a firewall intercepts TCP SYN packets and rewrites the MSS option to a safe value, typically 1380-1400 bytes for IPSec. This forces the source to send smaller segments that fit inside the tunnel after encapsulation, avoiding fragmentation and packet loss entirely.

# Cisco IOS IPSec MSS clamping example
interface Tunnel0
 ip address 10.0.0.1 255.255.255.0
 ip tcp adjust-mss 1379

# Palo Alto Networks
set security flow tcp-mss-adjust 1379

If clamping is not available, manually reduce the MTU on the tunnel interface or on hosts behind it. Test with ping and traceroute to confirm PMTUD is working. Use tools to calculate safe MSS values based on your tunnel overhead.

• →Enable MSS clamping on tunnel endpoints or firewalls
• →Verify ICMP is not blocked between tunnel endpoints
• →Test with ping -M do (Linux) or ping -f (Windows) to confirm PMTUD
• →Monitor tunnel statistics for fragmentation and retransmission
• →Document the effective MTU for your tunnel type and use it in design