IP MTU and TCP MSS Missmatch – an evil for network performance

By on 15 Dec 2014

Categories: Development Tech matters

Tags: , ,

41 Comments

Blog home

When we conduct a technical workshop, a common query from the participants relates to the Maximum Transmission Unit (MTU) size manipulation on a router interface and its relationship with the TCP Maximum Segment Size (MSS). I will try to discuss this in detail from a network engineers point of view.

The Maximum Transmission Unit (MTU) is the maximum length of data that can be transmitted by a protocol in one instance. If we take the Ethernet interface as an example, the MTU size of an Ethernet interface is 1500 bytes by default, which excludes the Ethernet frame header and trailer. It means that the interface cannot carry any frame larger then 1500 bytes. If we look inside the frame, we have a 20 byte IP header + 20 byte TCP header, leaving a 1460 byte of the payload that can be transmitted in one frame. This is what we refer to as TCP MSS. The diagram below visualizes this concept:

tcp-mtu-mss

 

 

 

In a normal transmission case, if there is no additional encapsulation (that is, IPsec, MPLS and so forth) carried out on a transiting router, the source device may use the maximum payload length of 1460 bytes without any potential risk of packet fragmentation/drop. This is negotiated during the TCP three-way handshake stage between the source and destination host.

Now in a likely case where a router in transit is carrying out additional encapsulation, that is, MPLS label swapping, this will add an additional label header as below:

tcp-mss-mpls

 

 

 

This will eventually increase the size of the frame exiting a transiting router (in the case above it is 1508 bytes.) It will also create similar issue for IPSec and IPv6 in IPv4 tunnel etc.

The maximum MTU of an interface will depend on the hardware platform, but the IEEE 802.3 standards require a minimum MTU of 1500 bytes. Also if you notice the following CLI on a router, the maximum IP MTU is capped at the Ethernet MTU of 1500 bytes.

interface-mtu

 

The problem is we cannot increase the IP MTU size on the router Ethernet interface because the MPLS label encapsulation frame size can potentially exceed the maximum MTU. If the source device creates a full size packet with a TCP MSS of 1460 bytes, it is likely this transiting router will drop/fragment the packet. This is bad for our network performance.

What is the best way to solve this?

Shrinking the IP MTU on the interface to 1448 bytes  will create space for 12 additional bytes on the MPLS label header [if you consider a maximum of 3 MPLS label LDP + VPN + TE] plus a 20 byte TCP + 20 byte IP header within a 1500 byte Ethernet MTU.

Please remember, when we shrink the IP MTU on a transiting router, that will also shrink the maximum possible TCP MSS size to 1448 bytes from the source device. If the router does not signal the source and destination during the TCP handshake, the optimal TCP MSS  could potentially create a dropping/fragmentation problem.

Here are the steps to solve this:

Transiting traffic (traffic going via this router):

Router1(config)#int ethernet 1/0
Router1(config-if)#ip mtu 1448

The above command will shrink the IP MTU on the interface to 1448 bytes, creating a 12 byte space for the MPLS label header.

Router1(config-if)#ip tcp adjust-mss ?
<500-1460>
Router1(config-if)#ip tcp adjust-mss 1448

The above command will signal the source and destination device during the three-way handshake to use the TCP MSS size of 1448 bytes so that if they create the full size packet there will still not be any drop/fragmentation on the router. 

Terminating traffic (doing SSH, telnet to this router):

Router1(config)# ip tcp mss 1448?
<68-10000>  MSS

I do hope this blog piece will help you to understand how to handle the IP MTU and TCP MSS size on a router interface.

 

Read another technical guide by Nurul.

Rate this article

The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.

41 Comments

    1. John

      media-router-fp2.prod1.media.vip.bf1.yahoo.com.http

      I keep seeing this in my server tcpdump logs? spamming me with http requests? I’m not connected to my server via vpn only ssh……. why is this happening lol I saw it has an [mss] attached to it and the numbers keep changing

      Reply
  1. Nicolette Foote

    Thanks. Any firewall in between need to allow ICMP I suppose. Otherwise TCP MSS adjust message will be blocked. Correct me if I misunderstood.

    Reply
    1. Roman

      Yes you are correct. Usually router send ICMP message “Packet too big” to the sender PC. For that you need to allow ICMP Type3 code 4 on the firewall.

      Reply
  2. cniki10

    I think i’m missing something: should’t ip mtu be shrink to 1488 and tcp to mms to 1448 ?
    Thus,
    1488 + 12 = 1500 => max eth. size
    1448 + 12 + 20 + 20 =1500 also ?

    Reply
    1. Roman

      Yes correct, if it only 3 MPLS label IP MTU can be 1488 and TCP MSS 1448.

      Note: IPv4 header is variable. Including option it could be 60byte. To avoid ambiguity we used 1448. Thanks.

      Reply
      1. Rafael

        Ok .. so even worse, no? So accounting for IP mtu could be 60 …

        MSS = IP MTU – (headers) = 1488 – 60 (IP) – 20 (tcp)
        MSS = 1408

        Reply
  3. Burhan

    Hi Romon,
    I have one confusion here. if MTU of ethernet interface is 1500 bytes, then why we are not including the overhead of ethernet layer itself in MTU calculation (18 bytes).

    1500 = 20 (TCP) + 20 (IP) + 1460 (IP payload)
    1518 = 20 (TCP) + 20 (IP) + 1460 (IP payload) + 18 (Eth)

    so TCP-MSS should not be 1442 if interface MTU is 1500.

    Reply
    1. Roman

      Hi Burhan,

      Thanks for the comments.

      This is IP MTU of the interface and which excludes the Ethernet frame header and trailer. On the diagram it is explained.

      Regards

      Reply
  4. Roy

    Hi Roman,

    Thanks for the wonderful explanation. Things has been presented in its simplest and understandable form, however a small question.
    In the example you have hard coded the MTU of the interface to 1448, so that supposed to mean the router via that particular interface will forward that bytes of traffic minus the frame header and trailor.
    i.e :
    1448+4+4= 1456.
    Also why do we have to lower the MTU? Instead can’t we simple downgrade the MSS? it should serve the same purpose right ?

    Thanks
    Roy

    Reply
    1. Roman

      Thanks Roy. Yes you are correct. To signal downgrade the TCP MSS used below config.

      Router1(config-if)#ip tcp adjust-mss 1448

      -Roman

      Reply
  5. Vijay

    Hi Roman,

    Nice blog. And in some case am sending with full payload and when the packet is in transit and if it encounters with a router that is having let me say 1200 (for example), then the router will fragment the packet and will re transmit it right?
    So if thats the case then destination device how it will defragment the recieved packets? and what if one more router with MTU size 1500 is present, then will it do de fragment in order to utilise full B/W?
    Thanks. (these MTU sizes just giving for better understanding)

    Reply
    1. Roman

      Thanks Vijay. To answer your first question if the DF bit is set from the source then the router will simply drop the packet. If DF bit is not set then the router will fragment it. On the next hop router if the fragmented packets are not allowed (most likely) they will drop it.
      Answering second part of the question. De-fragmentation will be done on TCP. Will add overhead, best effort basis etc. Hope this answer your question.

      Reply
  6. Yaswanth

    Hi Roman,

    Thanks for the post.

    IP MTU = Ethernet Frame – [IP header (20) + TCP header + segment size (20) – MPLS Labels (12)]

    1448 = 1500 – 20-20-12

    If IP header is 60 bytes then

    IP MTU = 1408 = 1500 – 60 – 20 – 12

    Is my understanding correct ?

    Reply
    1. Roman

      Thanks Yaswanth. Yes your understanding is correct.

      Just a note: On high end routers normally packets are processed by hardware based line card and IPv4 “Option headers” packets are passed to CPU. Sometime operators prefer to discard those packets with Option header on (60 byte long packet).

      Reply
  7. Roxy

    Hi Roman, some problem to understand exactly how to check MTU and where to set interface smaller MTU and who have to send ICMP message “Packet too big” to the sender machine ? can you explane best ie if we have a firewall with 4 interface int, lan, dmz and a tinc vpn tun interface, on int interface we go on Internet where some machine can be configured to filter ICMP (blackhole) and internally on lan we have a machine doing VPN server (softeth VPN Server) ? ie. if we have a pc in lan with MTU =1500 and we send a packet with ping greater then supported MTU with DF bit set we get “need to fragment” and a drop, but no packet travel on the network (on the gateway tcpdump does not see this packet try to flow). Is the local machine to prohibit to send packet like this ?

    Reply
    1. Roman

      Thanks Roxy. In your case I suppose you will get “packet too big” ICMP message back from the gateway. In your experiment both source and destination should stay with default MTU. On Cisco boxes if you don’t use “ip tcp adjust-mss” command then any bigger packet will be dropped. If you use this command on the gateway then the gateway will send ICMP message back to the source to adjust the TCP MSS. Your maximum MTU is fixed on the ethernet interface anyway. So only way to reduce the TCP MSS sizs on the source to get the packet through.

      Reply
  8. maggie

    Since every TCP segment has more than 20bytes in its header as we are usually exchanging mss and window size in every packet sent across. so 1460 won’t do it… wouldn’t it be better to aim for 1452, that is 1-2 rows of options that would at least cover those two options?

    Reply
  9. Randhir

    Hi Roman,

    I am using ADSL2+ connection with PPPOA configuration. Internet connection is working fine but some of the websites not opening.

    Spoke to ISP who advise to adjust MTU to 1400

    My question is do I have to do both mtu size and TCP adjust to make it work?

    Like:

    int dialer 1
    ip mtu 1400
    ip tcp adjust-mss 1360

    or just using ip mtu cmd is enough.

    Another question Cisco router has ATM0 interface for dialer connectin do I have to make same adjustments to ATM int as well.

    Thanks
    Randhir

    Reply
  10. Jigar

    Hi,

    I have one mpls network and in network we are facing problem of Mtu. due to mtu my some application not working which is only working through 9000 mtu.

    So kindly suggest.

    Reply
  11. Farha Diba

    Hi Roman Bhai,

    I have some silly question. In some MPLS network, 1532 bytes MTU is used. How to calculate this MTU?
    Does this MTU has used Cisco and Mikrotik both devices?
    If anyone wants 9012 bytes in an MPLS network. Is it possible to deploy to any network consists of Cisco/Mikrotik devices?

    Thanks
    Farha Diba

    Reply
    1. Jessica

      Dear Farha,

      I have listed my answer as following, feel free to contact me for further queries.

      Q: In some MPLS network, 1532 bytes MTU is used. How to calculate this MTU?
      A: On some devices, 1532 is the Maximum MTU of fast Ethernet interface, then some people use this value to be on the safe side.

      Q: Does this MTU has used Cisco and Mikrotik both devices?
      A: Yes, it is strongly recommended that MPLS MTU is configured to the same value on all routers forming MPLS cloud because of effects MPLS MTU has on MPLS switched packets. But which exact value is used, you need to consider about the services in your network.

      Q: If anyone wants 9012 bytes in an MPLS network. Is it possible to deploy to any network consists of Cisco/Mikrotik devices?
      A: MTU ranges of different devices and interface types are different, you need to actually check these on your device or check the specifications with vendors.

      Reply
  12. Ronald Lee

    Please help to advice if correct or not.

    Assumption:
    1. Ethernet MTU size : 1500 byte (Ethernet header & FCS are not included)
    2. MPLS label size: 3 MPLS label LDP + VPN + TE: 12 byte
    3. 20 byte TCP + 20 byte IP header
    4. TCP MSS size = payload size
    5. IP MTU size = payload size + TCP header + IP header
    6. MPLS MTU size = payload size + TCP header + IP header + MPLS label size

    Therefore:
    IP MTU = payload size + TCP header + IP header = Ethernet MTU – MPLS Labels =1500-12 = 1488

    TCP MSS = Ethernet MTU – IP header (20) – TCP header (20) – MPLS Labels (12) = 1500-20-20-12=1448

    Conclustion:
    Shall configure with ip mtu 1488 instead of ip mtu 1448.

    Router1(config)#int ethernet 1/0
    Router1(config-if)#ip mtu 1488
    Router1(config-if)#ip tcp adjust-mss 1448

    Reply
  13. TruthTeller

    I can’t believe so many people praised this article. The “ip mtu” and “tcp adjust-mss” CANNOT both be 1448 !!!

    Reply
  14. Seth

    Why do I have to change the IP MTU to 1488. I get the TCP MSS part of changing it to 1448. Shouldn’t only that command be just needed “ip adjust-mss” as we are talking about routing/forwarding MPLS Lablelled packet?? Unless we are also assuming the router to forwards plain IP packet

    Lets say a MPLS Labelled packet with 3 Labels comes in a transiting router. The transit outgoing interface mpls mtu is 1500 (default) based on the interface max MTU setting, So any Layer 3 packet > 1488 will cause fragmentation. I have already signaled with tcp adjust-mss 1460 max segment size to originating router.. Shouldn’t this command be enough. ??

    Reply
  15. Seth

    Just rephrasing the typo..

    I have already signaled with tcp adjust-mss 1448 max segment size to originating router.. Shouldn’t this command be enough. ??

    Reply
  16. vm

    “..we cannot increase the IP MTU size on the router Ethernet interface because the MPLS label encapsulation frame size can potentially exceed the maximum MTU.”
    “What is the best way to solve this? Shrinking the IP MTU..”

    We cannot increase the MTU because the frame will exceed the MTU. Therefore, we will reduce the MTU.
    I didn’t understand how it works.

    Reply
  17. Anderson Stivanelli

    Doubt about making MTU adjustment on an L3 interface with the same value for clamp mss, in theory this value of 1448 will still be lost in the 40bytes of the ip + tcp headers.
    It is worth thinking about what is mtu L2 and mtu L3 and mtu tcp.

    This article may need to be reworded.

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

Top