When we conduct a technical workshop, a common query from the participants relates to the Maximum Transmission Unit (MTU) size manipulation on a router interface and its relationship with the TCP Maximum Segment Size (MSS). I will try to discuss this in detail from a network engineers point of view.
The Maximum Transmission Unit (MTU) is the maximum length of data that can be transmitted by a protocol in one instance. If we take the Ethernet interface as an example, the MTU size of an Ethernet interface is 1500 bytes by default, which excludes the Ethernet frame header and trailer. It means that the interface cannot carry any frame larger then 1500 bytes. If we look inside the frame, we have a 20 byte IP header + 20 byte TCP header, leaving a 1460 byte of the payload that can be transmitted in one frame. This is what we refer to as TCP MSS. The diagram below visualizes this concept:
In a normal transmission case, if there is no additional encapsulation (that is, IPsec, MPLS and so forth) carried out on a transiting router, the source device may use the maximum payload length of 1460 bytes without any potential risk of packet fragmentation/drop. This is negotiated during the TCP three-way handshake stage between the source and destination host.
Now in a likely case where a router in transit is carrying out additional encapsulation, that is, MPLS label swapping, this will add an additional label header as below:
This will eventually increase the size of the frame exiting a transiting router (in the case above it is 1508 bytes.) It will also create similar issue for IPSec and IPv6 in IPv4 tunnel etc.
The maximum MTU of an interface will depend on the hardware platform, but the IEEE 802.3 standards require a minimum MTU of 1500 bytes. Also if you notice the following CLI on a router, the maximum IP MTU is capped at the Ethernet MTU of 1500 bytes.
The problem is we cannot increase the IP MTU size on the router Ethernet interface because the MPLS label encapsulation frame size can potentially exceed the maximum MTU. If the source device creates a full size packet with a TCP MSS of 1460 bytes, it is likely this transiting router will drop/fragment the packet. This is bad for our network performance.
What is the best way to solve this?
Shrinking the IP MTU on the interface to 1448 bytes will create space for 12 additional bytes on the MPLS label header [if you consider a maximum of 3 MPLS label LDP + VPN + TE] plus a 20 byte TCP + 20 byte IP header within a 1500 byte Ethernet MTU.
Please remember, when we shrink the IP MTU on a transiting router, that will also shrink the maximum possible TCP MSS size to 1448 bytes from the source device. If the router does not signal the source and destination during the TCP handshake, the optimal TCP MSS could potentially create a dropping/fragmentation problem.
Here are the steps to solve this:
Transiting traffic (traffic going via this router):
Router1(config)#int ethernet 1/0
Router1(config-if)#ip mtu 1448
The above command will shrink the IP MTU on the interface to 1448 bytes, creating a 12 byte space for the MPLS label header.
Router1(config-if)#ip tcp adjust-mss ?
<500-1460>
Router1(config-if)#ip tcp adjust-mss 1448
The above command will signal the source and destination device during the three-way handshake to use the TCP MSS size of 1448 bytes so that if they create the full size packet there will still not be any drop/fragmentation on the router.
Terminating traffic (doing SSH, telnet to this router):
Router1(config)# ip tcp mss 1448?
<68-10000> MSS
I do hope this blog piece will help you to understand how to handle the IP MTU and TCP MSS size on a router interface.
Read another technical guide by Nurul.
The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.
Good and simple solution to avoid drop/fragmentation in router.
media-router-fp2.prod1.media.vip.bf1.yahoo.com.http
I keep seeing this in my server tcpdump logs? spamming me with http requests? I’m not connected to my server via vpn only ssh……. why is this happening lol I saw it has an [mss] attached to it and the numbers keep changing
It’s really a good one to adjust and work with MTU. Thank you very much.
Thanks. Any firewall in between need to allow ICMP I suppose. Otherwise TCP MSS adjust message will be blocked. Correct me if I misunderstood.
Yes you are correct. Usually router send ICMP message “Packet too big” to the sender PC. For that you need to allow ICMP Type3 code 4 on the firewall.
Its really important and helpful to avoid dropping packet. Thank you bhaia.
Thank you Roman bhaia for your nice article. If you create a video tutorial with example about this topic, It will be helpful for us.
Thanks to everyone for your valuable comments.
The article is precise but in-depth light shading on the MTU issue. Thanks for nice article.
I think i’m missing something: should’t ip mtu be shrink to 1488 and tcp to mms to 1448 ?
Thus,
1488 + 12 = 1500 => max eth. size
1448 + 12 + 20 + 20 =1500 also ?
Yes correct, if it only 3 MPLS label IP MTU can be 1488 and TCP MSS 1448.
Note: IPv4 header is variable. Including option it could be 60byte. To avoid ambiguity we used 1448. Thanks.
Ok .. so even worse, no? So accounting for IP mtu could be 60 …
MSS = IP MTU – (headers) = 1488 – 60 (IP) – 20 (tcp)
MSS = 1408
MTU should be configured with 1488=1500-12(space for label MPLS)
and THE MSS=1460-12=1488
and MSS=1460-12=1488
sorry
corect is
MSS=1460-12=1448
Hi Romon,
I have one confusion here. if MTU of ethernet interface is 1500 bytes, then why we are not including the overhead of ethernet layer itself in MTU calculation (18 bytes).
1500 = 20 (TCP) + 20 (IP) + 1460 (IP payload)
1518 = 20 (TCP) + 20 (IP) + 1460 (IP payload) + 18 (Eth)
so TCP-MSS should not be 1442 if interface MTU is 1500.
Hi Burhan,
Thanks for the comments.
This is IP MTU of the interface and which excludes the Ethernet frame header and trailer. On the diagram it is explained.
Regards
Hi Roman,
Thanks for the wonderful explanation. Things has been presented in its simplest and understandable form, however a small question.
In the example you have hard coded the MTU of the interface to 1448, so that supposed to mean the router via that particular interface will forward that bytes of traffic minus the frame header and trailor.
i.e :
1448+4+4= 1456.
Also why do we have to lower the MTU? Instead can’t we simple downgrade the MSS? it should serve the same purpose right ?
Thanks
Roy
Thanks Roy. Yes you are correct. To signal downgrade the TCP MSS used below config.
Router1(config-if)#ip tcp adjust-mss 1448
-Roman
Hi Roman,
Nice blog. And in some case am sending with full payload and when the packet is in transit and if it encounters with a router that is having let me say 1200 (for example), then the router will fragment the packet and will re transmit it right?
So if thats the case then destination device how it will defragment the recieved packets? and what if one more router with MTU size 1500 is present, then will it do de fragment in order to utilise full B/W?
Thanks. (these MTU sizes just giving for better understanding)
Thanks Vijay. To answer your first question if the DF bit is set from the source then the router will simply drop the packet. If DF bit is not set then the router will fragment it. On the next hop router if the fragmented packets are not allowed (most likely) they will drop it.
Answering second part of the question. De-fragmentation will be done on TCP. Will add overhead, best effort basis etc. Hope this answer your question.
Hi Roman,
Thanks for the post.
IP MTU = Ethernet Frame – [IP header (20) + TCP header + segment size (20) – MPLS Labels (12)]
1448 = 1500 – 20-20-12
If IP header is 60 bytes then
IP MTU = 1408 = 1500 – 60 – 20 – 12
Is my understanding correct ?
Thanks Yaswanth. Yes your understanding is correct.
Just a note: On high end routers normally packets are processed by hardware based line card and IPv4 “Option headers” packets are passed to CPU. Sometime operators prefer to discard those packets with Option header on (60 byte long packet).
Small correction
IP MTU = Ethernet Frame – [IP header (20) + TCP header (20) – MPLS Labels (12)]
Hi Roman, some problem to understand exactly how to check MTU and where to set interface smaller MTU and who have to send ICMP message “Packet too big” to the sender machine ? can you explane best ie if we have a firewall with 4 interface int, lan, dmz and a tinc vpn tun interface, on int interface we go on Internet where some machine can be configured to filter ICMP (blackhole) and internally on lan we have a machine doing VPN server (softeth VPN Server) ? ie. if we have a pc in lan with MTU =1500 and we send a packet with ping greater then supported MTU with DF bit set we get “need to fragment” and a drop, but no packet travel on the network (on the gateway tcpdump does not see this packet try to flow). Is the local machine to prohibit to send packet like this ?
Thanks Roxy. In your case I suppose you will get “packet too big” ICMP message back from the gateway. In your experiment both source and destination should stay with default MTU. On Cisco boxes if you don’t use “ip tcp adjust-mss” command then any bigger packet will be dropped. If you use this command on the gateway then the gateway will send ICMP message back to the source to adjust the TCP MSS. Your maximum MTU is fixed on the ethernet interface anyway. So only way to reduce the TCP MSS sizs on the source to get the packet through.
Since every TCP segment has more than 20bytes in its header as we are usually exchanging mss and window size in every packet sent across. so 1460 won’t do it… wouldn’t it be better to aim for 1452, that is 1-2 rows of options that would at least cover those two options?
what about this equation:
T(transmission) =[D/MTU] * (H+MTU) / R
Hi Roman,
I am using ADSL2+ connection with PPPOA configuration. Internet connection is working fine but some of the websites not opening.
Spoke to ISP who advise to adjust MTU to 1400
My question is do I have to do both mtu size and TCP adjust to make it work?
Like:
int dialer 1
ip mtu 1400
ip tcp adjust-mss 1360
or just using ip mtu cmd is enough.
Another question Cisco router has ATM0 interface for dialer connectin do I have to make same adjustments to ATM int as well.
Thanks
Randhir
Hi,
I have one mpls network and in network we are facing problem of Mtu. due to mtu my some application not working which is only working through 9000 mtu.
So kindly suggest.
Hi Roman Bhai,
I have some silly question. In some MPLS network, 1532 bytes MTU is used. How to calculate this MTU?
Does this MTU has used Cisco and Mikrotik both devices?
If anyone wants 9012 bytes in an MPLS network. Is it possible to deploy to any network consists of Cisco/Mikrotik devices?
Thanks
Farha Diba
Dear Farha,
I have listed my answer as following, feel free to contact me for further queries.
Q: In some MPLS network, 1532 bytes MTU is used. How to calculate this MTU?
A: On some devices, 1532 is the Maximum MTU of fast Ethernet interface, then some people use this value to be on the safe side.
Q: Does this MTU has used Cisco and Mikrotik both devices?
A: Yes, it is strongly recommended that MPLS MTU is configured to the same value on all routers forming MPLS cloud because of effects MPLS MTU has on MPLS switched packets. But which exact value is used, you need to consider about the services in your network.
Q: If anyone wants 9012 bytes in an MPLS network. Is it possible to deploy to any network consists of Cisco/Mikrotik devices?
A: MTU ranges of different devices and interface types are different, you need to actually check these on your device or check the specifications with vendors.
Please help to advice if correct or not.
Assumption:
1. Ethernet MTU size : 1500 byte (Ethernet header & FCS are not included)
2. MPLS label size: 3 MPLS label LDP + VPN + TE: 12 byte
3. 20 byte TCP + 20 byte IP header
4. TCP MSS size = payload size
5. IP MTU size = payload size + TCP header + IP header
6. MPLS MTU size = payload size + TCP header + IP header + MPLS label size
Therefore:
IP MTU = payload size + TCP header + IP header = Ethernet MTU – MPLS Labels =1500-12 = 1488
TCP MSS = Ethernet MTU – IP header (20) – TCP header (20) – MPLS Labels (12) = 1500-20-20-12=1448
Conclustion:
Shall configure with ip mtu 1488 instead of ip mtu 1448.
Router1(config)#int ethernet 1/0
Router1(config-if)#ip mtu 1488
Router1(config-if)#ip tcp adjust-mss 1448
I can’t believe so many people praised this article. The “ip mtu” and “tcp adjust-mss” CANNOT both be 1448 !!!
One of the first comments pointed this out. However, the article wasn’t changed…
Really it’s a good article to know about IP MTU
This post is very helpful for me . thanks
Why do I have to change the IP MTU to 1488. I get the TCP MSS part of changing it to 1448. Shouldn’t only that command be just needed “ip adjust-mss” as we are talking about routing/forwarding MPLS Lablelled packet?? Unless we are also assuming the router to forwards plain IP packet
Lets say a MPLS Labelled packet with 3 Labels comes in a transiting router. The transit outgoing interface mpls mtu is 1500 (default) based on the interface max MTU setting, So any Layer 3 packet > 1488 will cause fragmentation. I have already signaled with tcp adjust-mss 1460 max segment size to originating router.. Shouldn’t this command be enough. ??
Just rephrasing the typo..
I have already signaled with tcp adjust-mss 1448 max segment size to originating router.. Shouldn’t this command be enough. ??
hi,
i think that MSS is confused with IP MTU ..(in the article)
“..we cannot increase the IP MTU size on the router Ethernet interface because the MPLS label encapsulation frame size can potentially exceed the maximum MTU.”
“What is the best way to solve this? Shrinking the IP MTU..”
We cannot increase the MTU because the frame will exceed the MTU. Therefore, we will reduce the MTU.
I didn’t understand how it works.
Doubt about making MTU adjustment on an L3 interface with the same value for clamp mss, in theory this value of 1448 will still be lost in the 40bytes of the ip + tcp headers.
It is worth thinking about what is mtu L2 and mtu L3 and mtu tcp.
This article may need to be reworded.