Analyzing MPTCP adoption in the Internet

By on 23 Aug 2022

Category: Tech matters

Tags: , , , , ,

2 Comments

Blog home

Multipath TCP (MPTCP) has been in development since 2013 (RFC 6824) and has seen significant interest from researchers and industries alike. The protocol aims to use multiple available network paths between endpoints simultaneously.

Several organizations have publicly announced MPTCP incorporation within their products and services, including:

  • Apple uses MPTCP in its iOS devices (for example, Siri, Music, Maps, Wi-Fi Assist) and also allows third-party developers to use the protocol in non-system applications.
  • Linux uses MPTCPv1 (RFC 8684) in all its machines with kernel v5.6 and upwards. The older development version is now known as MPTCP version 0 (MPTCPv0).

Despite significant interest by these and other organizations, the current state of MPTCP deployment and usage in the Internet remains largely unknown. This blog post summarizes the key findings from our study to provide the most accurate picture of the true MPTCP deployment to date. Part of the study was published in IFIP Networking 2021, and the extended version is currently under submission. If you are interested in learning more about our work, we encourage you to read our detailed analysis and download our ongoing scans at mptcp.io.

Key points:
  • MPTCP deployment spans more than 80 economies.
  • MPTCPv1 support was almost non-existent over both IPv4 and IPv6 until October 2021.
  • Almost half of all IPv4 MPTCPv0 hosts are deployed in Hong Kong, thanks to Hong Kong Broadband.
  • At the end of 2021, Apple ranked second in MPTCPv0 support in IPv4 but had the largest presence in IPv6, and was the only organization that was actively supporting MPTCPv1. In February 2022, it increased its MPTCPv1 over IPv4 support by 10x.

Scanning for MPTCP — a primer

To understand our measurement methodology, it’s essential to understand the MPTCP protocol implementation and operation first.

MPTCP extends regular TCP and uses optional TCP header fields for signalling (see RFC 6824). The most notable option is the MP_CAPABLE, which includes the MP_CAPABLE flag, signalling that the particular host supports MPTCP during connection establishment. Additionally, the option also includes a random 64-bit sequence known as the sender’s key that the hosts use to authenticate themselves.

Figure 1 shows the MPTCPv0 connection establishment procedure between two hosts, Bob and Alice. The procedure mimics TCP’s three-way handshake. Bob initiates the connection by sending a SYN packet containing the MP_CAPABLE flag with MPTCPv0 support and its own sender’s key. If Alice also supports MPTCP, it will reply with a SYN-ACK containing the MP_CAPABLE flag and its own sender’s key. Bob finally establishes the connection by sending an ACK with both keys in the MP_CAPABLE option. If any of the MPTCP options are dropped during the exchange (for example, Alice does not support MPTCP), the connection falls back to a regular TCP.

Infographic showing MPTCPv0 Key exchange at connection establishment.
Figure 1 — MPTCPv0 Key exchange at connection establishment.

MPTCPv1 connection establishment, on the other hand, differs slightly from MPTCPv0. In addition to the different MPTCP version numbers in MP_CAPABLE, Bob does not send a sender’s key in the SYN, which is only sent to Alice in the final SYN-ACK. We will discuss the impact of this change in the handshake mechanism later in this post.

Infographic showing steps for scanning for MPTCP support using ZMap
Figure 2 — Scanning for MPTCP support using ZMap.

We can leverage the MPTCP’s handshake mechanism to determine MPTCP support in the Internet. We used ZMap and generated an MPTCP SYN with the MP_CAPABLE option. For MPTCPv0 scans, we also used a fixed 64-bit sequence as a key and sent it in the SYN. If the targeted host responded with a SYN-ACK that also included the MP_CAPABLE option and a sender’s key, it was likely to support MPTCP.

We probed the entire IPv4 address space over port 80 (≈74M unique responsive IPs) and port 443 (≈52M unique responsive IPs). In IPv6, we used the IPv6 Hitlist to probe port 80 (≈746k responsive IPs) and port 443 (≈544k responsive IPs) due to the size of the address space. We drew our results from MPTCPv0 scans for eighteen months (July 2020 to December 2021) and MPTCPv1 for nine months (April to December 2021). In addition, our scans are still ongoing and our latest results are available at mptcp.io.

Screenshot of mptcp.io scans.
Figure 3 — We plan to continue our scans for MPTCP adoption for the foreseeable future, which you can view and download at mptcp.io.

Internet middleboxes vs MPTCP: Fight for compatibility

Firstly, we found that only a small percentage of probed hosts respond to our scans with an MP_CAPABLE flag in the SYN-ACK (≈5%). Looking at the absolute number of hosts instead of percentages (close to 300,000 in MPTCPv0 and 200,000 in MPTCPv1), it might seem that MPTCP is extensively supported in the Internet. However, we observed in our MPTCPv0 scans that only a fraction of the 300,000 hosts (≈4%) sent us a different sender’s key value in the SYN-ACK than the static sender’s key we sent in the original SYN. This behaviour is contrary to RFC 6824, which says: “… a key MUST be hard to guess, and it MUST be unique for the sending host at any one time.

Bell graph showing hamming weight distribution of sender's key from MPTCPv0 IPv6 hosts in SYN-ACK
Figure 4 — Hamming weight distribution of sender’s key from MPTCPv0 IPv6 hosts in SYN-ACK.

Since the key is a random 64-bit sequence, the sum of independent random variables should tend toward a normal distribution, according to the central limit theorem. Figure 4 shows the hamming weight of the sender’s keys received from potential MPTCP IPv6 hosts on port 443. Note the outlier at hamming weight 16, which does not follow the distribution. We found this outlier present in all our MPTCPv0 scans for both ports 80 and 443 (more prevalent in IPv4 than IPv6). Interestingly, the outlier’s hamming weight exactly matches the weight of the static key we sent in our SYN probe to the hosts.

The distribution highlights the prevalence of middleboxes in the Internet that do not support the TCP header extensions and therefore handle such packets in unconventional ways. Many middleboxes drop packets with TCP header extensions, while others may remove the extensions but forward the packet to the destination. The case above shows MPTCPv0 scans to be affected by middleboxes that replay the header extensions in the SYN probe back to us. We filtered out such middleboxes from our analysis by checking if the returned sender’s key in SYN-ACK differed from that sent in the SYN.

Note that MPTCPv1’s design decision to remove the sender’s key in the SYN was to specifically handle such replaying middleboxes, as the returned SYN-ACK is expected to have a sender’s key value along with the sent MP_CAPABLE options. We filtered out middleboxes in our MPTCPv1 analysis using this exact criterion.

After filtering, we found the number of responsive hosts dropped considerably — almost 100,000 IPv4 and 2,000 IPv6 over MPTCPv0, and 120 IPv4 and 80 IPv6 over MPTCPv1. We also found that MPTCPv1 support was almost non-existent over both IPv4 and IPv6 until October 2021, after which we observed a sudden spike in responsive hosts.

MPTCP support in IPv4 and IPv6

We further inspected for the presence of interfering middleboxes by running Tracebox towards all targets that sent an MP_CAPABLE option in our ZMap scans. In the Tracebox reply, we received responses from intermediate routers on the path, including any modifications made. For the detailed methodology, we encourage you to read our paper.

Firstly, we observed that a large percentage of targets did not respond to our Tracebox queries and were unreachable. In IPv4, almost 90% and 48% of hosts on ports 443 and 80 were unreachable. In IPv6, only port 443 had unreachable hosts (≈82%). Further analysis revealed target Internet Service Provider (ISP) blocking and transient hosts to be among the root causes.

Graph showing MPTCP (v0 and v1) support for HTTP and HTTPS in IPv4 across port 80 and 443.
Figure 5 — MPTCP (v0 and v1) support for HTTP and HTTPS in IPv4.

After removing the unreachable hosts, we found the true support of MPTCPv0 in IPv4 at the end of December 2021 to be ≈16,500 over port 80 and ≈13,500 over port 443. In the case of IPv6, 1,195 and 1,184 hosts over ports 80 and 443 truly support MPTCPv0.

MPTCPv1 support is significantly lower than MPTCPv0 as 100-150 hosts supported the newer version on IPv4 and IPv6 over either port 80 or 443. Figure 5 shows the port overlap of the true MPTCP (both v0 and v1) hosts — highlighting significant simultaneous support for both HTTP and HTTPS.

MPTCP usage in organizations

We also found that almost half of all IPv4 MPTCPv0 hosts are deployed in Hong Kong, thanks to Hong Kong (HK) Broadband (Table 1), which has the largest share. The majority of the hosts within HK Broadband became operational in 2021, before which Apple had the highest MPTCPv0 support.

Autonomous
System
Number (ASN)
#Port80#Port443RankEconomyOwner
AS926964616370364HKHK Broadband
AS61851347134413,577USApple
AS611576745341,368DEPlus Server
AS122152945676AUTelstra
AS186183903903,915USWest Central
AS189433533523,533USYelcot
AS792241920927USComcast
AS119762322312,360USFidelity
AS202870404216,712ITDimensione
AS1512936914,034USGeneseo Tele.
Table 1 — Top-10 ASes hosting IPv4 MPTCPv0 hosts.

At the end of 2021, Apple ranked second in MPTCPv0 support in IPv4 but had the largest presence in IPv6 (Table 2), with most of its servers located in the USA.

ASN#Port80#Port443RankEconomyOwner
AS61851163116314,234USApple
AS63949437,042USLinode
AS4811332,034CNChina Telecom
AS714226,949USApple
AS2011552226,184CHEmbeDD
Table 2 — Top-5 ASes hosting IPv6 MPTCPv0 hosts.

We also observe that Apple is the only organization on the Internet that has actively started supporting the newer MPTCPv1 (Table 3).

ASN#Port80#Port443RankEconomyOwner
AS6185989814,234USApple
AS3969862216,570USBytedance
AS2062930322,645DEProIO GmbH
AS714126,949USApple
AS320902221DEVodafone
Table 3 — Top-5 ASes hosting IPv4 MPTCPv1 hosts.

Figure 6 shows the number of IPv4 hosts in Apple ASes (AS6185 and AS714) that reportedly support MPTCPv0 and MPTCPv1 since April 2021. Apple’s MPTCPv0 hosts remain fairly consistent — likely supporting their core applications and Apple services. We found that Apple had no support for MPTCP in IPv6 until September 2021 and MPTCPv1 in IPv4 until October 2021, which is also the cause for the sudden rise in MPTCPv1 hosts globally.

Graph showing Apple-hosted IPv4 hosts on different ports.
Figure 6 — MPTCP (v0 and v1) IPv4 hosts within Apple ASes.

In February 2022, Apple increased its MPTCPv1 over IPv4 support by 10x, almost equalling the MPTCPv0 hosts. Further investigation revealed that these are the same hosts that previously supported MPTCPv0 but now also support MPTCPv1.

Additionally, many network operators and ISPs across the globe are using MPTCP to enhance their operations, for example, Korea Telecom and Yelcot Telecom. Overall, we found that the current MPTCP deployment spans more than 80 economies across the globe.

Are you using MPTCP? If so, let us know

We plan to continue our scans for MPTCP adoption for the foreseeable future, which you can view and download at mptcp.io. Please read our paper to learn more about the impact of middleboxes on MPTCP data transfers and MPTCP IPv4 and IPv6 traffic share in the Internet.

We are also very interested in learning about applications that use MPTCP over the public Internet. If you are aware of such use cases, manage servers that are involved in such exchanges, or would like to know more, please get in touch with us at info@mptcp.io.

Tanya Shreedhar is a postdoctoral researcher at the University of Edinburgh.

Rate this article

The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.

2 Comments

  1. Leszek

    Nice write-up Tanya. I would like to ask about your thoughts regarding Multipath QUIC vs MPTCP – is there a place to co-exist for both of them or do you see one of them as the most probable winner of worldwide adoption ?

    Reply
  2. Monika Prakash

    “I would like to ask about your thoughts regarding Multipath QUIC vs MPTCP – is there a place to co-exist for both of them or do you see one of them as the most probable winner of worldwide adoption ?”

    I am interested in this question and looking forward to the answer 🙂

    Reply

Leave a Reply

Your email address will not be published.

Top