Analyzing MPTCP adoption in the Internet

Multipath TCP (MPTCP) has been in development since 2013 (RFC 6824) and has seen significant interest from researchers and industries alike. The protocol aims to use multiple available network paths between endpoints simultaneously.

Several organizations have publicly announced MPTCP incorporation within their products and services, including:

• Apple uses MPTCP in its iOS devices (for example, Siri, Music, Maps, Wi-Fi Assist) and also allows third-party developers to use the protocol in non-system applications.
• Linux uses MPTCPv1 (RFC 8684) in all its machines with kernel v5.6 and upwards. The older development version is now known as MPTCP version 0 (MPTCPv0).

Despite significant interest by these and other organizations, the current state of MPTCP deployment and usage in the Internet remains largely unknown. This blog post summarizes the key findings from our study to provide the most accurate picture of the true MPTCP deployment to date. Part of the study was published in IFIP Networking 2021, and the extended version is currently under submission. If you are interested in learning more about our work, we encourage you to read our detailed analysis and download our ongoing scans at mptcp.io.

Scanning for MPTCP — a primer

To understand our measurement methodology, it’s essential to understand the MPTCP protocol implementation and operation first.

MPTCP extends regular TCP and uses optional TCP header fields for signalling (see RFC 6824). The most notable option is the MP_CAPABLE, which includes the MP_CAPABLE flag, signalling that the particular host supports MPTCP during connection establishment. Additionally, the option also includes a random 64-bit sequence known as the sender’s key that the hosts use to authenticate themselves.

Figure 1 shows the MPTCPv0 connection establishment procedure between two hosts, Bob and Alice. The procedure mimics TCP’s three-way handshake. Bob initiates the connection by sending a SYN packet containing the MP_CAPABLE flag with MPTCPv0 support and its own sender’s key. If Alice also supports MPTCP, it will reply with a SYN-ACK containing the MP_CAPABLE flag and its own sender’s key. Bob finally establishes the connection by sending an ACK with both keys in the MP_CAPABLE option. If any of the MPTCP options are dropped during the exchange (for example, Alice does not support MPTCP), the connection falls back to a regular TCP.

MPTCPv1 connection establishment, on the other hand, differs slightly from MPTCPv0. In addition to the different MPTCP version numbers in MP_CAPABLE, Bob does not send a sender’s key in the SYN, which is only sent to Alice in the final SYN-ACK. We will discuss the impact of this change in the handshake mechanism later in this post.

We can leverage the MPTCP’s handshake mechanism to determine MPTCP support in the Internet. We used ZMap and generated an MPTCP SYN with the MP_CAPABLE option. For MPTCPv0 scans, we also used a fixed 64-bit sequence as a key and sent it in the SYN. If the targeted host responded with a SYN-ACK that also included the MP_CAPABLE option and a sender’s key, it was likely to support MPTCP.

We probed the entire IPv4 address space over port 80 (≈74M unique responsive IPs) and port 443 (≈52M unique responsive IPs). In IPv6, we used the IPv6 Hitlist to probe port 80 (≈746k responsive IPs) and port 443 (≈544k responsive IPs) due to the size of the address space. We drew our results from MPTCPv0 scans for eighteen months (July 2020 to December 2021) and MPTCPv1 for nine months (April to December 2021). In addition, our scans are still ongoing and our latest results are available at mptcp.io.

Internet middleboxes vs MPTCP: Fight for compatibility

Firstly, we found that only a small percentage of probed hosts respond to our scans with an MP_CAPABLE flag in the SYN-ACK (≈5%). Looking at the absolute number of hosts instead of percentages (close to 300,000 in MPTCPv0 and 200,000 in MPTCPv1), it might seem that MPTCP is extensively supported in the Internet. However, we observed in our MPTCPv0 scans that only a fraction of the 300,000 hosts (≈4%) sent us a different sender’s key value in the SYN-ACK than the static sender’s key we sent in the original SYN. This behaviour is contrary to RFC 6824, which says: “… a key MUST be hard to guess, and it MUST be unique for the sending host at any one time.“

Since the key is a random 64-bit sequence, the sum of independent random variables should tend toward a normal distribution, according to the central limit theorem. Figure 4 shows the hamming weight of the sender’s keys received from potential MPTCP IPv6 hosts on port 443. Note the outlier at hamming weight 16, which does not follow the distribution. We found this outlier present in all our MPTCPv0 scans for both ports 80 and 443 (more prevalent in IPv4 than IPv6). Interestingly, the outlier’s hamming weight exactly matches the weight of the static key we sent in our SYN probe to the hosts.

The distribution highlights the prevalence of middleboxes in the Internet that do not support the TCP header extensions and therefore handle such packets in unconventional ways. Many middleboxes drop packets with TCP header extensions, while others may remove the extensions but forward the packet to the destination. The case above shows MPTCPv0 scans to be affected by middleboxes that replay the header extensions in the SYN probe back to us. We filtered out such middleboxes from our analysis by checking if the returned sender’s key in SYN-ACK differed from that sent in the SYN.

Note that MPTCPv1’s design decision to remove the sender’s key in the SYN was to specifically handle such replaying middleboxes, as the returned SYN-ACK is expected to have a sender’s key value along with the sent MP_CAPABLE options. We filtered out middleboxes in our MPTCPv1 analysis using this exact criterion.

After filtering, we found the number of responsive hosts dropped considerably — almost 100,000 IPv4 and 2,000 IPv6 over MPTCPv0, and 120 IPv4 and 80 IPv6 over MPTCPv1. We also found that MPTCPv1 support was almost non-existent over both IPv4 and IPv6 until October 2021, after which we observed a sudden spike in responsive hosts.

MPTCP support in IPv4 and IPv6

We further inspected for the presence of interfering middleboxes by running Tracebox towards all targets that sent an MP_CAPABLE option in our ZMap scans. In the Tracebox reply, we received responses from intermediate routers on the path, including any modifications made. For the detailed methodology, we encourage you to read our paper.

Firstly, we observed that a large percentage of targets did not respond to our Tracebox queries and were unreachable. In IPv4, almost 90% and 48% of hosts on ports 443 and 80 were unreachable. In IPv6, only port 443 had unreachable hosts (≈82%). Further analysis revealed target Internet Service Provider (ISP) blocking and transient hosts to be among the root causes.

After removing the unreachable hosts, we found the true support of MPTCPv0 in IPv4 at the end of December 2021 to be ≈16,500 over port 80 and ≈13,500 over port 443. In the case of IPv6, 1,195 and 1,184 hosts over ports 80 and 443 truly support MPTCPv0.

MPTCPv1 support is significantly lower than MPTCPv0 as 100-150 hosts supported the newer version on IPv4 and IPv6 over either port 80 or 443. Figure 5 shows the port overlap of the true MPTCP (both v0 and v1) hosts — highlighting significant simultaneous support for both HTTP and HTTPS.

MPTCP usage in organizations

We also found that almost half of all IPv4 MPTCPv0 hosts are deployed in Hong Kong, thanks to Hong Kong (HK) Broadband (Table 1), which has the largest share. The majority of the hosts within HK Broadband became operational in 2021, before which Apple had the highest MPTCPv0 support.

Autonomous System Number (ASN)	#Port80	#Port443	Rank	Economy	Owner
AS9269	6461	6370	364	HK	HK Broadband
AS6185	1347	1344	13,577	US	Apple
AS61157	674	534	1,368	DE	Plus Server
AS1221	529	456	76	AU	Telstra
AS18618	390	390	3,915	US	West Central
AS18943	353	352	3,533	US	Yelcot
AS7922	419	209	27	US	Comcast
AS11976	232	231	2,360	US	Fidelity
AS202870	404	2	16,712	IT	Dimensione
AS15129	369	1	4,034	US	Geneseo Tele.

At the end of 2021, Apple ranked second in MPTCPv0 support in IPv4 but had the largest presence in IPv6 (Table 2), with most of its servers located in the USA.

ASN	#Port80	#Port443	Rank	Economy	Owner
AS6185	1163	1163	14,234	US	Apple
AS63949	4	3	7,042	US	Linode
AS4811	3	3	2,034	CN	China Telecom
AS714	2	2	6,949	US	Apple
AS201155	2	2	26,184	CH	EmbeDD

We also observe that Apple is the only organization on the Internet that has actively started supporting the newer MPTCPv1 (Table 3).

ASN	#Port80	#Port443	Rank	Economy	Owner
AS6185	98	98	14,234	US	Apple
AS396986	2	2	16,570	US	Bytedance
AS206293	0	3	22,645	DE	ProIO GmbH
AS714	1	2	6,949	US	Apple
AS3209	0	2	221	DE	Vodafone

Figure 6 shows the number of IPv4 hosts in Apple ASes (AS6185 and AS714) that reportedly support MPTCPv0 and MPTCPv1 since April 2021. Apple’s MPTCPv0 hosts remain fairly consistent — likely supporting their core applications and Apple services. We found that Apple had no support for MPTCP in IPv6 until September 2021 and MPTCPv1 in IPv4 until October 2021, which is also the cause for the sudden rise in MPTCPv1 hosts globally.

In February 2022, Apple increased its MPTCPv1 over IPv4 support by 10x, almost equalling the MPTCPv0 hosts. Further investigation revealed that these are the same hosts that previously supported MPTCPv0 but now also support MPTCPv1.

Additionally, many network operators and ISPs across the globe are using MPTCP to enhance their operations, for example, Korea Telecom and Yelcot Telecom. Overall, we found that the current MPTCP deployment spans more than 80 economies across the globe.

Are you using MPTCP? If so, let us know

We plan to continue our scans for MPTCP adoption for the foreseeable future, which you can view and download at mptcp.io. Please read our paper to learn more about the impact of middleboxes on MPTCP data transfers and MPTCP IPv4 and IPv6 traffic share in the Internet.

We are also very interested in learning about applications that use MPTCP over the public Internet. If you are aware of such use cases, manage servers that are involved in such exchanges, or would like to know more, please get in touch with us at info@mptcp.io.

Tanya Shreedhar is a postdoctoral researcher at the University of Edinburgh.

Rate this article

---

The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.