Once you head away from the areas that are serviced by modern terrestrial cable infrastructure, the available digital communication options are somewhat limited.
Some remote areas are served by high-frequency radio systems, using radio signals that bounce off the ionosphere to provide a long-distance but limited bandwidth service. Or there are satellite services based on spacecraft positioned in geostationary orbital slots, also referred to as a Geosynchronous Equatorial Orbit (GEO) satellites.
These GEO satellite systems can offer far greater bandwidth than high-frequency radio but have significantly higher latency — the satellite is orbiting at 35,786km above the equator to synchronize the orbital period with the earth’s revolution. It takes a minimum of 239ms of the signal propagation time (assuming that the earth end is directly under the spacecraft on the equator) for a signal to travel to the spacecraft and back.
In the context of Internet communications, a satellite hop will add at least 477ms to a round trip time (RTT) measurement, as a signal will need to head up to the satellite and back twice for the complete round trip. When the earth endpoints move away from directly under the spacecraft, then these paths will be longer. At the horizon of satellite visibility, the distance to the spacecraft is 42,640km away, and the signal propagation time is 284ms or 568ms as an RTT value (Figure 1).
When you add switching delays and terrestrial components and use a land point in the mid-latitudes, a circuit that includes a satellite hop will generally see RTT measurements of around 600ms. Network queues will add to this delay, of course.
Into this mix, we are now seeing the introduction of Low Earth Orbit (LEO) services, which can offer a potent combination of higher bandwidth and lower latency. LEO satellites start at altitudes as low as 160km and as high as 2,000km. The objective is to keep the satellite’s orbit high enough to stop it from slowing down by ‘grazing’ the denser parts of the earth’s ionosphere, but not so high that it loses the radiation protection afforded by the Inner Van Allen belt. At a height of 550 km, the minimum signal propagation delay to reach the satellite and back is 3.7ms.
Read: Everything you wanted to know about LEO satellites, part 1: The basics
Several LEO systems are being announced, launched and bought into service. As well as SpaceX’s Starlink service, there is also OneWeb, with some 428 satellites in its constellation and a further 220 or so satellites planned for launch. OneWeb uses a higher orbital plane, at an altitude of 1,200km, and the satellites use a polar orbit, compared to Starlink’s inclined orbits. Amazon’s FCC filings for its proposed Kuiper LEO constellation describe a planned constellation of up to 3,300 satellites. Boeing has obtained regulatory permission to launch some 150 LEO broadband spacecraft. There is also a Chinese project underway.
Communicating with LEO satellites is quite different to GEO satellites. LEO satellites move quickly across the sky but are relatively close to the earth, so there is no need to use a large, heavy steerable dish antenna. For LEO systems, it is possible to use a phased array panel to direct the focus of the antenna to each satellite as it passes overhead.
The LEO system we will be testing here is Starlink. As of March 2022, 2,335 Starlink spacecraft have been launched, with 2,110 still in orbit. Of those, 1,608 are in the licensed operational shells and 447 are undergoing orbit raising towards those shells.
The Starlink satellites currently communicate to ground stations and not to each other. This limits the extent of areas served by Starlink, as each active satellite has to be in the range of an earth station to provide service.
While these satellites orbit at an altitude of 550km, they appear over the horizon at a distance of 2,705km. The satellites have velocities of around 7.6km/sec and have an orbital period of around 95 minutes, so each satellite is visible from horizon to horizon for 12 minutes. This corresponds to a relative change of 1 degree of elevation every 15 seconds. In practice, the range of useful visibility of these satellites is far more restricted than this.
Currently, these satellites are ‘mirrors in the sky’ and need a line of sight to both the end user and an earth station to provide a service. Starlink is planning to use inter-satellite communications with laser links, which will extend its potential coverage to most parts of the earth. This would also improve the end-to-end signal propagation speed by up to 50% by replacing long-distance fibre optic hops with inter-satellite laser signals.
Read: Starlink’s high-speed satellite Internet: What’s the catch?
In Australia, there is a large amount of rural and remote areas that are not well served by wired services. These days there are two connection options for rural and remote areas. The first is a system (Australian National Broadband Network’s Sky Muster service) that is based on two GEO satellites and ten ground stations, offering an Internet access service that is rate managed per user to a maximum of 25Mbps per user. The second is the Starlink service, which is available in certain parts of Australia and can offer service speeds up to 90Mbps download and 14Mbps upload.
We’ve been interested in comparing the achieved protocol performance of these two satellite services for some time, and in early 2022 we had the opportunity to place a monitoring unit at a remote site that is connected to both Starlink and Sky Muster services. This is a report of what we found when we tested both services with several different TCP protocols.
TCP congestion control protocols
There are three TCP flow control protocols in widespread use today. These are sender-side algorithms that control the amount of data being passed into the network based on the network state that is inferred by the incoming ACK stream.
The ideal steady state for all these TCP flow control algorithms is to pass new segments of data into the network at the same rate as the receiver removes segments from the network. If the protocol can discover the available capacity of the network path, then the algorithm will operate efficiently across the network.
However, the path capacity of the network is unknown and subject to change because of the interaction with other data flows. The way these protocols adapt to this situation is to increase the sending rate to probe the network with increased sending rates and react by decreasing the sending rate when it receives an indication that its sending rate is greater than the patch capacity. The protocols achieve this is by changing the size of the sender’s sending window, or congestion window (cwnd).
Reno is the currently used version of the conventional Additive Increase Multiplicative Decrease (AIMD) TCP flow control algorithm. This protocol increases cwnd by one segment every RTT until it encounters packet loss. Upon receipt of a loss signal (a duplicate ACK), it halves the cwnd value and then resumes its linear inflation of cwnd. This algorithm will slowly fill the network queues with data, assuming the network operator has dimensioned these queues at a size that is the delay x bandwidth product of the link that is driven by the queue. When the queues are full and congestion drop happens, the network contains twice the delay x bandwidth product of the end-to-end path. By rate-halving at this point Reno allows the network queues to drain, resuming its linear increase of cwnd. This behaviour of cwnd is shown in Figure 4, in the red trace.
Cubic has been adopted by many Linux systems as the congestion control protocol of choice. Cubic attempts to hold the cwnd value at a level that is sustainable by the network for longer intervals. When it probes upward into higher sending rates it does so cautiously at first then increases rapidly if no loss signal is received. It achieves this by using a cwnd inflation function based on a cubic polynomial function. Cubic reduces the cwnd value by 1/3 when it sees packet drop. This behaviour is shown in Figure 4, in the blue trace.
BBR uses a different approach to flow control. Once it has an estimate of the delay x bandwidth product of the network path it maintains the sending rate at this setting for 6 x RTT intervals. On the seventh interval, it increases the sending rate by 25% and on the next interval, it drops the sending rate by the same amount. If BBR’s network capacity estimate is too low, then the pulse of increased traffic will be handled by the work without the formation of queues, so the RTT of these packets will remain steady.
In this case, BBR will increase its estimate of the delay x bandwidth product of the network path. If the pulse of increased traffic causes a rise in the RTT of these packets then this is a signal that network queues are forming, and the sending rate should not be altered. The BBR management of cwnd is shown in Figure 4, in the green trace.
Read: The game theory behind running CUBIC and BBR on the Internet
However, these days several additional factors also govern TCP performance.
- TCP Selective Acknowledgement (SACK) allows TCP sessions to recover from lost packets without closing down the TCP windows.
- TCP Timestamp options allow the TCP systems to have a more accurate view of the routing trip time between the end systems.
- Explicit Congestion Notification (ECN) allows the TCP session to be informed of network congestion before the onset of queue drop.
The internal queuing disciplines have become more intricate. These local optimizations allow the TCP session to maintain a more accurate view of the capacity of the network path, and, more importantly, allow a TCP sender to maintain a sending rate that does not necessarily push the network into regular queue buildup and congestion loss. In other words, while the internal cwnd TCP control variable behaves as per the flow control model, the packet sending behaviour on modern Linux systems can perform with a far better level of adaptation to the available path capacity.
What we are primarily interested in here is the ability of these modern TCP implementations to efficiently use the network across network paths that use GEO and LEO satellite components.
The control: Terrestrial fibre
Firstly, as a control point, let’s look at the other end of the spectrum of retail consumer connectivity options in Australia for the satellite measurements. The service we will use as a reference point is a fibre-based service that delivers 275Mbps/25Mbps.
This measurement uses an on-site Intel NUC device running Debian 10 as the client, and the server is a Dell unit, again running Debian 10. The server is located in Brisbane, Australia and the on-site fibre-tail connected device is in Canberra, Australia, roughly 1,000km apart.
For the fibre test, the path between the server and the on-site device has a minimum RTT of 19.5ms with a mean of 1.1ms of additional network queuing delay in IPv4. When using IPv6 the Ping results show a minimum of 19.622ms RTT, with a mean of 0.9ms of additional network queuing delay (Figure 5).
For this measurement, we will be running TCP sessions that download a data stream from the server to the client. The measurement will start with a single stream (Stream 1), then 20 seconds later a second stream (Stream 2) is added. 100 seconds later the initial stream is terminated, and the second stream continues for a further 20 seconds.
Figure 6 shows the results of this test on the downstream fibre link, which is provisioned at a sending rate of 275Mbps, using the Reno TCP flow control protocol.
Reno can find the path capacity very quickly and has stabilized the link capacity within 0.4 of a second, or 40 RTT intervals. When the second Reno stream starts, the initial division of the path capacity is at a ratio of 3:1 between the two streams. It takes a further 80 seconds, or some 400 RTT intervals for the two sessions to stabilize on a roughly equal division of the available path capacity.
When the first session stops, the second session quickly fills the available path capacity. This is not a conventional Reno inflation of the sending rate by 1 MSS per RTT. The rapid change in the sending rate in Stream 2 points to the use of a Linux kernel structure that holds the current estimated path capacity per destination address — when one stream finishes, then the other stream is rapidly inflated to occupy the entire path capacity.
Figure 7 shows the same two-stream measurement using the Cubic flow control protocol.
The Cubic protocol behaves in several different modes. The initial start of Stream 1 shows a high level of variation in the sending rate, varying the sending rate by 75Mbps over successive 100ms intervals (5 RTT intervals). The last 20 seconds of Stream 2 show a far more stable sending rate. This is quite similar to the behaviour of Reno. However, unlike Reno, the two Cubic streams cannot sustain an equal sharing of the link capacity over an extended period.
Figure 8 shows that BBR is a more stable protocol in this scenario. Once BBR establishes the link capacity it maintains it with a very small high-frequency variation. When the second stream opens, it takes 2.5 seconds (approximately 125 RTT intervals) for the two streams to equalize and maintain this state.
GEO satellite service
Now let’s look at the other extreme in terms of end-to-end delay, namely a GEO satellite service.
As already noted, GEO satellites sit at an altitude of around 35,786km, so the minimum signal propagation distance for a round trip using this satellite service is 143,144km or some 477ms. The measured RTT is somewhat higher when using a server located in Brisbane, Australia and as shown in Figure 9 the measured RTT is some 660ms.
The instability of the ping measurements would tend to indicate the build-up of internal queues within the terrestrial feeder network. This is an IPv4 only service, so there is only a single ping trace. This was a small ping test (500 samples), but there was no packet loss.
These are not ideal conditions for efficient TCP service. The long RTT means that the TCP sender takes up to half a second to respond to changes in the characteristics of the forwarding path, and the high levels of jitter mean that the sender’s smoothed RTT estimate will also be off by up to 100ms.
Let’s move on to a downlink test of a 45 Mbps service provided through a GEO satellite. As with fibre, we will use a two-stream test, starting one stream (Stream 1) 40 seconds earlier than the second (Stream 2). The first of the traces of stream throughput is using the Reno flow control protocol.
It appears that this particular satellite service uses large internal queues. These two streams can share the available bandwidth for 60 seconds in a relatively stable manner. However, when this situation was disrupted at the 105-second point (possibly due to cross-pressure from some external sessions) both Reno sessions became unstable. The result was that total link efficiency dropped by more than 50%. When the first stream stopped at the 240-second point the second stream was still unable to reestablish a stable flow profile. The combination of high latency, large queues and capacity contention across multiple users creates extended periods of high instability in this service, and this flow control algorithm cannot reestablish stability.
Can the Cubic algorithm fare any better under these conditions? Figure 11 shows the same experiment using Cubic.
There are the similar patterns of disruption in service, and the same unstable response from the Cubic streams as was seen in Reno in Figure 10. However, Cubic appears to extract a higher level of efficiency of use of the service when the streams are unstable.
Again, BBR appears to be far more capable of pushing back into the network to sustain a stable flow rate (Figure 12). This algorithm attempts to maintain a constant sending rate into the network for sequences of six successive RTT intervals, which is reflected in the imposition of a stable sending rate most of the time. The two streams rapidly equilibrate and share the available path capacity, and they combine to sustain a highly efficient use of the circuit.
Any path with such a high RTT is a challenge to use efficiently and if you add jitter, large buffers, and the periodic build-up of standing queues, then the outcome is quite hostile to efficient TCP flow control algorithms. In this limited experiment, it appears that BBR produces a better outcome under these quite challenging conditions.
LEO satellite service
Let’s now look at the LEO satellite service offered by Starlink. A ping test with 2,000 samples shows an interesting result:
2000 packets transmitted, 1991 received, 0.45% packet loss, time 2009903ms rtt min/avg/max/mdev = 37.284/60.560/214.301/13.549 ms
Firstly, there is some packet loss present, with nine ping packets lost out of the 2,000 that were sent. It is unclear if this is due to rain fade or satellite tracking handover or a combination of the two.
Secondly, there is a high rate of variation of the RTT measured by this test. The minimum RTT was 37ms, while the average is 60.5ms and the mean standard variation is 13.5ms. The plot of the ping delays is shown in Figure 13.
The transmission delays for this service include some 12ms of the terrestrial network, and the remainder is associated with the satellite link. The distance between the satellite and a visible earth station varies between 550km when the satellite is directly overhead and 2,704km when the satellite is on the horizon. This correlates to a one-way signal propagation time of between 1.8ms and 9ms, or a potential variation in RTT times of between 7.3 and 36ms. There are also encryption delays and presumably codec delays, so the overall delivered RTT performance was observed to be an average of 60ms for this service with a variation of +/- 13ms, which matches the geometry of the LEO constellation. It is not clear whether the isolated RTT measurements of more than 100ms are due to crosstalk or satellite tracking handover.
Let’s now turn to the download performance of this Starlink service.
The constantly changing RTT of this service appears to have thrown the Linux settings that allow the sender to latch onto a path capacity to a destination address and maintain it. The signature Reno linear inflation of the sending rate and the periodic rate halving is now clearly visible. The increased packet loss probability of this Starlink service limits the ability of Reno to open its cwnd, and the overall path capacity is not sustainable over 100Mbps (Figure 14).
Cubic fares no better here. The classic Cubic pattern of window inflation and rate halving can be seen from the 20 second mark to the 55 second mark. The constantly varying RTT appears to prevent the Linux drivers from ‘latching’ onto a sustainable path capacity rate, and the sporadic packet loss events prevent the Cubic streams from opening up the sending window.
It takes approximately 20 seconds for the two streams to reach the notional path capacity of 90Mbps, and the incidence of packet loss causes the collapse of the sending window, which is not subsequently recovered.
BBR again appears to be far more successful here. A single BBR session appears to be able to drive the network path at a capacity of between 150Mbps – 200Mbps. When there are two active BBR sessions, then BBR finds it challenging to stabilize at any particular sending rate, and it is constantly making adjustments. While the sending rate is not stable, BBR achieves considerably greater overall throughput than either Reno or Cubic under these conditions.
This has been a surprising investigation
The Linux implementation of TCP has some unanticipated behaviours. There appears to be an overlay element in the kernel that assumes control of the TCP session if it believes that the path capacity to the destination and its RTT are stable. The Linux kernel’s route cache has undergone some changes over the various Linux versions, but the system still behaves in a way that attempts to associate destinations with a cached path capacity value and ‘latch’ the TCP sending rate to this capacity value.
Read: Satellite constellations are coming. Are we prepared on Earth to receive all the data?
In both the fibre and GEO satellite service profiles we cannot directly observe the ‘sawtooth’ periodic sending behaviour of AIMD control algorithms, nor the polynomial curves associated with Cubic. In its place, we see the sender rapidly lock onto the path capacity and maintain this constant sending rate. This is an extremely effective outcome for the user, in that the TCP sender can rapidly drive a network path at capacity and sustain this sending rate over extended periods.
It’s also evident that there are network paths whose characteristics disable this managed performance TCP behaviour, as this steady-state sending behaviour is not evident in the Starlink test, where we see the result of conventional TCP control algorithms controlling the sending rate. It is unclear to me what characteristics and thresholds control this TCP override. Up-to-date documentation of this aspect of Linux TCP behaviour is not readily available.
This host TCP behaviour is further compounded by the behaviour of modern network interfaces that are performing offloading of TCP segmentation and checksums, which is most relevant for high-speed paths, where a significant proportion of the per-packet processing loads are effectively shunted over to the network interface processor.
Where there is a stable network path, then Linux is extremely good at driving the network to the point of maximal efficiency. For low-delay, high-capacity paths such as terrestrial fibre it does not make much difference whether Reno, Cubic or BBR is used as the TCP flow control protocol. Linux will operate the network path at maximal efficiency.
The BBR protocol operates surprisingly well over GEO satellite services. These high latency services present many challenges for feedback-controlled flow systems because of the significant lag in providing feedback to the server. But BBR appears to manage this situation far more efficiently than the other two loss-based control algorithms. However, none of these TCP protocols can compensate for the high end-to-end path latency, so these GEO satellite services are ineffective in providing highly responsive services. The lag between command and response cannot be avoided. If the application is based on high volume data transfer, then a combination of a Linux host and the BBR TCP algorithm can drive these services at maximal efficiency.
The Starlink LEO service has some unexpected characteristics.
First, the packet loss rate is higher than expected. This may be a combination of using phased array antennae that are tracking satellites moving through the sky at a relative speed of 1 degree of elevation every 15 seconds, together with the need to perform satellite handover at regular intervals. The client dish does not have a very high power transmitter, so there is also the issue of rain fade that will degrade the service during showers.
The RTT variance in this service is also higher than expected. However, even with a 60ms RTT, the responsiveness of this service is at a level that any GEO satellite service simply cannot emulate.
When used with Linux endpoints, the choice of TCP flow control protocol appears to be critical for a Starlink service. Both Reno and Cubic were observed to collapse, while BBR was able to extract highly efficient performance over this Starlink service.
TCP is at its most efficient over low-latency, highly stable infrastructure, so these satellite services are poor substitutes for terrestrial services. If responsiveness is important, which is often the case for voice and video applications, then Starlink is the logical choice, due to an RTT latency that is a fraction of the RTT of any GEO satellite service. However, it appears that the choice of which TCP control protocol to use is a critical factor here.
For closed environments where there is control of both the sender and receiver, it is possible to tune TCP to make optimal use of these satellite services. In these experiments, using BBR appears to be a clear preference.
However, in the public Internet, there is no such tight control over every content source, and a server has no prior knowledge of the characteristics of the path to reach the client. For many years the conventional wisdom has been to rely on ample provisioning of internal queues within the network in an attempt to minimize packet loss events and generalize the network behaviour to adequately respond to most forms of traffic management over diverse paths.
Recent research is now pointing in a different direction, where the combination of small network queues and delay-sensitive end-to-end control algorithms can provide extremely efficient service outcomes. However, mapping this requirement for highly stable underlying path propagation delays into the LEO environment is going to pose some challenges for the LEO folk.
Even so, in networking, the delay is almost everything, and many other issues become far more tractable when the underlying end-to-end latency is short. For that reason, LEO systems will continue to be a very attractive solution to many rural and remote connectivity requirements.
Acknowledgements
This measurement has proved to be far more challenging than was envisaged. Setting up a Linux server in an unattended remote location is always going to be a challenge and I am indebted to APNIC’s Paul Wilson and George Michaelson for setting up and installing the measurement system to perform these tests.
While setting up runs of iperf3 and performing a packet capture of the actual file transfers was relatively straightforward, the task of relating the observed TCP behaviour to the documented Linux behaviours has been very frustrating at times. Thanks to Joao Damas of APNIC for assisting me in working through what folk optimistically label as ‘Linux documentation’ and discovering the tcp_probe kernel diagnostic!
The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.
Thanks for doing this analysis, very interesting!
Nice work, geoff! I’d been making available my flent tests of starlinks’ latency under load here:
https://docs.google.com/document/d/1puRjUVxJ6cCv-rgQ_zn-jWZU9ae0jZbFATLf4PQKblM/edit
Give flent a shot!
It looks like you ran into TSQ.