The early morning of 8 July 2026 was not a good day for Telstra, a major network service provider in Australia that offers data, mobile, and fixed-line communications services. Problems in the internal infrastructure platform can quickly escalate into cascading failures, leading to a large-scale outage that impacts many services at once.
Such an outage can have major ramifications. In this case, it impacted various urban and regional transport services, and in a number of areas, the train services were halted. There were point-of-sale terminal failures, as well as outages on Telstra’s mobile platform. In a news report on the day, the failure was attributed to ‘a synchronization failure across data centres’. It’s entirely plausible in a service provider platform hosted in many locations that an internal synchronization failure could crash parts of the service platform.
Today’s Internet runs on a vast infrastructure of replicated data, and the difference between order and chaos lies in maintaining consistency through synchronization across the many distributed points where that data is published. For example, the Border Gateway Protocol (BGP) ensures that active routing devices share a consistent view of routing information. This allows each router to make independent forwarding decisions that move packets closer to their destinations, while synchronization between BGP speakers keeps routing information up to date and preserves an accurate view of the network topology.
The Domain Name System (DNS) infrastructure presents a similar set of challenges of synchronization across a highly distributed data environment. What tools can we use to synchronize data across multiple instances? There are many ways of performing this function, and here I’d like to look at several approaches, their strengths and weaknesses and their applicability.
Brute force!
One of the basic approaches to data synchronization is for a secondary server to retrieve a complete copy of the data from a primary source and overwrite its local copy with this data. Each time it needs to refresh its local copy of the data, it performs a complete pull of the entire data set. It’s a very simple approach and is used in many contexts of data distribution and replication. It only requires an understanding of the location of the original source of the data, and all others pull a copy of the entire data collection when they wish to synchronize their local copy with this master set (Figure 1).
The criteria used to determine whether the file has changed are the file’s size and its modification date and time, which is why synchronized time is also important in networked systems. For the most part, this is a fast calculation that is moderately effective, but where the distributed system does not use a uniform view of the time, then you have to resort to a more expensive calculation, such as the MD5 or SHA hash of the file contents.
This approach is used in the FTP-based file system mirror mode synchronization. It’s simple to use but can be extremely inefficient. For example, log files, which are written in append mode, will be transferred each time the file synchronization operation is performed. FTP in mirror mode is a poor choice of synchronization protocol given its mode of clear-channel data transfer, open credential exchange, and its inefficiency. If the file has been altered in any way, the entire file is downloaded from the server. When very large file collections are being synchronized, then the client and the server must walk through the entire file space, checking each file for changes in its size or modification time.
Can we improve on this behaviour?
RSYNC — Block-by-Block
One such mechanism is the rsync protocol. This protocol was designed by Andrew Tridgell in 1996 and was designed to be a highly efficient file synchronization protocol that addresses some of the shortcomings of FTP-based mirror synchronization. rsync attempts to minimize the amount of transferred data when synchronizing, as rsync treats a file as a sequence of blocks, and the update is performed on a block-by-block basis, rather than the entire file.
An rsync receiver sends a list of file names and checksums of each block in the collection of files that it is attempting to synchronize. Where the checksums differ, the sender sends a new block to replace that held by the receiver (Figure 2). A ‘block’ is defined by rsync as the maximum of 700 bytes and the square root of the file length. There is a little more to it than this simple description, designed to cope with insertions and deletions of data within the file, as distinct from simple appending, but this approximate description should suffice for our purposes.
rsync works well in some cases, particularly where the number of files in the data collection is relatively modest, the files are not massive, and the changes are incremental (such as appending in a log file). There is always a tradeoff going on in such protocols, and here the tradeoff is between the level of processing load being imposed on both the server and the receiver in terms of checksum calculation and block alignment and the load on the network in transferring data. rsync pushes a greater burden on the two ends of the synchronization function in an effort to reduce network traffic volumes.
When there is a large number of files in the data set, generating and sending the block-by-block sequence of checksums for each file can itself be an intensive process. The sender also must make a significant effort in checking these checksum sequences and realigning its block boundaries if necessary, and all this work is a constant overhead irrespective of the extent of the changes in the underlying data collection. For example, in a collection of a million files, this workload is a constant overhead, even if just one byte was changed in just one file. It is sometimes faster to simply forgo the checksum calculations being performed by the sender, and perform a simple FTP mirror pass, as the file checks are far faster to compute.
While rsync has its advantages in some scenarios, rsync does not appear to scale easily in terms of the size of the data sets being synchronized.
Snapshots and deltas
If you regard the current state of a data collection as a sequence of editing transactions performed in sequence on a previous data state, then this allows for an approach to data synchronization.
This approach is best described in the operation of the DNS IXFR protocol (RFC 1995). When a DNS zone changes, the primary server updates its Start of Authority (SOA) serial number. When a secondary server wishes to synchronize the local copy of the zone, it sends an IXFR request to the primary server, including its current SOA serial number. If the primary supports IXFR and holds a history of those changes in its journal file between the provided SOA value and the current SOA, the primary sends this sequence of additions and deletions to the secondary. As well as DNS data synchronization, this approach has been used in several contexts, including the Near Real Time Mirroring (NRTM) (RFC 7682) protocol for Route Registry synchronization. and the RPKI Repository Delta Protocol (RRDP) (RFC 8182) used in the RPKI.
There is a constant overhead placed on the server to take each component of the data set, such as a file, assign it a version number, and record in a journal the changes that were applied to the data between successive version numbers. The advantage of this approach is that what is passed from the primary source to the secondary systems is only the sequence of operations in the data that will transform the data from one version to the next. It is possible to apply this approach to individual files or collections of files, so even very large file collections can be treated as a single journalled artefact.
However, in certain scenarios this approach can be inefficient. To retrieve a current copy of the data, the client is sent the sequence of older changes that were made to the data from one version to the next, even if the changes are subsequently undone within the sequence of changes.
A better understanding of the nature of the data can improve this situation. For example, BGP is a synchronization protocol that can perform delta compression on the individual updates. Each BGP peer is only updated every (approximately) 30 seconds if Minimum Route Advertisement Interval (MRAI) timers are used. When a routing update is processed, it’s not the update itself that is stored for subsequent sending to a BGP peer, but the prefix value that is queued up.
When the MRAI timer expires, and it’s time to update the peer, the local BGP speaker dequeues each prefix and looks up its current state in the local Forwarding Information Base (FIB) and sends this state as an update to the BGP neighbor. If the local BGP speaker also keeps a local record of what it has sent to the peer, then if the current state is the same as the stored state, then no update is sent at all. This type of delta compression requires a certain data structure where all data items have a unique primary ‘key’ and a value. Multi-valued keys cannot be readily supported in this model.
Merkle trees
A Merkle tree is a hierarchical data structure in which large datasets are summarized into a single hash value. Individual data items are sorted in some canonical manner into a sequence. Each data item is cryptographically hashed. These hashes are paired according to their original order, and a hash is generated of the pair. This process of hashing hash pairs is repeated recursively until a single root hash remains (Figure 3). If the data is changed in any way, the hash value is altered, and this change in the hash value is propagated up the tree to the root hash.
This can allow two data sets to be compared very efficiently by comparing their root hash values. If the root hash values differ, then the data sets are not the same. The points of difference in the underlying data can be efficiently found by a comparative tree descent, ignoring those data items where the superior hash values are the same. These hash trees are used in many applications, including the ZFS file system, Bitcoin, the Interplanetary File System and the certificate transparency framework.
The prerequisite is the ability to define a canonical order of data items. The synchronization function operates efficiently over both large and small data sets, and it can efficiently locate the points of difference between two data sets.
Its use is currently proposed in the Resource Public Key Infrastructure (RPKI) and the ERIC protocol, where every client must maintain a synchronized local copy of the entire PKI, and as the use of the RPKI increases, the burden of synchronization grows as the product of the number of clients and the number of objects in the PKI. Merkle trees can break this cycle of growth with the observation that Merkle trees operate at a scale of the log of the number of data objects.
A further improvement in the comparison operation can be made by using this hash value in the tree as the data identifier for retrieval. Retrieving an object identified by its hash value for an intermediate node in a Merkle tree would return the ordered couplet of the two subordinate hash values, while the value of a terminal node would be the data item itself. Such a data-name scheme lends itself to anycast replication, further improving the scalability of the synchronization function.
Pull vs push
This quick run through various synchronization protocols has not addressed the other fundamental synchronization question. Should a client interrogate the server to see if its local copy differs from the server and pull the data if it differs, or should the server push the data to the client?
Let’s look first at the pull model using the CDS/CDNSKEY construct (RFC 8078) as an example. In the CDS framework, the delegated (child) domain publishes a DNSSEC-signed CDS resource record with the value of the record being a new key hash value that the delegated zone admin wants the parent to publish as a new DS resource record for this delegated zone. In this framework, the parent periodically scans the delegated zone using a query for a CDS record at the delegated zone’s apex.
If this record can be DNSSEC-validated using the delegated zone’s Zone Signing Key (ZSK), then the parent zone may then publish this record as the DS record for this delegated zone. This ‘periodic scan’ may be an issue in terms of performance and efficiency. RFC 8078 proposes that “… the period from the child’s publication of CDS/CDNSKEY RRset to the parent publishing the synchronized DS RRset should be as short as possible.” For an operator of a large DNS zone, such an imposition of querying each delegated zone for a new CDS record to pull may be a difficult task if the delay between successive pull queries to detect any change in the data is ‘as short as possible’.
Another example of a pull model is the RPKI publication system. RPKI clients are expected to maintain a synchronized copy of the entire set of published credentials, but the client does not know when an RPKI publisher has updated their published material. Without clear guidance as to the performance parameters of synchronization, the client processes are motivated to poll the publication points at a high frequency, adding to the scaling load experienced by the system.
In the DNS, recursive resolvers hold a copy of authoritative data in their local cache. However, the DNS provides the resolver with guidance on the synchronization parameters of the data. DNS records are given a Time-to-Live (TTL) parameter, and this informs the recursive resolver how it should use this locally cached information before checking for an update to the data. This is an important aspect for the overall performance of the DNS, allowing each data publisher to select their own preference between faster update propagation and improved server performance.
While pull is simple, it exposes the server to an unconstrained load. Not only will load increase with the number of clients performing this pull-based synchronization, but the load will also increase as clients decrease the time between successful pull checks. The DNS response to this situation, where the parameters of the level of synchronization are specified in a TTL field, provides a useful response to this issue, allowing the data to specify its own parameters of the degree to which a local copy of the data may fall out of synch with the authoritative version of the data.
Another major advantage of a pull model is that there is no need to enrol clients. There is no inherent limitation in a pull model as to who can synchronize via a pull function.
An alternative is a push model. In a push model, the server pushes the data to clients whenever the data changes. This model naturally lends itself to a delta model of synchronization, where the server pushes the change data to the client. In this case, the server has control over the granularity of synchronization, and there is little latitude for clients to improvise with their own procedures. An example of a push model is the BGP protocol, described above.
There is a hybrid model where the server sends a push notification to a client that an update to the data is available, but it is left to the client to subsequently perform a pull. This takes the uncertainty out of the synchronization function as it removes the need to perform polling. An example of this approach can be seen in the Generalized DNS Notification specification (RFC 9859)
Peer systems
So far, these models assume that there is a single source of the data, and a collection of clients who desire to synchronize their local copy of the data against this authoritative source. To improve the scalability of a distributed system, it is possible to impose a hierarchy on the system. Such a hierarchy expects that all participating synchronization points can act as both a client and a server. The authoritative server is assigned level 0 (for example), and all other systems are assigned a level number greater than 0. The constraint is that clients at level n can only synchronize with servers at level n – 1 (a more resilient system would permit a client to synchronize with any system with a level less than its own).
But what if there is no authoritative source of the data? What if every agent could alter the common data set, and all agents synchronize with this shared data state? This calls for an entirely different approach to synchronization where in addition to the data synchronization function, there is a need for this collection of agents to resolve conflicts without external intervention.
Distributed synchronization across peer systems ensures nodes agree on data states, order events, and coordinate tasks without central servers. Because peer-to-peer (P2P) systems lack a global clock or single source of truth, they rely on specific logical clocks, consensus models, and state propagation techniques to achieve consistency.
Logical clocks are sequential counters. In this model, a peer increments its clock and attaches it to outgoing messages. If a peer receives a higher timestamp, it updates its local clock to the received value + 1. This produces a total ordering consistent with causality, but concurrent events may still appear arbitrarily ordered.
Peers must determine how data is viewed across the network. In a ‘strong consistency’ model, the intended outcome is that once an update operation has completed, all future reads by any peer will return the latest version, incorporating this update. In true peer-to-peer contexts, this often requires heavy coordination protocols like Paxos or Raft. A weaker model termed ‘eventual consistency’ commits that updates to data will propagate to all peers but not necessarily immediately. There is also ‘causal consistency’, which ensures that operations which are causally related are seen by every peer in that exact order.
In scenarios where peers must agree on a single, shared state or value, systems may use Byzantine Fault Tolerance (BFT) to handle node failures and malicious peers. BFT allows a distributed network of peers to reach an agreement even if some peers fail or act maliciously. This approach forms the foundation of blockchain and consensus-driven P2P systems.
Conclusions
There are many approaches to data synchronization across distributed systems. They represent various points in a design space where network load is traded against client processing load, and there is scalability, complexity, robustness, and the internal structure of the data itself.
There is much to be said in favour of a brute force overwriting replication as it’s incredibly simple. But it cannot scale to serve distributed systems with many clients, large data sets or a requirement for continuous synchronization.
If the aim is to perform synchronization within a file system, there is much to be said for the rsync protocol, as it is simple to set up and use. But again, it cannot scale, and while it can dramatically reduce network loads, it increases the processing loads on both end systems.
Snapshots and deltas have their areas of application. Such systems can cope with large data collections, as the synchronization load is based on the volume of altered data rather than the volume of the underlying data collection. For large volumes of relatively static data, this form of synchronization may outperform the other approaches.
Merkle trees can be very effective in areas where there are large data collections and large numbers of clients. Such data structures are efficient to poll for changes, and they can be efficient in identifying the data that has changed and needs to be passed to the client.
These approaches are useful where the task is to synchronize a set of clients against an authoritative source. Peer-to-peer systems where there is a need to develop some form of consensus over what constitutes the authoritative shared state require different approaches that need to impose some form of sequential behaviour upon a distributed system to assist in the process of consensus formation.
Clearly, there is no single answer to synchronization. Much depends on the size and dynamic behaviour of the data collection, the number of clients, and the nature of the communications medium. In a system with reliable communications, it is possible to deploy a snapshot plus delta model, as the system can safely assume that all delta objects have been passed to all clients without alteration, while an unreliable channel may find that rsync may be more appropriate. Merkle trees are a good match for very large systems, as the synchronization check and the task of locating the components of the data set that require updating can be processed extremely efficiently.
The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.