The potential of erroneous outbound traffic

Image generated with AI.

Network monitoring has traditionally been built around a simple assumption: Most problems start from outside the network boundary.

As a result, operators and researchers have invested significant effort in analysing inbound traffic, looking for scans, attacks, and misbehaving external hosts. While this perspective has proven extremely valuable, it leaves an important question largely unexplored: What can we learn by observing the traffic that leaves our networks — especially when it does not receive a reply?

In this post, we highlight research that shows erroneous outbound traffic (traffic initiated by internal hosts that elicit no response) is a powerful and underused signal for uncovering silent anomalies. By focusing only on this narrow slice of traffic, we show how operators can unveil long-standing misconfigurations, stale deployments, and even signs of compromise that would otherwise remain hidden.

What is erroneous outbound traffic?

We identify several situations in which outbound traffic can be considered erroneous, namely when it exhibits one of the following behaviours:

Connection attempts that receive no response (for example, when the destination host is offline, firewalled, or does not exist).
Packets that trigger ICMP error messages, such as port unreachable or host unreachable.
ICMP error packets generated by internal hosts themselves, typically in response to late or unexpected replies.

These packets are often treated as background noise. In reality, they are frequently the by-product of misconfigured applications, faulty clients, outdated software, or malicious activity.

Erroneous traffic reflects services that are technically operational but behaving in unintended ways — a dangerous middle ground where problems persist quietly without triggering alarms.

Why focus only on failures?

At first glance, analysing only failed traffic may seem overly restrictive. However, erroneous outbound traffic has several properties that make it uniquely attractive for network operators.

It has a high signal-to-noise ratio, since it is by definition different from healthy traffic flows and therefore leads to a limited number of false positives pointing directly to actionable problems

Moreover, it represents a very small fraction of total traffic. In our measurements, erroneous outbound packets accounted for roughly 0.06% of all outbound traffic. This makes it feasible to log and analyse without significant storage or processing costs.

Capturing only what matters

To isolate erroneous outbound traffic on-the-fly, we implemented a stateful and privacy-friendly Software-defined networking (SDN) -based monitoring system, called Chamaleonet, that detects failure patterns in real time. Instead of collecting all packets and filtering offline, the system observes flows as they move through the network and dynamically identifies those that exhibit erroneous behaviour.

When an erroneous packet is detected by a timeout or an ICMP error, the SDN control plane installs flow rules that forward the erroneous traffic to a packet logger. Well-formed, bidirectional flows are ignored entirely, protecting privacy too.

This approach differs fundamentally from traditional intrusion detection systems. Rather than inspecting everything and raising alerts after the fact, it opportunistically collects only traffic that is already suspicious by nature.

A three-month view inside a campus network

We deployed this SDN system on a /16 campus network and collected erroneous outbound traffic continuously in our campus network.

Two high-level observations immediately stood out.

Erroneous traffic is persistent over time. Its volume remains relatively stable, independent of working hours or user activity. This strongly suggests that most erroneous traffic does not originate from transient human mistakes, but from continuously misbehaving hosts.
The traffic was highly skewed. A very small subset of internal hosts is responsible for a large fraction of all erroneous packets. In practice, this means that most of the ‘damage’ is caused by a handful of machines, making remediation far more tractable than you might expect.

Figure 2 — Persistent presence of erroneous packets.

Figure 3 — Few internal source IPs (x-axis) contribute disproportionately to overall erroneous traffic.

Silent anomalies brought to light

By examining representative cases, we uncovered several categories of previously unknown issues, none of which had been flagged by existing monitoring tools.

Suspected malicious activity was among the most concerning findings. We observed hundreds of internal hosts sending ICMP port unreachable replies to traffic originating from the public DNS resolver 1.1.1.1 on UDP port 500. This behaviour is consistent with either external port scanning or a reflection attack using a spoofed source address. In another case, a single internal host periodically contacted IP addresses associated with dismissed LockBit ransomware infrastructure, strongly indicating a compromised machine attempting command-and-control communication.
Faulty configurations were also widespread. Dozens of clients repeatedly sent DNS queries to bogon or localhost IP addresses — behaviour often caused by misconfigured VPN software, or incorrect DNS resolver settings. Similarly, we identified misconfigured Network Time Protocol (NTP) clients continuously querying unreachable peers, generating a steady stream of erroneous packets.
Stale deployments proved equally problematic. One internal system, part of a high-performance computing facility, kept contacting a decommissioned external software repository every second for months, generating millions of failed HTTP connections. The service was ‘working’ from the system’s point of view, but wasting resources indefinitely.
Aggressive but unintended behaviour from legitimate tools. Some hosts running DNS accelerator software issued parallel queries to many resolvers, but closed sockets as soon as the first response arrived. Late replies triggered ICMP errors, creating large volumes of erroneous traffic that masked other issues.

Remarkably, these few cases alone accounted for 11% of all erroneous outbound traffic, despite being unknown to administrators before this analysis.

Failure is informative

The key insight is simple: Failure is informative.

Erroneous outbound traffic offers a complementary view of network healthiness. It surfaces problems that are persistent, localized, and often easy to fix, once they are visible.

For operators, this means better security posture, reduced resource waste, and improved reliability, all achieved by analysing a tiny fraction of traffic that networks already generate.

Sometimes, the most valuable signals are not in what works, but in what quietly fails.

Andrea Sordello is a PhD student in Computer and Control Engineering at Politecnico di Torino, within the SmartData@PoliTO research centre. His research focuses on network traffic analysis.

The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.