In the first part of this series, I looked into well-known protocols like NetFlow v5, NetFlow v9, and IPFIX. In this article, I’ll continue the discussion and focus on protocols that offer more flexibility:
- sFlow v5
- Port mirror
- Sampled port mirror
- Sampled port mirror over GRE
- Raw headers over IPFIX or Netflow v9
sFlow v5
I’d like to start by discussing the protocol’s name. Despite having ‘flow’ in its name, it has nothing to do with flows, as it lacks any flow tracking logic and works on a packet basis. The name might suggest similarities with the NetFlow protocol family, but this is not the case — it is a completely different protocol. This protocol was developed by the talented team at InMon Corporation.
In the NetFlow family of protocols, sampling is optional, but for sFlow, it is an integral part of the protocol design, and you must enable sampling to use it. The main advantage of sFlow is the near-complete absence of state information and flow tracking tables. To simplify the sampling rate encoding, which is complex in NetFlow v9 and IPFIX, sFlow provides the sampling rate value directly in each packet (similar to NetFlow v5).
Instead of parsing each network packet and providing it as a set of fields (as in NetFlow) to the collector, sFlow simply provides the first X bytes of the packet (typically between 60 to 140 bytes or more) as-is, deferring the packet parsing to the collector. In this case, collector developers need to create their own network packet parser, which we’ve done for FastNetMon. While this might sound more complex, it allows us to support non-standard packets (such as MPLS, VXLANs, and GRE) and include fields that are not typically provided by vendors in NetFlow (such as TTL). For DDoS detection, this information is absolutely necessary.
Let’s take a look at the sFlow sample header in C structure format (Figure 1).
As Figure 1 shows, the sampling rate value is provided directly in the packet, along with some meta information about the source and output interfaces. Everything else is contained in the binary packet header, which needs to be parsed.
From a theoretical standpoint, sFlow is arguably the best protocol for DDoS detection on paper. However, the reality is quite different. Unfortunately, many sFlow agent implementations have serious issues (refer to my own article about Huawei or check the list of limitations from Juniper MX’s sFlow v5 implementation), which negatively impacts the popularity of the sFlow protocol.
Some models of switches and routers simply do not allow setting a sampling rate that would provide accurate traffic rate calculations, making it impossible to use certain network equipment for attack detection. This issue is typically caused by slow CPUs on the control plane.
To provide a hint about reasonable sampling rate values, I’ll quote an article from InMon (Table 1).
| Link speed | Sampling rate |
| 10Mb/s | 1 in 200 |
| 100Mb/s | 1 in 500 |
| 1Gb/s | 1 in 1,000 |
| 10Gb/s | 1 in 2,000 |
Let’s summarize our view on sFlow v5.
Benefits of sFlow v5
- Almost instant export of observed traffic (hundreds of milliseconds)
- Provides access to packet header
- Simple sampling encoding
Issues with sFlow v5
- The sampling rate selection process is not easy to understand
- The performance of sFlow agents on routers and switches is heavily constrained by hardware, leading to extremely high sampling rates that do not provide an accurate overview of traffic.
- Traffic parsing is complicated and very hard to do securely (IPv6 headers, MPLS, QnQ)
- Lack of useful meta information (MPLS tags, VRF IDs, next hop)
- There is a long list of constraints and limitations from the router side (lack of LAG support for example)
Port mirror
Also known as TAP, SPAN, or RSPAN, port mirroring is typically a last resort when no other network traffic telemetry method works. It is more expensive in terms of port cost, forwarding capacity, and hardware for monitoring traffic compared to other methods. I recommend thoroughly evaluating all possible alternatives before choosing this option.
Almost every switch can mirror traffic, but it may be more challenging for routers (especially soft routers). Be sure to check your vendor’s documentation. Port mirroring is implemented by selecting a list of source interfaces, where all TX and RX traffic will be copied to one or more target interfaces. Keep in mind that the target interface must have much higher capacity, as it will receive both RX and TX traffic.
If your network equipment cannot implement port mirroring, you can install optical splitters, which allow you to achieve the same result at the transmission medium level.
After configuring port mirroring, the next step is to connect the target interface to a machine running Linux. From there, you can use these specialized tools that function as sFlow v5 or NetFlow agents to collect and analyse the mirrored traffic:
- hsflowd (sFlow v5)
- ipt_netflow (Netflow v9 and IPFIX)
- nProbe
If you’re looking to process high-bandwidth traffic using your own tools, check out my series of blog posts dedicated to traffic processing on Linux.
After reading this article, you’ll learn that receiving 100G of traffic on a machine requires significant effort, demanding exceptional hardware, network, and Linux expertise. In the case of DDoS attacks, we may encounter line-rate scale packet-per-second attacks, which make this task nearly impossible to manage. As discussed, regular port mirroring is not an ideal option for DDoS detection.
Let’s summarize our view on port mirror.
Benefits of port mirror
- Complete access to all information in the packet
- Supported by almost any network switch or router
Issues with port mirror
- Requires a lot of CPU time for the collector to receive and then parse traffic
- Lack of meta information (ASN, VRF IDs, source and destinations ports)
- Requires spare ports on router
- Requires high performance network cards on the collector
Sampled port mirror
To address the extraordinary demand for port capacity and CPU power required to process 1:1 port mirror traffic, we can leverage network equipment that supports statistical packet sampling, delivering only a small fraction of all traffic observed by a router or switch to the monitoring machine. Unfortunately, only a few vendors, like Juniper on their MX platform and Nokia, support sampled mirroring. Additionally, Juniper MX can crop packets, sending only the first X bytes of each packet instead of the entire packet, which further reduces the CPU resources needed to handle the traffic.
In this case, you can monitor a 100G interface fully loaded in both TX and RX through a 1G interface, which will only be loaded with 200 Mbit (RX + TX). A 1G interface is much easier to handle, and you can even use tools like tcpdump to inspect the traffic.
Regular port mirroring requires direct L2 connectivity between the interfaces you want to monitor and the machine receiving the mirrored traffic. This can be challenging to achieve. However, some vendors support sending port mirrors via GRE encapsulation, allowing you to place the traffic analytics machine anywhere in the network and route the traffic over L3.
Here’s our overview on sampled port mirror.
Benefits of sampled port mirror
- Requires way less port capacity
- Requires way less CPU on the collector side
- No need for high performance NICs
Issues of sampled port mirror
- Many vendors do not support it
- No way to get sampling rate, needs static configuration
- Lack of meta information (ASN, VRF IDs, source and destinations ports)
- GRE requires MTU tuning to deliver packets larger than 1,500 bytes
Payload via IPFIX or Netflow v9
This protocol sadly has no specific name, as it’s simply a peculiar way of using NetFlow v9 and IPFIX protocols to carry packet headers within them. We may call it ‘PSAMP’, as referred to in the RFC, but it’s not entirely correct either.
Do we have vendor specific names? Sure. Juniper calls it inline monitoring services and Cisco calls it IPFIX 315.
What are the differences from regular Netflow v9 and IPFIX?
- It’s sampled by default and cannot be used in 1:1 mode
- The lack of any flow tracking or flow aggregation addresses all issues related to hardware overload.
- Instead of parsing the packet and providing it as a list of fields, the router simply exports the first X bytes of the packet, which brings all the benefits of the sFlow v5 protocol while using the well-known Netflow v9 or IPFIX encoding.
- It’s implemented on the data plane of routers and is lightning fast
Let’s look at an example flow (Figure 2).
As Figure 2 shows, it’s significantly different from regular Netflow v9 and IPFIX. Instead of dozens of fields provided by the router, we only have interface numbers, traffic direction, and a large binary field carrying the packet header, which must be parsed by the collector.
Unfortunately, to carry the sampling rate, this protocol family uses the standard encoding employed by Netflow v9 and IPFIX (Figure 3), which is quite challenging for the Netflow collector to parse.
The only known routers that support this family of protocols are Juniper MX, Juniper PTX, Cisco NCS, and Cisco ASR 9000.
This protocol is essentially the best choice for DDoS detection, and the majority of our largest deployments are built around it.
Benefits of the PSAMP family of protocols:
- Clearly the best and most capable protocol on the market
- Almost instant traffic delivery
- Well-defined format for sampling rate encoding
- Provides all information available in the header
- Provides meta information (interface numbers, direction)
- Can be extended easily
Issues with the PSAMP family of protocols
- Only a few vendors support it
- Extremely high complexity of integration for the collector side
- Limited by the set of fields provided by the vendor
Clearly, this protocol competes with sFlow v5 and has one great advantage — vendors have implemented it correctly on their top-tier routers.
Thank you for reading! I hope more vendors will address issues with existing protocols and support state-of-the-art protocols discussed in this article. Feel free to share your feedback in the comments.
Pavel is the author of FastNetMon, an open source DDoS detection tool with a variety of traffic capture methods and works in software development and community management.
Adapted from the original on Pavel’s blog.
The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.