‘Functional, free and secure by default’, OpenBSD remains a crucial yet largely unacknowledged player in the open-source field.
This series aims to highlight the project’s signature security features and development practices — razor-sharp focus on correct and secure code coupled with continuing code audit — as well as the project’s role as a source of innovation in security practices and an ‘upstream’ source for numerous widely-used components such as OpenSSH, PF, LibreSSL and others. Part 1 focused on the history, Part 2 focused on usage and user experience, and this final part will looks at that packet filter (PF).
I mentioned PF, the OpenBSD packet filter, earlier. I must confess that PF has been an important part of my life in various contexts since the early 2000s. Over the years, things I have written have contributed to creating the popular but wrong perception that OpenBSD was primarily a firewall operating system. There are a lot of useful and fun features that turned up in or in connection with PF over the years and were pioneered by OpenBSD. Some features were ported to or imitated in other systems, while others remain stubbornly OpenBSD only.
So I will touch on some of my favourite PF and PF-attached features, in quasi-random but almost chronological order.
Beating up spammers with OpenBSD spamd(8) since OpenBSD 3.3
Way back when I started playing with OpenBSD in general and PF in particular, I was already responsible for the SMTP mail service for my colleagues. My gateways by then ran OpenBSD, while the mail server, Rosalita, named after a Springsteen song, was a not too badly specced server running FreeBSD with exim as the mail transfer agent that fed the incoming messages to spamassassin and clamav for content filtering before handing off to user mailboxes.
So when it dawned on me that I could set up spamd(8), the spam deferral daemon on the Internet-facing gateway to save load on the poor suffering Rosalita that was running hot with content filtering, I was quick to implement a setup that sucked in well-known block-lists.
Going grey, then trapping
The effect was obvious and immediate; the mail server’s fans grew noticeably quieter. When greylisting was introduced in spamd soon after, I implemented that too and witnessed yet another drop in pitch and intensity of the sound from Rosalita’s fans. Then a couple of releases later, greytrapping — the practice of adding IP addresses of incoming SMTP connections to blocklists if the attempted delivery is aimed at a known-bad address in the target domain — was introduced, and that sounded like enough fun that I just went ahead and did it.
The idea of detecting spam senders by the bogus addresses they were already trying to deliver to just sounded too good to not try. And we knew that getting started would be pretty easy too. We had seen rejects for addresses that had never existed in our domains in our mail server logs for quite a while, so it was simply a matter of harvesting from a fairly bountiful source and adding stuff that we were sure would never be deliverable here to the spamtrap list. I think the first setup had only a couple of hundred entries in it, but I did not note the exact number at the time.
By July 2007, I had decided to publish both the list of spamtrap addresses and an hourly dump of the greytrapped addresses. Both remain free to download. The list of spamtraps, harvested from various log sources, by now numbers just over 270,000 imaginary friends, while the number of trapped hosts is typically in the 3,000 to 5,000 range. We occasionally see the list swell to 20,000 or more when high-volume campaigns run with bad address lists fed to them. I am pretty sure it went over 100,000 at one point.
It’s fun to watch, and it looks like a significant subset of the spamtraps has made it into the address lists of active spam operations. I frankly never thought I would still be collecting spamtraps from logs all these years later. Yes, it all sounds a bit absurd, but it is effective for keeping our mailboxes largely spam free, even though it feels at times like running a weird found object-ish art project. Anyway, a summary of the lists we publish can be found in this article.
The brutes, the password gropers and the state tracking options
If you run an SSH service or any kind of listening service with the option to log in, you will see some number of failed authentication attempts that generate noise in the logs. The password guessing, or as some of us say password groping, turned out to be annoying enough that OpenBSD 3.6-current and later OpenBSD 3.7 introduced a set of features to use data that would be available in the state table, to track the state of active connections. It would also act on limits you define such as the number of connections from a single host over a set number of seconds.
The action could be to add the source IP that tripped the limit to a table. Additional rules could then subject the members of that table to special treatment. Since that time, my Internet-facing rule sets have tended to include variations on the following:
table <bruteforce> persist block quick from <bruteforce> pass inet proto tcp from any to $localnet port $tcp_services \ flags S/SA keep state \ (max-src-conn 100, max-src-conn-rate 15/5, \ overload <bruteforce> flush global)
This means that any host that tries more than 100 simultaneous connections or more than 15 new connections over 5 seconds is added to the table and blocked, with any existing connections terminated.
It is a good practice to let table entries in such setups expire eventually. At first, I followed the spamd(8) defaults example and set the expiry at 24 hours, but I switched to four weeks, then six weeks as groperbots tend to stay broken for that long. And since they target any service you may be running, state tracking options with overload tables can be useful in a lot of non-SSH contexts as well.
It is also worth noting that state tracking actions are useful for essentially all services. The article Forcing the password gropers through a smaller hole with OpenBSD’s PF queues has a few suggestions on how to handle noise sources with various other services.
One final point I would like to make about state tracking and actions is that much like the greytrapping feature of spamd, this feature gives you the tools to build a configuration that adapts to network conditions and learns from the traffic it sees.
While this does not rise to the level of being an actual Artificial Intelligence or AI, this has enough ‘buzzwordability’ potential that I remain to this day extremely puzzled that none of the other big names at least imitated those features in their own products and marketed it for all it would be worth. I certainly know what I would have done in their position. But then I am more of an engineer than a marketer and in the contexts where I call the shots, the best option is just to keep running OpenBSD.
We went to modern queueing
OpenBSD has had traffic shaping available in the ALTQ subsystem since the very early days. ALTQ was rolled into PF at some point, but the code was still marked experimental 15 years after it was written, and most people who tried to use it in anger at the time found the syntax inelegant at best, infuriating or worse at most times.
So Henning Brauer took a keen interest in the problem and concluded that all the various traffic shaping algorithms were not needed. They could all, except one, be reduced to mere configuration options, either as setting priorities on a pass or match rules, or as variations of the mother algorithm Hierarchical Fair Service Curve (HFSC for short) theme.
Soon after, another not-small diff was making the rounds. The patch was applied early in the OpenBSD 5.5 cycle, and for the lifetime of that release, older ALTQ setups were possible side-by-side with the new queuing system.
The feedback I get is that the saner syntax in the new queueing system led to more users taking up traffic shaping. Here is the queue setup that I developed for one of my sites:
queue rootq on $ext_if bandwidth 20M queue main parent rootq bandwidth 20479K min 1M \ max 20479K qlimit 100 queue qdef parent main bandwidth 9600K min 6000K \ max 18M default queue qweb parent main bandwidth 9600K min 6000K \ max 18M queue qpri parent main bandwidth 700K min 100K \ max 1200K queue qdns parent main bandwidth 200K min 12K \ burst 600K for 3000ms queue spamd parent rootq bandwidth 1K min 0K max 1K \ qlimit 300
While tying the queues into the subsequent rules with a set of match rules just following the above block.
This is what triggered the need to write the third edition of The Book of PF. The book includes descriptions of both the new and the old system as well as tips on how to make a smooth transition. The ALTQ code was removed from OpenBSD during the OpenBSD 5.6 cycle but continues to live on in some form in FreeBSD and NetBSD.
And yes, if you think my queues setup punishes spammers a bit more in addition to being subjected to spamd(8), you’re right.
pflow(4) offers network insights lite
Everybody who has been tasked with looking after a network has been a little curious about what moves around there at some point. At times we will see situations where it is essential for troubleshooting purposes to see the traffic flows with data about endpoints, packets and bytes transferred, protocols and so forth.
If you do not need to see the data itself, but rather the metadata, the NetFlow standard and its close cousin, IPFIX, offers just that. Netflow tools existed as packages on OpenBSD already, but from OpenBSD 4.5 PF has the pflow state tracking option, paired with the pflow(4) virtual network interface, which together offers a full netflow sensor package.
Set up one or more pflow interfaces to send data to one or more collectors, and add the pflow option to specific rules or as a state default and you have started your collecting. You can even have metadata for traffic matching specific rules going to separate pflow devices and collectors.
My field notes in Yes, You Too Can Be An Evil Network Overlord — On The Cheap With OpenBSD, pflow And nfsen offers some practical examples and insights, including how we use a pflow setup to track down a noisy machine on a somewhat critical network as well as some pointers to valuable further reading.
LibreSSL, the great deobfuscation
People tell me they think that the reason LibreSSL was created was the Heartbleed bug. This isn’t the case but is damn close.
The LibreSSL project was started a few weeks before Heartbleed became common knowledge. LibreSSL is the result of a group of OpenBSD developers taking the existing OpenSSL code and starting to fix it.
This time it was not a matter of a bad license. This was the result of the number of OpenBSD developers who examined the OpenSSL code that had been part of the OpenBSD base system since quite early on, and turned away in disgust and with symptoms of physical pain, reached a critical mass of sorts. I had heard OpenBSD developers complain about the absolute horror of the OpenSSL code for at least ten years. The code quality was just that bad.
What happened next was that a group of hardened OpenBSD developers grabbed the OpenSSL code and started two activities in parallel. One was looking in the OpenSSL request tracker for bugs that had not been addressed. The other was reformatting the OpenSSL code into something resembling the OpenBSD style of readable and maintainable C.
With the code in more readable form, discovering what it did become easier. In addition to a few obvious eye-stinging bugs, the LibreSSL developers found several oddities, including, but not limited to:
- Code was never deleted even when it became irrelevant or obsolete.
- OpenSSL did not use the system memory allocation system but rather opted for their own, which never actually deallocated memory, but rather used LIFO recycling, and could easily be made to put private info into logs.
- It all being written in ‘OpenSSL C’, which according to beck@ is a dialect of the ‘worst common denominator’.
It is worth digging out the various articles and presentations made by LibreSSL developers over the years, with specific emphasis on Bob Beck’s BSDCan talk on the first 30 days of LibreSSL (available on YouTube), which is the original source of the term code flensing.
Since the OpenBSD 5.6 release in 2014, LibreSSL has been the default TLS library in OpenBSD. LibreSSL has been ported elsewhere based on the -portable variant.
For my own part, I can only attest to not ever running into a TLS problem that was LibreSSL’s fault. It probably still has bugs, but it is a much healthier choice than its predecessor.
This series was my list of life-improving OpenBSD events — I’d love to hear yours
As I warned earlier, this has been about my personal list of OpenBSD events that I remember fondly.
I am sure your list is at least a little different. I am sure there are things from the innovations page that I have simply forgotten about.
Each release comes with a detailed list of changes, and pointers back to the equivalent pages for previous releases. Here is the Changelog for OpenBSD 7, which has just been released.
I would love to hear about your favourite OpenBSD moments.
Peter N. M. Hansteen is a puffyist, daemon charmer and penguin wrangler.
Adapted from original post which appeared on BSDLY.
The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.
openbsd gained fq_codel some time back. Try it. I’m not sure if it’s correct…