A management interface, who is it for?
Modern web-based management interfaces help with the economy of scale. If you are a software vendor making a solution, supporting it is easier with clearly defined UI options rather than debugging obscure configuration file parameters. If you are an end-user, a management interface is there to make life easier for you as well.
Having a management interface helps you:
- Deploy the solution
- Make complex changes to it
- Generate management-level reporting for the key KPI
These features often become tender items and a vendor will find itself in a position where developing a management interface web UI is a must-have instead of a nice to have. Too often features are implemented in software through a tick box comparison since the rationale is that we must have them since our competitor has them. It doesn’t really matter whether the features actually serve the customer and their business function or not. From the customer’s perspective the scalable solution might actually be a management REST API, but many of the solutions still heavily rely on this manual management model instead, which doesn’t really help with scalability issues at the customer end at all.
Is it for your eyes only?
As the name suggests, a management interface is used to configure and maintain a device or a solution. Traditionally, this has meant console access either remotely or through a local terminal. Nowadays, a management interface often spells a web UI. The web is there mostly for ease of use but also to be able to visualize and manage complex configurations, which over a text-based command line interface, might be quite challenging.
In security-conscious applications, the management interface has been segregated from the user-facing services. The simplest way to do this is to make the management interface listen to a different port than the main application. This way, the management interface can be firewalled or access controlled even if the application or solution has only one public-facing IP. Too often, default configurations for (physical) appliances make it too easy to misconfigure the solution so that the management interface is listening on the WAN interface instead of the LAN one.
In either case, separating the management function from the rest of the business functions is a must-have and helps reduce the external attack surface of the application, since should a vulnerability be discovered in the interface it is not exposed to everyone and their grandma to talk to.
Attack surface vs attack vector
A management interface exposed to the whole Internet represents an attack surface that often is a softer target than the user-facing part of a given application or solution. As stated above, this attack surface is usually either ignored or left open by accident. Often, the interface is exposed to the Internet because it is not easy to restrict the management interface to a separate port or network interface. Furthermore, even if the interface is sitting on a separate port, there may not be an easy solution to expose it only to specific source IPs that are dedicated to management access. Nevertheless, the actual reason for this public exposure seems to be ease of access for the administrator — at the cost of operational security. Furthermore, the administrators may even be outsourced, which further complicates the issue, since configuration management rests with them and not with you.
Newly discovered or residual vulnerabilities can turn an attack surface into an attack vector. Too often, it is only then that people take notice. Even when a given vector is mitigated, nobody seems to care that the attack surface remains in place — ready for the next round of exploitation. The vendor stance often is that the interface is not meant to be exposed to the Internet. This is a poor excuse not to invest in something that would stand the test of time, and exposure to criminal elements.
Management interface vulnerabilities
In the past couple of years, management interfaces have faced scrutiny from security researchers and as a result, a plethora of vulnerabilities have surfaced. A common misconception seems to be that if a solution is a security solution its code base must be secure as well. Let’s take a firewall for example. Its user-facing functions are most likely secure, but its soft underbelly is the web-based management interface, which I will characterize below.
I mean a Java application that controls a firewall is likely to have vulnerabilities in it in the same manner as any other Java application. Moreover, such a management application needs privileged access to the solution, which in turn, means that a vulnerability discovered in it is likely to have serious consequences. Too often, we see directory traversal vulnerabilities in these applications, which means that the developers are not really focused on security.
How to dodge the bullet
The best mitigation for an on-premises solution is to use a dedicated management network with a dedicated physical interface through which the solution is managed. This is fine and dandy if we are talking about a physical appliance in the first place that resides on your premises. More often than not a solution is virtualized and resides in the cloud, which means the management network will have to be virtual as well.
Tip: A tried and true method is to use a secure bastion host to control the access to the management interface of a given solution. Using a public key authenticated SSH tunnel and a SOCKS proxy to gain access to the management web UI or API will raise the bar for attackers quite a bit. This, of course, places a burden to secure the bastion host properly, but that is quite a bit easier with OpenBSD and SSH rather than using an interface such as Ransomware Deployment Precursor (RDP), for example.
A common way to protect a web-based management interface is to use a Web Application Firewall (WAF) in front of the solution, but usually, this is just a sign of poor implementation of the solution where the user and management functions have not been properly separated. In any case, an often overlooked aspect of these bastion hosts or middleboxes is the lack of proper Authentication and Access Control (AAA). Making sure that the management connections and actions are actually properly logged and the logs are duplicated outside the bastion host is a good measure for the worst-case scenario — that the admin account is compromised on the client side.
Attack vector examples from recent history
Below, I will detail some of the recent public exposure related to management interfaces of various solutions. The sad fact seems to be that no one seems to note this exposure until either a high-profile breach has occurred through them or a critical RCE is disclosed to the public — or both. Three of the examples represent a traditional appliance model (F5, Sophos, Zyxel) and two of the examples a modern API interface (Kubernetes and WinRM).
F5 TMOS vs TMUI
F5 produces middlebox security products, which help organizations manage their web applications through load balancing, reverse proxying or by providing a web application firewall. These complex applications of course also call for complex management interfaces, which have been subject to severe vulnerabilities over the past two years.
The most recent attack vector disclosed in May 2022 is CVE-2022-1388, which allows remote code execution on the management interface. Above, I have visualized the total number of publicly reachable F5 TMOS platforms (33,900) and the total number of publicly reachable F5 TMUI management interfaces (1,650). 4.9% may not sound bad, but these middlebox applications are something these organizations depend on to protect their API endpoints and web applications, that is, the core of their online business.
Sophos Firewall
Sophos produces firewalls that ship with a web portal for user access, as well as admin access. This attack surface was turned into an attack vector through CVE-2022-1040, which allows an unauthenticated user to execute code remotely on the firewall. Even if these vulnerabilities are fixed, Sophos does recommend its users not expose these portals to the WAN interface.
The total number of exposed management interfaces in May 2022 was roughly 160K, where approximately 1/3 of these devices were located in India and the US.
Zyxel Firewall
Zyxel Networks is the manufacturer of Zyxel firewalls. CVE-2022-30525 turned the zero-touch provisioning feature into unauthenticated remote code execution via OS command injection through the CGI binary. To me, this type of vulnerability smells strongly of the 1990s, since who is using CGI anymore in this day and age, especially for security device management?
For Zyxel firewalls, the attack vector is most prominent in France and Italy, which accounted for roughly half of the 16K vulnerable devices in May 2022.
Kubernetes API
The Kubernetes API lets you query and manipulate the state of objects in Kubernetes. The core of Kubernetes’ control plane is the API server and the HTTP API that it exposes. Users, the different parts of your cluster, and external components all communicate with one another through the API server.
Source
The statement above defines the Kubernetes API as a very powerful management interface, since if poorly configured, a malicious third party may manipulate your running workloads or generate their own. That is why it is recommended that you do not expose this API directly to the Internet to provide some defence-in-depth for the management interface of your Kubernetes cluster.
Warning: As of May 2022, approximately a million Kubernetes API endpoints have been exposed to the entire Internet making it one of the biggest attack surfaces in the cloud environment.
PaloAlto Unit 42 has written an in-depth analysis of the unsecured Kubernetes clusters, which can be used as an attack vector; luckily it is only numbered at 6K API servers.
Windows Remote Management Service, WinRM
Another remote management interface with a massive attack surface globally is WinRM. According to Rapid7 research, it is a SOAP interface that you can use to manage the box remotely. One mitigating factor may be that by default the service may be in a non-functional state, which limits the attack vector even if the attack surface is present.
Roughly 1.8M WinRM interfaces are exposed on a monthly basis to the whole Internet. The biggest geolocations for publicly exposed WinRMare USA (~600K), China (~500K) and Hong Kong (~180K). In terms of an attack vector, CVE-2021-31166, a remote code execution vulnerability in WinRM exemplifies why this interface must not be directly exposed to the Internet.
Wrapping up
By now, it should be evident that an exposed attack surface will, in time, be turned into an attack vector and the only practical way to mitigate that attack surface is to control access to it. In this post, I’ve focused on management APIs, which in my opinion, form a dangerous attack surface hidden in plain sight. Too often, sysadmins seem to be led astray by thinking that an interface is secure since it has been developed by an information security company. Given the number of vulnerabilities in security products over the past year(s) one should not be lulled into complacency by authority alone. Letting everyone interact with your management API is not a good practice in the long run.
Tip: It is better to invest your time into operational security at your leisure, rather than expend it in a mad dash of incident response engagement. In the context of this write-up, it specifically means identifying and securing access to your management interfaces. As stated above, properly secured bastion hosts and SSH are your friends in this endeavour.
Lari Huttunen has been working in cybersecurity since the late 1990s, with an interest in early warning, researching the impact of known vulnerabilities and exposures on a global scale, and understanding cybercrime from the victim’s perspective.
This post is adapted from the original at Public Exposure.
The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.
Thanks for the interesting article. Ever since my first workshop with APNIC in Lae, Papua New Guinea Cyber Security has become an important part of all my ICT Infrastructure deployment both hardware and software.
This article has given me a better understanding of the importance of segregation of management services to end user services.