Nowadays it is quite impossible to defend against cyber threats with technology only. There have been far too many instances of security breaches caused by cyber attackers breaching the security perimeter and performing a lateral movement with various known and unknown (at least not published) evasion tactics. As a result, automated security systems are becoming less effective and cyber attackers remain undetected within the network while they carry out their harmful activities (for example, successful data breaches).
This is why skilled human resources are an essential part of improving the cybersecurity posture for corporations and governments.
We at BGD e-GOV CIRT assist local corporations and industry groups in Bangladesh with improving their security personnel’s experience by developing scenarios to help build the skills required to effectively deal with cyber threats.
During our 2021 Financial Institution Cyber Drill, 204 security professionals in 38 teams were given the task to act as ‘Incident Handlers’ and identify, investigate and provide recommendations to resolve these issues from the artifacts provided by BGD e-GOV CIRT.
The cyber drill scenario simulated a situation where one of Bangladesh’s prominent financial organizations, named FIN, is worried about a potential cybersecurity breach because one of the IT personnel found a threatening text file in their Active Directory Domain Service (AD DS) and Exchange Server Administrator profile (in the desktop of both servers).
The network of the simulated environment was quite simple (Figure 1). Both the Domain Controller and the Exchange Server were inside the demilitarized zone. The Exchange Server had Internet connectivity and ports have been forwarded from the Internet Gateway Router to facilitate email communication.
Here is the related technical information:
Server: Domain Controller (DC)
Operating System: Windows Server 2016
Role: Active Directory Domain Service (AD DS)
Domain: FIN.LOCAL
IP address: 10.10.10.1/24 [Not routable to the Global Internet]
Server: EXCHANGE2016
Operating System: Windows Server 2016
Role: Mail server (Exchange Server 2016)
Domain: FIN.LOCAL
IP address 1: 10.10.10.10/24 [Not routable to the Global Internet]
IP address 2:192.168.85.210/24 [Assume these IP address network was routable to the Global Internet]
Login credential for Administrator (privileged user) and non-privileged user ITsupport was provided.In this post, we will go through the scenario with you as we did with the participants. We have categorized the overall challenge into five sections:
- Conformation of the incident.
- Persistence.
- Lateral movement.
- Privilege escalation to domain administrator.
- Stealthy advanced persistent threat.
Validating the cyber incident
As incident handlers, our first duty is to analyse the attack surface (internal or external) and validate whether a cyber incident has already happened or if it is only an assumption. As per the network diagram (Figure 1), we can see the Exchange Server is accessible via the Internet.
First, we need to consider the recent attack trends on the Exchange Server and test them. If we consider external cyber attacks, then from the network diagram, we found the email communication system (Exchange 2016) should be the attackers first target for entering the AD Domain.
A global wave of cyber attacks and data breaches began in January 2021 after four zero-day exploits were discovered in on-premises Microsoft Exchange Servers, giving attackers full access to user emails and passwords on affected servers, administrator privileges on the server, and access to connected devices on the same network. Microsoft named Hafnium as the group responsible for the 2021 Microsoft Exchange Server data breach. Microsoft identified the following vulnerabilities as being exploited: CVE-2021-26855, CVE-2021-26857, CVE-2021-26858, and CVE-2021-27065. In our scenario, we need to validate whether our Exchange 2016 system is vulnerable to this attack and if it has already been exploited. Microsoft has provided detection tools and guidance to assist users needing to do this. By running the Exchange On-premises Mitigation Tool with MSERT in detect-only mode, we will be able to determine whether our Exchange 2016 system is vulnerable:
The script output (Figure 2) shows that our Exchange 2016 system is vulnerable to this cyber attack.
Now we have to find out whether active exploitation already took place or not. To do so we used the Microsoft script Test-ProxyLogon.ps1, which detects successful exploitations.
Figure 3 shows that our system has indeed been exploited (red box). Note the timestamp is 2021-10-05T10:55:41.116Z (this is in Universal Coordinated Time (UTC) format). This timestamp is very important as we use this to check all other system logs to detect further activities of the attacker.
As the attacker is attacking a web service (SSRF vulnerability), there is a high chance they initially created a web-based backdoor into the system. As it is in an Exchange Server, which runs over IIS, this backdoor file extension is likely to be an .aspx.
From our C:\inetpub\logs\LogFiles\W3SVC1\u_ex211005.log we found one odd looking aspx file, which was named WgUP.aspx (Figure 4).
In the server (Figure 5), we found that the ExchangeAdmin folder has nothing in it but the ADADMIN folder had some user profiles.
None of these names were provided by the ‘FIN authority’ therefore they can be considered unusual. Interestingly, both folders were created after the system was initially exploited. For example, the ExchangeAdmin folder was created on 5 October 2021 at 19.15.27 and the ADADMIN folder was created on 6 October 2021 at 10.41.47 (Figure 6).
From the above analysis, it becomes clear that the attacker started their malicious activity at 5 October 2021 at 10:55:41.116Z (UTC) or 16:55 (GMT +6) local time.
The Windows Event Viewer shows a log of application and system messages, including errors, information messages, and warnings. More information can be obtained from the Event Viewer so we will use our detected timestamp and filter logs from the Windows Event Viewer for further analysis.
Note, Microsoft released updates to patch this exploit on 2 March 2021 for Microsoft Exchange Servers 2010, 2013, 2016 and 2019.
Persistence
After successfully exploiting a targeted system, attackers usually plant a backdoor. There is a hypothesis that if attackers inject a backdoor, it runs during system bootup or login. Autoruns is a useful tool from Microsoft to detect if any unusual programs are running during system bootup or login.
Using Autoruns for our exploited Exchange 2016 system, we detected an unusual program named cgWxiylwSDs.vbs, which loads from the C:\Windows\Temp Directory (Figure 7). The name, AD_SERVER_START, makes it seem like a legitimate process.
If we submit this information into www.virustotal.com for further analysis (Figure 8), we can see that this is VisualBasic (VB) malware.
Lateral movement
After some manual analysis, we detected some PowerShell events with ID 400 and 403. We found the attackers downloaded a tool named wget.exe, and one PowerShell script named Invoke-Mimikatz.ps1, which are both found in the root of C: drive.
The wget.exe is an executable file that can download files from another web server (Figure 9).
Mimikatz (here Invoke-Mimikatz.ps1) is a leading post-exploitation tool that dumps passwords from memory, as well as hashes, PINs and Kerberos tickets. It’s also used for pass-the-hash and pass-the-ticket attacks or for building Golden Kerberos tickets. In essence, this makes post-exploitation lateral movement within a network easy for the attacker. The proof of download of this script can be seen in Figure 10.
Under the C:\EXCHANGEADMIN, we detected a few text files with a creation/last written date of 5 October 2021 after 18:00 (GMT +6).
Now we need to look for suspicious files.
From the above screenshot (Figure 11) we can assume the attacker performed network queries like ping and arp for obtaining information on other systems.
In Exchange 2016 (Figure 12), we see execution of the Invoke-Mimikatz.ps1 PowerShell script at 18:01:42 (GMT +6). By running Invoke-Mimikatz.ps1, the attacker dumped the windows NT (New Technology) LAN Manager (NTLM) hash and may have extracted the clear text.
Mimikatz requires administrator or SYSTEM rights to perform its actions. As the attacker successfully ran Mimikatz (we know this by the hash output), we can confidently assume they gained the local administrator or SYSTEM rights of the Exchange 2016 system. This administrator privilege is only limited to the local Exchange 2016 system and not the entire domain.
In the Domain Controller (DC) server, we found Event ID 1149 (Figure 13), which tells us that on 5 October 2021 at 19:47:29 (GMT +6) user ITsupport was successfully authenticated using the Remote Desktop (RDP). As we could not find any trace of tools to use techniques like pass-the-hash and login, it indicates the attacker successfully cracked the NTLM hash password for the non-privilege user ITSupport of the FIN.LOCAL domain and successfully logged on to the organization’s DC system.
Privilege escalation to domain administrator
Obtaining a non-privilege user’s credentials is not a cyber attacker’s final goal. The attacker needs to escalate privileges to the highest level, which is the domain administrator. They will have to detect a privilege escalation vulnerability from the initially exploited Exchange 2016 system and gradually take control of the entire domain. As of now, the attacker has the capability to only login to the domain server.
Let’s go back to our Exchange 2016 system and carefully check the PowerShell event logs. We found one event, ID 800, was executing a script (C:\EXCHANGEADMIN\s.ps1) with command argument $sourceSpooler (Figure 14).
After some research (Figure 15) we can see the purpose of this script is to find out whether the spooler (MS-RPRN) is remotely available or not. So, the attacker checked the Exchange 2016 system whether the DC system’s spooler (MS-RPRN) service is running or not. This knowledge can help the attacker pick their next target.
In the DC system, after checking the PowerShell event log (Figure 16), we found one PowerShell script named a.ps1 was executed (From C:\Users\ITsupport\Desktop) at 19:59:59 (GMT +6) by user ITSupport (the compromised user).
From this event log, we also found the a.ps1 script started with the following lines (Figure 17):
The script states this is POC code for the Windows Print Spooler service known as the PrintNightmare vulnerability. The exploit code also states that, if the exploitation is successful, it will add a new local administrator user. In addition, we also found Event ID 808 (Figure 18) that indicates that the PrintNightmare exploitation and attacker may have raised its privilege by using this method.
From Event ID 316 (Figure 19) we also found the attacker loaded a malicious .dll file (hacked.dll) with SYSTEM user privilege on 5 October 2021 at 20:02:17 (GMT +6):
If we execute net user and Get-ADUser, we found a username ADADMIN (Figure 20), which was not listed in the initial login information (this user was created on 5 October 2021 at 20:02:16 (GMT +6).
By exploiting this vulnerability on the DC system, the attacker created the ADADMIN user and added this user to the Domain Controller’s Administrator group, which allowed the attacker to access every resource in the FIN domain with elevated privileges.
This vulnerability is considered critical because it affects all versions of Windows, including servers and workstations. To secure the system, in addition to installing the security updates, Microsoft has listed several workarounds.
Stealthy APT
Returning to our network diagram (Figure 1), we know the DC’s IP is 10.10.10.1, which is not globally routable. So, how is the attacker maintaining access to the DC and the entire FIN domain? Have they created some rule for this persistent access? Let’s check the available system log for any new rule or any modification that took place on the exploitation date.
In the DC we executed the following PowerShell command (Figure 21) to check for any new firewall rules that were added to the domain network.
From this we found on that day, at 20:16:18 (GMT +6), some firewall rules were added after the attacker created the domain admin user. One of these inbound firewall rules, FDR3340, was created for the Exchange 2016 system (Figure 22).
This rule action allows Exchange 2016 to accept any TCP connection for the TCP port 3340.
Some of you may ask ‘how does the attack maintain access?’. We found that they have also created the following port forwarding rule (Figure 24) in the Exchange system to do this.
As the DC IP is only internally routable, the attacker performs simple port forwarding tricks in Exchange 2016 (as Exchange 2016 has an internal facing interface). If any TCP request comes to Exchange 2016 via port 3340, its port forward rule will redirect the request to the DC (10.10.10.1) RDP port. This way the attacker maintains persistent access to the entire domain.
No single solution will fully protect us.
In summary, the attacker performed the following task to control the FIN domain:
- They first found the CVE-2021-26855, CVE-2021-26857, CVE-2021-26858, CVE-2021-27065 vulnerabilities in the Exchange system to exploit.
- After gaining initial access to the Exchange system, they performed simple ping/arp queries to detect their next lateral movement target and found that the DC is connected.
- They then ran the post-exploitation credential harvesting tool, Mimikatz, and successfully cracked a non-privileged user of the DC system.
- They exploited the DC system’s vulnerability (PrintNightmare), which allowed them to create an admin privilege user in the DC system. The attacker then simply added this newly created admin user to the DC Admin group. This enabled the attacker to have full access to the entire domain.
- For maintaining persistent access the attacker then created a simple port forwarding rule as the DC system was not directly connected to global Internet. If any TCP request came to Exchange TCP port 3340 it would redirect to the DC system’s RDP/3389 port.
Critical vulnerabilities need to be identified, prioritized, and patched in an expedient manner. We need to employ the principle of least privilege wherever we can. No single solution will fully protect us.
Threat actors often employ sophisticated malware to alter registry files and other critical data on enterprise servers and workstations for malicious purposes. With privileged access, they can compromise vital business data and other sensitive information.
We should consider file integrity monitoring as an important layer of defence against cyber threats/malware/registry changes. We need multiple layers of security working together, all the time, in addition to constant network monitoring.
Debashis Pal is an Information Security Specialist from Bangladesh.
The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.