It all starts with a [SPAM] email

By on 29 Oct 2019

Category: Tech matters

Tags: , , , ,

Blog home

SPAM (or more specifically phishing) email has become one of the most popular and effective weapons used by cyber attackers. As such, it can be a useful artefact for security enthusiasts to analyze.

There are some cool commercial tools on the market to do this, but as I’m about to show you, it’s easy to set up your own analysis platform with open source tools.

What you’ll need

In this post, I’m going to explain how you can build a Security Orchestration Automation and Response (SOAR) platform from scratch, for free. You’ll need:

TheHive, a scalable, open source and free security incident response platform, designed to make life easier for SOCs, CSIRTs, CERTs and any information security practitioner dealing with security incidents that need to be investigated and acted upon swiftly. 

Cortex, a powerful observable analysis and active response engine.

imap2thehive, a python script for collecting SPAM email from an IMAP server. Thanks to Xavier Mertens aka ‘xme’ for writing the wonderful script.

Using these, I’ll show you how to get a SPAM email from a predefined mailbox using a python script and create a case in TheHive automatically. Based on the email observables, you can do some investigation/analysis using Cortex and respond accordingly. In the following examples, I’ve used an Ubuntu 18.04 LTS server with 8GB of RAM and 200GB of disk space.

Step 1: Installing Elasticsearch

TheHive uses Elasticsearch for storing its data, so it is mandatory for building TheHive system. Follow the procedure below to install Elasticsearch:

sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-key D88E42B4

echo "deb https://artifacts.elastic.co/packages/5.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-5.x.list

sudo apt install apt-transport-https

sudo apt update && sudo apt install elasticsearch

Do not start the service now, we will do it later.

Step 2: Configuring Elasticsearch

After installation, you’ll need to complete the following configuration on Elasticsearch. Open the file elasticsearch.yml and paste the lines below at the end of the file:

sudo vim /etc/elasticsearch/elasticsearch.yml

network.host: 127.0.0.1
script.inline: true
cluster.name: hive
thread_pool.index.queue_size: 100000
thread_pool.search.queue_size: 100000
thread_pool.bulk.queue_size: 100000

Step 3: Installing TheHive

It’s time to install TheHive in your server:

echo 'deb https://dl.bintray.com/thehive-project/debian-stable any main' | sudo tee -a /etc/apt/sources.list.d/thehive-project.list

curl https://raw.githubusercontent.com/thehive-Project/thehive/master/PGP-PUBLIC-KEY | sudo apt-key add -

sudo apt-get update

sudo apt-get install thehive

TheHive will not start after the installation. We will start it later.

Step 4: Installing Cortex

Perform the following steps to install Cortex on your server:

echo 'deb https://dl.bintray.com/thehive-project/debian-stable any main' | sudo tee -a /etc/apt/sources.list.d/thehive-project.list

curl https://raw.githubusercontent.com/thehive-Project/Cortex/master/PGP-PUBLIC-KEY | sudo apt-key add -

sudo apt-get update

sudo apt install cortex

Like TheHive, Cortex will not start after the installation. We will do it later.

Step 5: Configuring TheHive

The only required parameter in order to start TheHive is the key of the server (play.http.secret.key). This key is used to authenticate cookies that contain data. If TheHive runs in cluster mode, all instances must share the same key. Type the command below to generate the key:

cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 64 | head -n 1

A random number will be printed in your terminal. Copy that key and open the /etc/thehive/application.conf file with your favourite editor and add the following line at the bottom of the file.

play.http.secret.key="YOUR_SECRET_KEY"

Step 6: Configuring Cortex

The only required parameter to start Cortex is the key of the server (play.http.secret.key). This key is used to authenticate cookies that contain data. If Cortex runs in cluster mode, all instances must share the same key. Type the below command to generate the key:

cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 64 | head -n 1

A random number will be printed in your terminal. Copy that key and open the /etc/cortex/application.conffile with your favourite editor and add the following line at the bottom of the file.

play.http.secret.key="YOUR_SECRET_KEY"

Step 7: Starting services

Now that the TheHive and Cortex installation and configuration is complete, it’s time to start the service. Perform the steps below to enable and start the service of Elasticsearch, TheHive and Cortex.

sudo systemctl enable elasticsearch.service
sudo systemctl enable thehive.service
sudo systemctl enable cortex.service

sudo systemctl start elasticsearch.service
sudo systemctl start thehive.service
sudo systemctl start cortex.service

sudo systemctl status elasticsearch.service
sudo systemctl status thehive.service
sudo systemctl status cortex.service

It may take some time to start the services. By default TheHive will use port 9000 and Cortex will use port 9001. Wait until the ports come up.

Step 8: Logging in to TheHive

Now it’s time to log in to the TheHive. Open a browser and go to http://<YOUR_SERVER>:9000 to access the TheHive’s interface. For the first time you will see an image like the one below. 

Figure 1 — TheHive interface.

Click Update Database. It will redirect you to the admin user creation page, which looks like the below image.

Figure 2 — TheHive’s admin user creation page.

Now create an admin user, which we will use to login to TheHive. After creating the user, you will be redirected to the TheHive’s login page, which is similar to the image below.

Figure 3 — TheHive login page.

Enter your username and password and click Sign In. You will see a dashboard like the below image which means you have successfully installed and configured TheHive.

Figure 4 — TheHive’s dashboard.

Step 9: Logging in to Cortex

The process for logging into Cortex is similar to the TheHive, except for the port number.

Open a browser and go to http://<YOUR_SERVER>:9001 to access the Cortex interface. The first time you log in you’ll see an image like the one below. 

Figure 5 — Cortex interface.

Click Update Database. It will redirect you to the admin user creation page, which looks like the image below.

Figure 6 — Cortex’s admin user creation page.

Create an admin user, which you’ll use to login to Cortex. After creating the user, you will be redirected to the Cortex login page, which is similar to the image below.

Figure 7 — Cortex login page.

Enter your username and password and click Sign In. You will see a dashboard like the image below, which means you have successfully installed and configured Cortex.

Figure 8 — Cortex dashboard.

Step 10: Add organization to Cortex

The Cortex system is tenant-based, which means you can create organizations. The default organization is Cortex but we will create our own organization.

Click Add Organization and add your organization’s name and description. In my case, I named the organization training and description training organization

Figure 9 — Cortex’s organization creation page.

After creating the organization, it will show up in the organization window. Now let’s create users under this newly created organization.

Step 11: User management in Cortex

Cortex has four different rules for its users: read, analyze, orgAdmin and superAdmin.

Users who have the read role can only read the reports. Users who have an analyze role can read reports and run jobs. Those who have an orgadmin role can run jobs, view reports, and enable/disable the analyzer, but they cannot create another organization nor delete the organization. superAdmin can perform all activities except run jobs, view reports, enable/disable the analyzer and configure the analyzer.

Let’s create two users in Cortex with different roles. We will create one user with the read, analyze, and orgadmin roles enabled, which we will use for configuring the analyzer and the responders. For the other user, we’ll give them read, analyze roles, which will be used for integration with TheHive.

Click the training organization. On the left side of the screen you will see the Add user button. Click that button to create our first user with the read, analyze and orgadmin roles selected.

Figure 10 — Adding a user to an organization in Cortex.

After entering all the information click the Save user button. Like the first one, let’s create the second user but this time we will only select the read, analyze role.

Figure 11 — Creating a second user in Cortex.

Let’s log out from the system and login again with the orgadmin role user. In my case the username is imtiaz.

Step 12: Logging in to Cortex with orgadmin role user

After logging in with the orgadmin role user you will notice that the dashboard is different, as in the image below:

Figure 12 — Cortex dashboard view for orgadmin user.

It has different tabs like Job history, Analyzer, Responders, Organization. Click the Organization tab. Under this tab, click the Analyzers and Responders tab. The list will be empty now. We will load the analyzer and the responders.

Step 13: Running analyzer and responders

To run the analyzer, you will need to install the following software:

sudo apt-get install -y --no-install-recommends python-pip python2.7-dev python3-pip python3-dev ssdeep libfuzzy-dev libfuzzy2 libimage-exiftool-perl libmagic1 build-essential git libssl-dev

sudo pip install -U pip setuptools && sudo pip3 install -U pip setuptools

Download the repository below for Cortex analyzer and responders.

cd /opt

sudo git clone https://github.com/TheHive-Project/Cortex-Analyzers.git

Each analyzer and responder has some dependencies, which we can install from the requirements.txt file. Instead of installing them one by one you can use the command below to install them all.

for I in $(find Cortex-Analyzers -name 'requirements.txt'); do sudo -H pip2 install -r $I; done && \
for I in $(find Cortex-Analyzers -name 'requirements.txt'); do sudo -H pip3 install -r $I || true; done

It will take some time. Wait for the installation to finish.

After the installation, edit the /etc/cortex/application.conf file. Open the file and search for the word analyzer. You need to update the location of the analyzer. Edit the file and change the path value under analyzer (as below):

analyzer {
   # Absolute path where you have pulled the Cortex-Analyzers repository.
   path = ["/opt/Cortex-Analyzers/analyzers"]
   fork-join-executor {
 # Min number of threads available for analysis.
   parallelism-min = 2
 # Parallelism (threads) … ceil(available processors * factor).
   parallelism-factor = 2.0
 # Max number of threads available for analysis.
   parallelism-max = 4
  }
 }

Restart the Cortex service. 

sudo systemctl restart cortex.service

Be patient — Cortex will take some time to restart the service. After the service comes up, log in with the orgadmin role user and go to the Organization->Analyzers tab. You will see all your analyzers in a list similar to that shown below:

Figure 13 — Cortex Analyzers list.

Go back to your terminal and open the /etc/cortex/application.conffile. Paste the following lines after the analyzer section.

responder {
   # Absolute path where you have pulled the Cortex-Analyzers repository.
   path = ["/opt/Cortex-Analyzers/responders"]
   fork-join-executor {
 # Min number of threads available for analysis.
   parallelism-min = 2
 # Parallelism (threads) … ceil(available processors * factor).
   parallelism-factor = 2.0
 # Max number of threads available for analysis.
   parallelism-max = 4
  }
 }

Restart the Cortex service. 

sudo systemctl restart cortex.service

Again, be patient. Cortex will take some time to restart the service. After the service comes up, log in with the orgadmin role user and go to the Organization->Responders tab. You will see all your responders in the list similar to that shown below:

Figure 14 — Cortex Responders tab.

Step 14: The integration

Login to Cortex with an orgadmin role user and go to the Organization tab. You will see a page like the one shown below:

Figure 15 — Cortex Organization tab.

Remember we created a user called thehive earlier. We will use this user for doing the integration. Click the Create API Key button for thehive user. Your API key will be generated. You will now have three buttons beside your username (as below). 

Figure 16 — Creating the API key.

Click the Reveal button. Your API key will be displayed. Copy the API key.

Figure 17 — Copying the API key.

Go back to your server terminal and open /etc/thehive/application.conf file. Search for the text ‘cortex’ and uncomment the below line.

play.modules.enabled += connectors.cortex.CortexConnector

Change your Cortex configuration like below:

cortex {
   "CORTEX-SERVER-1" {
     url = "http://YOUR_CORTEX_SERVER_IP:9001"
     key = "PASTE_YOUR_API_KEY_HERE"
   # # HTTP client configuration (SSL and proxy)
     ws {}
   }
 }

After making the change, restart TheHive service. Type the below command to restart the service:

sudo systemctl restart thehive.service

Log in to TheHive. Click your username (top right corner of your screen) — in my case it is theHive Admin — then click About. A box will appear and if everything is OK, you will see the version of Cortex along with the server name you just configured with a green OK displayed in the box similar to the image below.

Figure 18 — TheHive About window.

Step 15: Enabling analyzer in Cortex

After integration it’s time to test the full system. Before this we need to configure and enable analyzer in Cortex.

Login to Cortex as the orgadmin role user. Go to Organization and click Analyzer. You will see all installed/loaded analyzers available in your system. Some of them are paid, some of them are free, some of them have both. For this lab we will use AbuseIPDB and VirusTotal analyzers.

To enable the analyzer we need an API key. I’m not going to cover how to create the key but if you create an account on VirusTotal or AbuseIPDB you will get your API key for free.

Let’s enable the AbuseIPDB analyzer first.

Click Analyzers config and find the AbuseIPDB analyzer. By default it will be disabled.

Click Edit. A window will appear like the below image:

Figure 19 — AbuseIPDB config.

Insert your AbuseIPDB API key here and leave the days field as it is or you can choose your own value. Click Save.

Figure 20 — Editing AbuseIPDB config.

Go to the Analyzers tab and find the AbuseIPDB analyzer and click Enable. A window will appear where you can change the value or you can leave it with the default value. Click Save.

Figure 21 — Saving the config.

Your AbuseIPDB analyzer is now ready for analysis.

You can enable the other analyzer, VirusTotal in the same way. Provide the VirusTotal API key from Analyzer config and Enable the analyzer from the Analyzers tab. Enable both VirusTotal_GetReport_3_0 and VirusTotal_Scan_3_0.

Figure 22 — VirusTotal config.

Step 16: Analyzing from Cortex

You can now use Cortex to perform some analysis.

Click New Analysis from the left corner of your screen. A window will appear. Leave the TLP and PAP value as AMBER. Choose the datatype as ip. Provide an IP address, for example 8.8.8.8 (but you can choose other addresses also). The purpose of this analysis is to check whether the system is working or not. I leave the TLP and PAP value as AMBER (it is the default option) — you can change it if you like. Check the Analyzers that you want to perform the analysis.

Figure 23 — Run analysis window.

Click Start to start the analysis. You will see the analysis appear in the Job History tab.

Figure 24 — Job history view.

After completing the analysis, the Status column will show the success (or failure) status. Click View to see the detailed report of the analysis. 

Step 17: Testing the integration

So far we have set up our TheHive and Cortex systems. Using these, let’s test the integration by creating a case in TheHive and doing the analysis.

Before doing this, create a user with read, write permissions in TheHive. After creating the user, logout from your current user and log in again with the newly created user.

Let’s do the same analysis but this time from TheHive by creating a case. Click New Case and fill out the form as below:

Figure 25 — Creating a case in TheHive.

Click Create caseThe newly created case will be look like the image below:

Figure 26 — Newly created case.

Click Observables > Add observable(s). Here you can select multiple observables to analyze. Click the Type field and select your observables. For this case, I want to check the status/reputation of the IP address, so I choose ip as type, 8.8.8.8 as the value, and ip_address as the tag. You can choose a different address here if you like. 

Figure 27 — Creating new observables.

Click Create Observables. Your observables will appear like the image below. Select the observables and click Action, then choose Run analyzers.

Figure 28 — Observables list.

After that, the analyzer will appear. Select the analyzer you want to use for this analysis. I chose both analyzers.

Figure 29 — Selecting analyzers to run.

Click Run selected analyzers. After finishing the analysis you will see the status of the analysis as below.

Figure 30 — Analysis status.

You can see your analysis report regarding the IP address here. Click the red link to see the report. It will show you the raw JSON output.

For a better view, you can download the report template using this link. Log out from your user and login to TheHive as an admin user. After logging in go to Admin->Report templates and click import templates. After importing the templates, log out from the admin user and log back in with your user account again. Check the report again from the same location. 

The TheHive has a Twitter-like feed where you can see all your activity. You can find it on the right side of your screen. If you have a result in TheHive then the integration is working.

You can now go for the real fun part, analyzing SPAM email.

Step 18: Creating a case from a SPAM email

This is the fun part of this tutorial. Here we will create a case in TheHive from a SPAM email where, if the subject of the mail is [ALERT], an alert will be created in TheHive. Download the repository from this link. (Thanks to the author for writing this amazing python script.) After downloading the repository, open the file imap2thehive.confand change the [imap] and [thehive] sections according to your environment. You will need to generate an API key from your TheHive interface. Log in with admin and go to Admin > Users. From here you can create an API key for a specific user to use in this file.

[imap]
host: <YOUR_IMAP_SERVER_IP>
port: <PORT>
user: <MAIL_USER>
password: <PASSWORD>
folder:thehive
expunge: true
spam: (X-Spam-Flag: YES)

[thehive]
url: http://<YOUR_THEHIVE_SERVER>:9000
apikey: <YOUR API_KEY>
observables: true
whitelists: imap2thehive.whitelists

You can also change the tlp value from [alert] and [case] sections if you like. Now that the configuration is done, log in to your IMAP email account and create a directory called thehive (you can choose any name) like the below image. I use roundqube as my email client.

Figure 31 — Roundqube folder list.

Now, whenever you get a SPAM email in your mailbox or reported by other users of your domain, put that email in thehive folder and make it unread.

Next, go to your terminal and install the required dependency to run the python script. Go to your imap2thehive directory and run the command below:

sudo pip install -r requirements.txt

This will download the necessary package for running the scripts. Next, make the imap2thehive.py file executable. Run the script (as below):

sudo ./imap2thehive.py -c imap2thehive.conf

If it’s connected to your IMAP server it will display Connected to IMAP server and will start processing the unread mail in thehive folder. It will create a case from your email and create the observables like domain, IP, and URL (if any).

When you log in to your TheHive server, you will see a new case with the email subject created. Click the newly created case and go to observables. Check which one you want to analyze. Click Action > Run analyzers. Follow the Twitter-like feed for the progress of your analysis. If it’s successful you will see results like those shown in the image below.

Figure 32 — TheHive analysis results.

Step 19: Enabling a response

Now it’s time to enable responders. Like analyzers, you can enable responders in Cortex.

Responders are very similar to analyzer though they have a different purpose. You can also write your own responders. We will use the Mailer responder which, when customized for your environment, should allow you to send emails to specific users from TheHive’s interface.  Log in to Cortex with an orgadmin role user. Go to Organizations >Responders Config and click Mailer. A window will appear like the image below.

Figure 33 — Editing a mailer responder.

According to your environment, provide your smtp_host, smtp_port and from (sender) address. Click Save. Go to the Responders tab and find Mailer_1_0. Click Enable. A window will appear like the image below:

Figure 34 — Enabling Mailer_1_0.

Like before, fill out the fields according to your environment and click Save. Click the Refresh responders button. Our responder is now configured. Let’s look at the responder action from TheHive.

Log in to TheHive. I have a case here which is generated from a SPAM email with the subject line ‘Awaiting for your confirmation‘. This is a real-life email that I received a couple of years ago. Using the python script, I’ll  pull the mail from a predefined mailbox and make a case in the TheHive. 

Figure 35 — A case in TheHive, generated from a SPAM email.

When you click a case in TheHive, you will see a section called Description at the bottom. Here you will see the original mail message. In my case it looks like the image below.

Figure 36 — The original email.

Now the message is reported by a user who has received the mail. The user forwards the mail to our preconfigured mailbox and we pull that message and start our investigation. From this Description window you can draft your reply email (See the image below for an example). You will find a pen icon below the Description. Click that icon and it will allow you to create a draft. After making the changes click the green tick mark to save.

Figure 37 — The description window.

Now for the real trick: go to Tags and add tags like the ones below.

You will have the email tag there already, so just add the mail tag. Replace YOUR_RECIPIENT_EMAIL_ADDRESS with the address you want to reply from.

Figure 38 — Tags.

Go to Responders and click Mailer_1_0 and click Yes, run it.

Figure 39 — Responders tab.

Go to Responders Jobs at the bottom of your case. If you configure the responders correctly then you will see an image like the one below with a status of a cool green Success message. 

Figure 40 — Job list and ‘success’ message.

Check with your recipient, they should have received a message from your TheHive system. 

Happy Hunting

The goal of this post was to show you how to build a system for doing security analysis and investigation in an automated way. You can do multiple analyses with your observables including for URLs, domains, files from a single place and you can respond accordingly. Build your system in a secure place first before going to production and be careful about the observables, especially URLs and files. Happy hunting!

Adapted from original post which first appeared on Imtiaz Rahman’s Blog.

Rate this article

The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.

Leave a Reply

Your email address will not be published. Required fields are marked *

Please answer the math question * Time limit is exhausted. Please click the refresh button next to the equation below to reload the CAPTCHA (Note: your comment will not be deleted).

Top