Detecting Linux kernel process masquerading with command line forensics

Linux kernel process masquerading is sometimes used by malware to hide when it is running. Let’s go over how you can unmask a piece of Linux malware using this tactic.

What is Linux kernel process masquerading?

On Linux, the kernel has many threads created to help with system tasks. These threads can be for scheduling, disk I/O, and so forth. 

When you use a standard process listing command, such as ps, these threads will show up as having [brackets] around them to denote that they are threads of some kind. Ordinary processes will not normally show up with [brackets] around them in the ps listing. The brackets denote that the process has no command-line arguments, which usually means it was spawned as a thread.

For example, the below listing shows kernel threads vs. normal processes:

ps –auxww

What does it look like?

Linux malware uses a variety of techniques to hide from detection. 

One method they will use is to try to impersonate a kernel thread by making the process show [brackets] around its name in the ps listing. Administrators can easily overlook a malicious process this way.

If you look at the listing below, we have started a process to hide itself by trying to look like a kernel thread. Can you see it?

How to impersonate a Linux kernel thread

Now that you know what Linux kernel thread masquerading looks like, let’s set up a test so you can play with how to find it using command line forensics. 

We’ll use the sleep command for our simulation as you can do it on any system without fear of causing trouble:

export PATH=.:$PATH 
cp /bin/sleep /tmp/[kworkerd] 
cd /tmp 
"[kworkerd]" 3600 &

The export path sets things so we can execute the file in the local directory without needing to put a “./” in front of it. This makes it look more legit.

We next copy the sleep command over to /tmp and then run it under the bogus name [kworkerd]. We put on a value of 3,600 seconds to the sleep command so it will quietly exit after an hour once testing is over.

Let’s look at our handiwork and we should see [kworkerd] running when we do our ps command.

ps -auxww

De-cloaking Linux kernel thread masquerading with process maps

The first method we’ll use to de-cloak a masquerading process is to see if it has any contents under /proc/<PID>/maps.

This location is normally where processes show libraries they are linking to and where they mapped to in memory. For real kernel threads, it should be empty. If you look at this location for a process that is named in [brackets] but it shows any data, then it is not a real kernel thread.

The basic command we’ll use is cat /proc/<PID>/maps where <PID> is the process ID we are investigating. In the above example, we think that [kworkerd] looks suspicious with PID 2121 so we’ll check it out:

cat /proc/2121/maps

If you see anything listed under this area and the process has [brackets] around it, then it’s likely malicious and trying to hide.

If you want, you can run this command to quickly go over all the system PIDs and see which ones are named with brackets but have maps files. Normally you should see nothing here. Anything that shows data should be investigated further.

ps auxww | grep \\[ | awk '{print $2}' | xargs -I % sh -c 'echo PID: %; cat /proc/%/maps' 2> /dev/null

This command outputs the image below if it finds something.

In the /proc/<PID>/maps;listing you’ll see some paths to investigate where the binary has links to itself or libraries it is using. In the above, we see the path /tmp/[kworkerd] which would be a high priority location to investigate. You may also see libraries that are suspicious, references to hidden directories, and so forth. Take a close look at it and be sure you don’t miss anything!

De-cloaking Linux kernel thread masquerading with cryptographic hashing

Another way to de-cloak a masquerading Linux kernel thread is to see if it shows a binary attached to the running process. Basically, you just use the technique we discussed on recovering malicious binaries that are deleted, but see if you can get a SHA1 hash. If you get a hash back, then it’s a normal process trying to hide and is not a kernel thread. Real kernel threads won’t have a link to the binary that started them. This technique was suggested by @r00tkillah on Twitter when this subject was first posted.

A process binary on Linux can be quickly copied if you simply look at /proc/<PID>/exe. You can copy this file to a new location and have an instant snapshot of the binary that started the process. You can also use this link to get an instant hash to check against databases of known malware. Real kernel threads won’t have this data available, only imposters will.

In our case, we’ll use this knowledge to investigate our suspicious PID 2121 like this:

sha1sum /proc/2121/exe

Now we see the hash, let’s recover the binary and copy it somewhere so it can be analyzed offline. Using the command below we’ll make a copy to /tmp/suspicious_bin. Now, we have our own copy in case the malware tries to delete itself in self-defence:

cp /proc/2121/exe /tmp/suspicious_bin

If you want to automatically crawl through the PIDs and get SHA1 hashes of imposters, you can run this command:

ps auxww | grep \\[ | awk '{print $2}' | xargs -I % sh -c 'echo PID: %; sha1sum /proc/%/exe' 2> /dev/null

The above command will try to get a SHA1 hash of all processes with [brackets] around them. Any that return a hash are likely imposters:

Now you have two solid ways to use the Linux command line to investigate suspicious processes trying to masquerade as kernel threads.

Adapted from original post which appeared on Sandfly Security.

Craig Rowland is Founder and CEO of Sandfly Security.

Rate this article

---

The views expressed by the authors of this blog are their own and do not necessarily reflect the views of APNIC. Please note a Code of Conduct applies to this blog.