Disclaimer: I have no intention of detailing how the box was exploited, with a map of how to break this system again. I intend to show the methods used to discover and trace the breach throughout the server.
Welcome to part 1 of 3 of my Forensics Analysis blog.
At some point in the security world, you are bound to come across a system that has been compromised. How do you recognize that malicious activities are taking place? Through common Indicators of Compromise or Events of Interest.
These include, but are not limited to:
- Slow processing speeds
- Strange or large files
- Unexpected communications (inbound/outbound)
- Alerts or email from downstream systems or ISPs
If a system is suspected of being compromised, the most important thing is to preserve the evidence. The gut reaction may be to pull the system offline immediately and shutdown or terminate the server. That is a mistake. Resetting the system wipes clean any volatile evidence stored in memory. In some advanced cases, the exploits may exist only in memory. This means a system admin may unwittingly assist a malicious user in removing any and all evidence by shutting down the compromised device. Granted, that is an extreme case, and is unlikely for most incidents.
As an alternative to shutting down the server or removing it entirely from a network, consider implementing strict access control rules for inbound and out bound communications. This may be managed via firewall rules.
As a general guideline, be sure to document everything from the point at which an indicator of compromise has been detected. Every command executed needs to be recorded. I cannot stress how important it is to document every action taken by a system admin or forensics analyst and any results of even seemingly trivial commands. This should be a limited list focused only on gathering volatile data and making a bit by bit copy of the potentially compromised system.
Once a system has been identified as compromised, run fdisk to identify all of the partitions and, if possible, make a binary copy of each one to preserve the evidence. Create MD5 hashes of each partition and use as comparisons for the bit-by-bit copies used for analysis. I will go into methods that may be applied to accomplish this during part 2 of this blog series.
After a copy has been created, a system admin should execute several quick commands and save the output to files. Try to avoid signing in as admin, root, guest, ubuntu, or user as these common accounts might have been accessed as part of the compromise and using them may overwrite the evidence. These commands are usually executed by a system admin as they can respond faster than a forensics analyst that may not be readily available, or currently hired. When the forensics analyst begins their processes, the compromised system should already be removed from the network. The system admin’s job is to preserve the evidence as best they can and provide support for a future investigation, which may be provided by an external 3rd party. It is helpful for a system admin to review the output of the following commands and look for suspicious or unexpected results. These can be marked for later analysis.
The list of commands to run on the compromised system varies by OS. At a minimum, these need to help the forensics analyst gain an understanding of how many disks are in use, when every user last accessed the system, running processes, and network connections.
Personally, I tend to run the following (for a Linux system):
- netstat –an
Displays the network connections and generic statistics
- ps –ef
Lists every currently running process
- top n 1 b
Displays running processes, their owners, how much memory each process is using, and how long the processes have been running. It also includes system information, such as uptime, number of users, memory usage, and the number of active tasks.
- dmesg
Outputs all kernel messages currently in the ring buffer
- who
Prints information on all current logged in users
- last
Lists all users who logged in or out since a file was created. By default, the file is /var/log/wtmp
- ls -lat <dir>
Search typical points of entry, where <dir> may be /dev, user home directories, and /root. Parse the results for suspicious files or files that are attempting to hide, such as files with names of “.”, “ “, or named close to expected system files. Consider file size and timestamps, looking for unreasonable results.
- lsof
Lists information about open files and can indicate which process is used it
- tail /etc/rc.d/rc.sysinit
Details how the system is initialized on boot and may indicate unexpected services or malicious programs
- ltrace
Search configuration files to try to find rootkits (ltrace –t /bin/ls)
For Windows, some of the basic equivalent PowerShell and cmd commands are:
dir, driverquery, regedit, netstat, netsh, wmic process, wmic startup list brief, Get-Process, and tasklist
I recommend getting familiar with these commands and their flags to tailor requests to provide the information that you want.
If it has not already been isolated, the system may be removed from the network. Removing a network connection can create issues or obscure evidence in the event a smart program is trying to cover its tracks. That is why I choose to run the initial commands above first.
With the snapshot and binary image in place, the compromised system may be shut down and the cloned system accessed for a more in-depth analysis.
Please join me for the next blog where I discuss methods of setting up a cloned system to perform the forensics analysis on.
