Disclaimer: I have no intention of detailing how the box was exploited, with a map of how to break this system again. I intend to show the methods used to discover and trace the breach throughout the server.
Welcome to part 2 of 3 of my Forensics Analysis blog.
At this point in the process, one or several indicators of compromise have triggered. Some commands have been executed, as mentioned in part 1 of this series, to gain a better understanding of what has occurred and obtain information which may be lost as a part of this analysis. This blog post details how to clone the server suspected of being compromised. How to perform the deep dive analysis on the copy system will be detailed in the concluding segment of this blog series.
Cloning a Compromised System for In-Depth Analysis
The purpose of this forensic analysis is to determine what has happened and the extent of the damage. If a compromise is confirmed, the end goal is to prevent similar incidents in the future by hardening the server or closing security gaps. The best method of accomplishing this is to take a bit-by-bit copy of the server for analysis. In some cases, an exploit may reside in unexpected locations within memory or swap space.
Depending on the environment, and data stored on it, there may be regulations instructing how to handle this process to ensure a proper chain of custody. A minimum guideline is to take and compare hash values of the partitions to ensure the files have not changed or been tampered with.
For servers in the AWS cloud, Patrick Olsen provides an excellent procedure to use for preserving the compromised server and creating a clone on which to perform analysis.
http://sysforensics.org/2014/10/forensics-in-the-amazon-cloud-ec2/
This follows the basic process of imaging the server, cloning it, and mounting it onto another system as read only. Now the mounted drive is ready for analysis.
For a Linux systems not in the cloud, begin by identifying the disks in use and their starting and ending locations. To limit the impact of these commands, do not use interactive mode.
fdisk -l
This will list all of the partitions, such as /dev/sda1 and their characteristics.
To only display the partition name, use the line:
fdisk -l /dev/sda | grep '^/dev' | cut -d ' ' -f1
This list must be iterated through to obtain the full copies for every partition. Repeat the below commands for every partition on the server.
md5sum /dev/sda1 > /mnt/target/suspect.md5dd if=/dev/sda1 of=/dev/<output storage location> bs=8k mt bsf 1 dd if=/dev/<output storage location> | md5sum
The above script will take a hash of the target drive, write a copy to a new location, moves the position of the storage tape, and generates a hash value for comparison. Be very careful with these steps as confusing the input file (if), with output file (of) will overwrite any stored data. Alternative is to use dcfldd to combine the above steps into a single command.
dcfldd if=/dev/<partition name> hash=md5 of=/media/<output storage location>.dd bs=<byte size> noerror
I recommend using the tool dcfldd as it combines multiple functions with a single command. This will create a disk image at the output directory location, generate a hash value of the partition to be used as a comparison later, and write 0’s where any error occurs. To help with the transfer, the byte size value of 512 should be used. The output disk image may be later mounted and accessed for analysis. At that time, compare the hash values as it doesn’t make sense to perform analysis on the copied drive if it cannot be proven that it is a clone of the original partition. These commands need to be repeated for every partition. Since this should already be scripted into a series of commands, why not have the script iterate through the list of partitions, assuming they follow a standard naming convention.
for partitionName in $( fdisk -l /dev/sda | grep '^/dev' | cut -d ' ' -f1 ); do md5sum $partitionName /mnt/target/suspect$partitionNumber.md5 dd if=$partitionName of=/dev/<output storage location>$partitionNumber bs=8k mt bsf 1 dd if=$partitionName | md5sum done
Or, for those who prefer dcfldd:
for partitionName in $( fdisk -l /dev/sda | grep '^/dev' | cut -d ' ' -f1 ); do dcfldd if=$partitionName hash=md5 of=/media/<output storage location<.dd bs=<byte size> noerror done
For a Windows machine, there are several tools available to create a cloned image for analysis. Some of the more popular free tools are Arsenal Image Mounter, EnCase Forensic Imager, and DumpIt. These tools are easier to ensure an entire disk gets copied than running similar commands from a Windows command line. That being said, PowerShell has some great commands to help with forensics and cloning partitions. There is a great PowerShell module by Justin Rich, called NimblePowerShell. It can be found at https://github.com/jrich523/NimblePowerShell. This lists the volumes on the system, creates a snapshot, clones the volume, and mounts the clone.
Now that an initial analysis has begun, and a cloned environment is setup, the full system analysis can begin. This will delve into files, timestamps, and logs to try to determine whether the server was compromised, and if so, how it was exploited. Obviously, the better the logging on the server, the more reliable the results. This combined with the analysis of the volatile aspects on the server should provide a complete picture of what exactly occurred on the server. So, join me on the concluding blog post to learn useful places to search in a server and what commands can be run to expose sources of a compromise.
