Disclaimer: I have no intention of detailing how the box was exploited, with a map of how to break this system again. I intend to show the methods used to discover and trace the breach throughout the server.
Welcome to part 3 of 3 of my Forensics Analysis blog.
Following the first 2 parts of this series, there have been several indicators of compromise that have triggered the initiation of the forensic analysis process. The suspected server has been cloned and mounted. Now the thorough analysis of the compromise may begin.
Beginnings In-Depth Analysis: Changes to Users or Sytem
The purpose of analyzing the cloned system is to determine whether a compromise has truly occurred, and if so, the extent of the damage. If a compromise is confirmed, the end goal is to prevent similar incidents in the future by hardening the server or closing security gaps.
Below lists a basic guideline to follow for a repeatable process to analyze the system.
Verify the kernel, programs, and shared libraries
Check boot services and cron jobs
Review password files, access logs, file modifications, and permissions
To do this, review password files for modifications. These should be owned by root and any changes may help direct the analysis.
ls -la /etc/passwd ls -la /etc/shadow
Often, a malicious script will be used to add or modify the users on a device. Changes within the passwd or shadow file may highlight accounts being used to introduce or further the compromise.
Within the password file, look for unexpected accounts with root privileges by implementing specific filters.
sort –nk3 –t: /etc/paswd | less egrep ‘:0:’ /etc/passwd
Other commands may identify unexpected account usage:
lastlog
Lists every user and the last time their account was accessed. The “last” command was mentioned within the first blog post of this series. However, if this command has not been run yet, it is a good idea to do so. Much of the value may have been lost or overwritten at this point, though.
last -d -a -f /var/log/wtmp
Use this command to view system reboots, login/logout information and any resolvable IP addresses from remote users
-d -> Resolves the IP address of a remote host who has logged in into its hostname
-a -> Displays the results of the –d tag in the rightmost column
By combining these tags, it is possible to associate a user with the most recent IP address using it and easily see if a blacklisted IP has gained access to the system.
psof
Lists all open files.
Hint: if a file called *.exe is found open on a linux system, chances are good it is malicious. The .exe extension is for executable files within Windows and cannot be used in linux. This is similar to finding a reference to a shell script in Windows. It is possible to run with a tool like Cygwin, but at the least such a file is suspicious.
It may also be helpful to run this command and verify that the information matches what is expected:
last -f /var/log/utmp
The utmp file details logins/logouts, system events, uptime, and the current system status.
Log Reviews
Look through log files generated by any running services. Tracing events and building a timeline are the primary goals of logging in the first place. It’s about time these were used for reasons other than debugginig.
Starting with failed login attempts, the search began with known bad or suspicious events around the timeline of the first identified event with this command:
grep sshd /var/log/auth.log | grep “Invalid” | less
There was no guest account on the machine. Therefore any attempt to login as guest was assumed to be malicious and immediately raised suspicions. Note that the IP address which was used for the failed login has been removed from the image.
The website, http://whatismyipaddress.com/blacklist-check, links to several anti-spam databases and can be used to verify whether a given IP address is known to generate unwanted activities.
Using this site, the IP address found in the logs was identified as a known spam vector. At this point in the analysis it is uncertain whether this was an effective compromise or harmless, but annoying, spam.
In reviewing that log file, additional events of interest were found.
Similar to the guest login attempts, when no login identification is provided, it also indicates malicious activity. Not surprisingly, these IP addresses were found to be blacklisted as known malicious, and likely bots. These IP addresses were noted and used to tie events together across logs.
Using the data pulled from the login logs, I was able to tie events together within the apache logs. The IP addresses flagged as malicious were found in the apache logs making anomalous requests. Specifically, events in the logs contained no associated user. Where most lines included “- username -“, there were a few of the form “- -“. The command was adjusted to include only the outlier login attempts to find other instances and expand the list of malicious IPs.
cat /var/log/apache2/access.log | grep "\- \-”
There were still some legitimate instances, which were removed based on expected processes with the grep –v command.
cat /var/log/apache2/access.log | grep "\- \-" | grep -v <Private IP Address>
This found requests of the form:
<Malicious IP address found on blacklist> – – [25/Mar/2016:09:53:03 -0400] “GET /manager/html HTTP/1.1” 404 272 “-” “Mozilla/3.0 (compatible; Indy Library)”
This and other requests were attempting to access scripts hosted by the server or take advantage of php injection exploits. This appears to be the actions of a bot or simple exploit attempts in that the requests repeatedly attempted to gain access or escalate privileges. Many of these attempts did not apply to the system and were ignored.
The important step in performing forensic analysis is to filter out unsuccessful malicious activity from legitimate exploits. Events that should raise concern are exploits for services to which the system is vulnerable or malicious attempts that are not immediately rejected.
Other log locations that should be reviewed are:
cat /var/log/syslog cat log/kern.log
syslog is a standard logger, which records messages from a variety of programs and services
kern.log captures messaging by the kernel at any log level
Searching for Changes in Files
After reviewing logs, search the entire file system for changes to properties near the timeframe of the suspected incident or specific events of interest.
find . -type f -newerat <starting time before incident – YYYY-MM-DD> ! -newerat <ending time after incident> >> ~/accessedFiles.log
This command searches the current directory and subdirectories for any newly created files between the specified start and end times. The results are written to a log file.
It may also be helpful to begin the above command with “find /” to search through all directories on the server.
Two other commands to use instead of newerat are:
newermt – lists files modified
newerct – lists files with changed permission
The search criteria for starting and ending times should be set based on the analysis. Searching through the entire filesystem will take time, so it is best to use as focused search as possible.
If time can be dedicated to lengthy and exhaustive searches, expand the starting time for the above commands. Often intruders persist inside of the server and leverage footholds over time before being identified. Therefore, it may be necessary to review changes to the file system long before the first identified incident to find the original point of entry into the system.
One piece of evidence will lead to the next until a clear series of events can be created. This can only happen with sufficient logging. It is easy to review logs and parse out unnecessary details. In some cases it may impossible to complete a forensic analysis and definitively identify the source of the compromise without proper logging.
Look in the tmp, mnt, and backup directories using ls to list all files and their owners.
ls -la /tmp ls -la /mnt ls -la /backup
This may identify unexpected files and indicate which user accounts have become compromised.
Other Helpful Forensic Commands
ps aux
ps – displays information on active processes
a – lists active processes for all users
u – displays the owner of each process
x – lists processes not attached to a terminal
cat ./.bash_history
This is not the best method to use, but reviewing the list of commands that have been executed may identify something unexpected.
cat /etc/crontab
Displays jobs scheduled within the cron table.
ls etc/cron.*
Lists files within the cron subdirectories.
find / -size +<Large File Size in Kilobytes>k –print
This searches for unusually large files. Depending on the file system and the content being served, there may not be a legitimate reason for such large files to be present on the system.
These methods can help to identify the sources of a compromise and close security holes. Depending on the environment, the primary goal of forensics should be to ensure that future servers are kept clean. Depending on the It is not to point blame or bring the case to trial.
There are many tools to help with the process of analyzing the cloned server. Sleuthkit, Autopsy, and chkrootkit are all useful in identifying recent changes to the system which may be the result of the compromise.
