Monitoring System Calls for Active Authentication with Detours

A variety of activity on a computer can be monitored in order to authenticate users, but at Coveros we chose to monitor system calls, or requests for service made to the operating system.  The motivation for this choice is that most of a user’s interactions with the desktop applications on a computer will result in one or more system calls.  Our hypothesis is that the sequence of calls generated by a user’s interaction with a computer can be used to uniquely identify them.

In order to collect system call traces we are leveraging Microsoft Detours, a C++ API that allows for the monitoring and interception of arbitrary Win32 functions, which we will refer to as system calls.  Microsoft Detours has both a free Express version and a Professional version that costs $9,999.95.  The primary advantages of the Professional over the Express version are that it supports 64-bit programs and all Windows processors.  However, if these types of support are not a concern for you, then the Express version is more than adequate.

In the remainder of this blog post I will explain how to install Detours and use it to monitor system calls and then describe how we are using it in the Active Authentication project.

Installing Microsoft Detours

In this tutorial I will assume that you have Microsoft Visual Studio with Visual C++ installed on your machine.  The Express edition of Visual Studio is available from (http://www.microsoft.com/visualstudio/en-us/products/2010-editions/express).  I will also assume that you have some experience using the Windows Command Prompt.  First I will walk you through the steps of downloading and installing Detours:

1. Start by downloading the installer for either the Express or Professional edition of Microsoft Detours from the following site: (http://research.microsoft.com/en-us/projects/detours/).  The Express version of Detours is sufficient for completing this tutorial.
2. Run the installer and follow the instructions. When prompted for the installation folder, choose a location where you have write permission. Otherwise, you will need to run the command prompt with administrative privileges in order to compile Detours.
3. Open a Visual Studio command prompt (this should be an available program after searching for “visual studio” in the Start Menu).  This starts a command prompt and modifies environment variables such that the build tools provided with Visual Studio can be used on the command line.
4. Change into the directory where you installed Microsoft Detours. Type “nmake” in order to compile the Detours libraries and all of the examples.

Monitoring System Calls

Now that you have installed Detours, you are ready to run an example program that collects the system call trace produced by an application.  The example program is called Traceapi and can be found in the samples\traceapi subdirectory of the Detours installation. This example prints tracing statements for 1401 Win32 API functions. The output from this trace is logged to the syelogd daemon, which is another sample program provided by Detours.  Here are the steps involved in getting this example to work:

1. First change into the samples\traceapi subdirectory of Detours.  Since all of the example programs were built with Detours, no further compilation steps are required.
2. Next type “nmake calc”.  This will first cause a separate command prompt with the syelogd monitoring daemon to appear, and then the calculator program will start.
3. You can now perform some calculations using this program and then close it.  Upon closing the program the daemon will close as well.
4. The API function call information can be found in the file “test.txt”.  Each line of this file contains the time at which the function was either called or the time at which it returned, the name of the function, its parameters, and its return value.

Modeling Behavior Using System Call Sequences

We leveraged the Traceapi example in order to create a prototype system that collects system calls for an application and uses these calls to authenticate users.  This system is currently only able to monitor one process at a time and can be run in either learning or authentication mode.  In learning mode the system collects system calls and simply stores them for later use in authentication mode.  Since it is unlikely that individual system calls correspond to meaningful behaviors, we store sequences of system calls.  For example, if an application produced the system calls

WriteFile, ReadFile, CreateFile, CreateFile, RegAddKey

then, for a sequence size of 3, our system would store the following sequences:

(WriteFile, ReadFile, CreateFile), (ReadFile, CreateFile, CreateFile), (CreateFile, CreateFile, RegAddKey)

In authentication mode the system compares these stored sequences of system calls with the live data from the running process in order to authenticate the user of that process.  If the sequences in the live stream are sufficiently different than the stored sequences, then the system registers that an anomalous event has taken place and asks the user to re-authenticate by entering their password.  In practice a variety of actions could be performed, such as notifying a systems administrator that the activity on a workstation differs from what is normal.

I will leave the explanation of exactly how the live stream of system calls is compared to the stored sequences for later posts.  I will also explain how Detours works and how it can be used to intercept and modify functions.  For more information about this project and the research that Coveros Labs is undertaking please visit: http://www.coveros.com/research/