Application Security Review Process – A Case Study

What is application security, or AppSec? Let’s talk about web application security first. OWASP was created in 2001 and has been known as the best community for web application security. Volumes of online resources for web application security defects, security testing, and security projects have been produced by OWASP. Yet web application security is only one branch of the application security. There are other types of applications unrelated to web, i.e., batch process, deployment scripts, PowerShell and bash scripts, cronjobs, etc.

Despite the volumes of web pages from OWASP and other websites, there isn’t much discussion on the process of application security review. A few websites do have some high-level description of their application security review process, but they all miss the details. It is believed that many financial companies do have solid processes in places already.

This article is based upon my previous experience working as an application security engineer for Company X. When I joined Company X 6+ years ago, an application security review process had just been established. Over the past few years, the process had been refined many times. However, the basic process and workflow remained the same. In this article, I will present a simplified view of their application security review process so that we will not be bogged down by the nitty-gritty details.

Company X is an Information Technology (IT) company with about 1,000 employees and about 150 applications. Those applications include Java applications, Single Page Applications (SPA’s), router and protocol management, server management applications, custom Linux kernels, Distributed Denial Of Service (DDOS) applications, threat intelligence applications, network, and application monitoring applications, data warehouse applications, encryption key management applications, an Internet crawler, etc.

Company X Technical Background

Company X has multiple IT departments, which in turn have multiple development teams. The Information Security department, or InfoSec department, is managed by the Chief Information Security Officer (CISO), a VP of the company.

InfoSec Department

There are five teams in the InfoSec department:

• Governance, Risk and Compliance Team
• Incident Response Team 
• Asset Management Team
• InfoSec Engineering Team
• Application Engineering Team

Policies and Standards

The strength of Company X’s InfoSec practices can be witnessed by the number of InfoSec security policies and standards. They have more than 30 security policies and standards, and a few examples are listed here. 

• Architecture Standard
• TLS Standard
• Secure Java Coding Standard
• Secure C/C++ Coding Standard
• Password Standard
• Encryption and Key Management Standard
• Data Classification Standard

The Process

The application security process has three quality gates: Architecture Approval, Software Design Approval, and Deployment Approve (see Figure 1 below).

The application security process covers four distinct tasks: Architecture Review, Software Design Review, Code Review, and Security Scan, and they are all bundled into a single process flow. Architecture Review and Software Design Review have their respective quality gates. Code Review and Security Scan are combined to provide the quality gate for Deployment Approval.

Below we will go through each of the four tasks.

Initial Step –Architecture Review

When a new project starts, the application system architecture must be developed and submitted to the application security team for review. The development team cannot officially start software design and coding until the architecture is approved (see Figure 2 below).

The architecture design document must include the following components.

• A system architecture diagram that depicts the system components and the interaction among them.
• A table that lists all the required hardware equipment and their specifications.
• A table that lists all the major software components.
• A table that maps software components to hardware equipment.
• A table that indicates the network communication among all components. Specifically, the following five columns are required: source, destination, protocol, ports, and comments. This table will later become the request to open firewall rules. No firewall rule request, or Access Control List (ACL), can be approved if the system architecture is not approved.

Evaluation Criteria

The criteria the AppSec engineers used to evaluate the system architecture are based upon, but not limited to, the security policies and standards. Here are some of the criteria: (1) Internet-facing applications must have four tiers: load balancer, web servers, application servers, and database servers. The web server cannot have application code. It is only a pass-through server. (2) No HTTP traffic is allowed. Instead, HTTPS must be used. (3) All the application components are in a network silo. Any connections to servers outside of the silo are not allowed. (4) Communication initiated from servers in the outer layers to reach servers in the inner layers requires additional scrutiny.

Second Step: Software Design Review

Once the system architecture design is approved, the development team can start working on the software design and then submit the design for approval (see Figure 3 below).

A large set of design questions are documented on a questionnaire and is provided to the software design team to respond. Every software design review request must be submitted together with the approved architecture document. Major software design changes may need a new architecture review under the discretion of the AppSec team. In addition, the software design review must be performed annually, and the AppSec team has the authority to request a new software design review on an as-needed basis.

Evaluation Criteria

The AppSec team evaluates the responses to the software design questionnaire based upon the company policies and standards, i.e., log management standard, password standard, encryption standard, etc.

One example of the evaluation criteria is whether the application invokes the OS commands directly. This is documented in the architecture standard and is one of the questions on the questionnaire. If the response indicates OS commands are invoked through the code, the AppSec engineers will ask for justification. If reasonable justification is provided, the subsequent security scan will exploit the possibility of Remote Code Execution (RCE).

Third Step: Development and QA Deployment

Once the software design is approved, the development team can start the development process and QA deployment. The development team is free to deploy to the QA environment as many times as they want (see Figure 4 below).

For this step, the Application Security team is not involved at all.

Fourth Step: Production Deployment

Once the development team decides to deploy the code to production, an application security review request must be submitted. Only if the request is approved can the code be deployed to production (see Figure 5 below).

The submitted review request must answer the following six questions.

• Q1: Overview of Changes
  A link to the release notes should be included.
• Q2: Security Fix
  Does this release include any security fixes? If yes, details must be provided.
• Q3: Architecture and Data Flow Changes
  Does this release include any architecture or data flow changes? If yes, details must be provided.
• Q4: Sensitive Data
  Does this release include new sensitive data? If yes, details must be provided.
• Q5: Technology changes
  Does this release include significant technology changes, i.e., OS or database upgrade? If yes, details must be provided.
• Q6: UI changes
  Does this release include Web UI or API interface changes? If yes, details must be provided.

The development team must also submit the following information for code review and security scan. Notes: Code URLs must be submitted in the form of Git Diff URLs. If the code repository URL is provided but not the Git Diff URLs, the review request will be rejected.

• Source code
• Database code
• Any shell script code
• Configuration settings
• Deployment script
• Reports from the static code analysis
• QA servers URLs and server login credentials
• Two sets of application login credentials for every role used in the application

Manual Review Criteria
The AppSec engineers review the responses to the six questions and the set of required information listed above. If anything is missing, the review request will be rejected (to be moved to a wait queue). The answers to the six questions are also used to determine whether new architecture and software design will be required or not.

The next step is to review the reports from the static code analysis. Every high severity issue must be resolved or flagged as false positives.

The last step is to perform a manual code review, and the code review criteria are based on, but not limited to, the company’s policies and standards.

Security Scan Criteria
The criteria is simple. If there is any high severity vulnerability, the deployment will not be approved. For a brand new application, even a medium severity issue can be used to reject the deployment.

Two login credentials for every role must be provided. This is to exploit the possible cross-reference between different login credentials, and privilege escalation to more privileged accounts.

In addition, AppSec engineers will login to QA servers to review the system settings and the log files. Login to the web server and the database server may not be necessary unless there is a sign of vulnerability of the servers. Splunk is also used to review the log entries from QA and production environments.

For minor releases, the answers to the six questions will be used as the focal points for the security scan, which can be completed in a short timeframe. On the other hand, major releases may require a comprehensive scan.

One of the requirements for security scan is that the QA environment must be identical to production, if possible. This is to make sure production issues can be reproduced in the QA environment. Furthermore, AppSec engineers are prohibited to touch production applications – segregation of duties.

Kanban Board View

A different view of the process can be illustrated on the Kanban board (see Figure 6 below). The figure shows the happy path of the review process. The bottom of the figure is labeled with Operational Level Agreement (OLA), which is the time commitment between the application security team and the development team. The term OLA is derived from Service Level Agreement (SLA) commonly used as the time commitment between a service provider and a client.

Review requests with any missing information from the development team will be moved to a wait, or rejection, queue. Review requests with high severity vulnerabilities identified during the code review and security scan will also be moved to the wait queue (see Figure 7 below).

Conclusion

A salient feature of the above application security process is the first two quality gates: architecture approval and software design approval. Since the approval of architecture and software design must happen before the development process begins and for every major change, the above process successfully shifts the security to the left side.

It may appear that the AppSec team should have been more involved with the development team during the software development (the third step). This may not be necessary. The development team is supposed to review the security policies and standards as the guidance for developing secure code.

It is my opinion that the guidance documented in the security policies and standards may not be sufficient. Much of the guidance documented in the security policies and standards are presented as principles. Concrete security test cases can be much helpful for developers to develop secure code.

Have you noticed that threat modeling is not addressed during the application security review process? In fact, threat modeling was suggested to be incorporated into the process, but the suggestion was rejected. I did agree with the decision for two reasons. Threat modeling was supposed to happen during the early stage of software development, but most of the important applications in Company X were existing applications. It was hard to identify design decisions made years ago. More importantly, who would be responsible for developing threat models? We only had three AppSec engineers, and there were 150 applications. Providing security training on threat modeling to hundreds of developers seemed to be a daunting job. It is not an option either.

Another implication from the above process is the job skills required to perform the application security review. As far as I know, some large companies divide their application security team into three groups: Application Security Architecture Group, Static Code Review Group, and Security Scan Group (this includes penetration testers). These three groups map perfectly with the four tasks specified above.

While I was still with Company X, we specifically had two criteria for job candidates. (1) The job candidate must have programming experience. How could a non-programmer develop code remediation to vulnerable code, if they do not know how to write code? (2) For mid-senior level jobs, the candidate must know Burp Suite. Without the skill of Burp Suite, it is not clear whether the candidate knows how to exploit security vulnerabilities.

Finally, have you noticed that the AppSec team does not have a quality control program in place? One of the AppSec engineers repeatedly raise medium and low severity issues to high severity. This stopped the deployment of the applications and triggered fights among managers. Another AppSec engineer yielded to the demand of managers and approved review requests without completing the review tasks.

Company X also has other review processes in place, i.e., product review, third party software library review, and vendor review. But these probably deserve another blog or blogs to discuss.