CEO Jeff Payne discusses best practices for protecting your software delivery process from supply chain attacks with Business Development Manager Kim Trott.
Kim Trott: Good day everyone. Today we’re going to be talking about software supply chain attacks with Coveros founder and CEO Jeff Payne. Jeff has been in the application security industry for nearly 30 years and is a recognized leader in application security and DevSecOps.
What is a Supply Chain?
Kim Trott: Let’s jump right in. There’s been a lot of press recently about supply chain attacks. Can you clarify for everyone exactly what a supply chain attack is?
Jeff Payne: Yeah, sure. But first, let me start with what a supply chain is. A supply chain is just a system of activities involved in the manufacturing, handling, distribution, and processing of any set of goods or services. As an example, if you’re building a car, after the process of getting the components manufactured, those components all sent to the assembly line, and then those things are all assembled into a car, and then that car is shipped out to dealers and other places. This is an example of a supply chain.
A supply chain attack is whenever anyone tries to compromise the supply chain—we also call them value chains. Someone infiltrates that set of activities with the intent of doing damage, either stopping the production of products or creating defective products or anything else that might be considered malicious to the end customer.
How does the concept of supply chain attacks apply to software?
Kim Trott: That’s very helpful. How does the concept of supply chain attacks apply to software? And should we be concerned?
Jeff Payne: We should definitely be concerned.
So today, a supply chain means a lot of different things. We’ve had supply chains forever but a recent example is the Colonial Pipeline shutdown that occurred. There are a lot of people are talking about that being a supply chain attack or supply chain issue, because here on the East Coast people couldn’t find gas and it was because a gas pipeline had been shut down due to somebody attacking the Colonial Pipeline organization and causing issues.
We have the same kinds of challenges and issues in software. If you think about the software development process, we we come up with ideas for products, we design those products, we implement those products, we test those products, and then ultimately, we send them out to customers. To us, that is a supply chain. It starts with an idea, and over time we build it into a product. We test and evaluate that product in different environments, different stages, and ultimately, we move it out into some kind of a production environment or system that our customers can use.
I think it was about a decade or so when I started hearing people talk about the software supply chain. At first I was like, “what the heck is that,” because in software, we always call it the software engineering process or the development process.
People in the security community started to talk about the fact that if you really want to damage and get into someone’s software, like let’s say Google, you could try to hack into Google and break into their system. Google has a world-class security organization, so it’s probably going to be pretty hard to break into them. What if, instead, you went to the manufacturers and the producers of software Google uses, and you figured out a way to compromise their software that you knew Google was going to get. Ultimately, it would let you get in and access things you weren’t supposed to, by attacking their suppliers and other pieces of the supply chain instead of the end customer, in this case Google. And that’s the concern that we have about supply chain attacks in software.
Now, a second point there is that the issues we have here are getting worse and worse. Because in our traditional physical supply chains, things like pipelines and automobiles supply chains, we are using more and more digitally-controlled devices and digitally-controlled processes. So our entire supply chains are becoming all digitally-controlled. They’re all software, right? That means they’re more susceptible to attack, not just physical attacks like someone blows up a warehouse or something like that, but maybe shuts down the system through some sort of a cyber attack. And that’s what is going to get worse and worse, as more and more systems get hooked together with Internet of Things and other challenges. It’s getting worse and worse.
…our entire supply chains are becoming all digitally-controlled. They’re all software, right? That means they’re more susceptible to attack…
Because people are realizing, like I mentioned with Google, it’s a lot easier to attack Google by attacking the people that provide Google their software than it is necessarily to attack Google themselves. So we need to watch out for those kinds of things.
One last example that many people have heard about is the recent Solar Winds supply chain attack. This was all around some network management software that a company built and distributed. They had a lot of very large customers, both commercial customers and federal customers, and somebody figured out how to put malicious code into their application as it was being built.
In software built means to take the code that developers wrote, and compile it into a language the machine understands how to run. That’s called building, right? Somebody figured out as that process was going, how to insert some code that would let them basically get into anybody’s network that deployed the software and use it when it got pushed out to customers. They ended up breaking into all sorts of places and stealing all sorts of data.
Best Practices for Protecting Yourself from Supply Chain Attacks
Kim Trott: What are some best practices for protecting yourself or your delivery process from supply chain attacks?
Jeff Payne: That’s a great question. It’s something we’ve been talking about for a decade at Coveros. To us, this is a fundamental part of your software development process. Certainly, it has gotten a lot more visibility, and there’s a lot more risk, because now we’re using DevOps and continuous integration, continuous delivery, and all about automating this supply chain process. We have to make sure that the the environments and the systems that we use—while automating that process of moving code from development through testing environments and into production—isn’t compromised, like it was with Solar Winds.
There’s no perfect answer, but I have some advice for the audience.
Protect All of Your Environments
The first is you’ve got to protect these environments that are used to build our software, implement our software, test our software, and then ultimately push our software out into production—whether these are cloud-based environments, or they’re machines in your office, or on premise of the data center. Wherever they are, you’ve got to make sure they’re protected. You can’t let anybody compromise those systems, because if they do, they could add malicious code into the process and that’s going to be a big problem.
Follow the Principle of Least Privilege
The second thing is, we have this concept of the principle of least privilege that means if somebody needs access to a system to do some job, we want to give them that access, but we don’t want to give them access to anything more than they need. A lot of times we get a little sloppy, and we give people pretty general administrative privileges or other kinds of escalated privileges on a system that gives them access to a whole bunch of stuff that’s not part of their job. Of course, if somebody breaks in using their credentials—they figure out how to compromise their password or whatever—then the hacker has all this opportunity to traipse through and do things in the system that really never should have been given access to the person in the first place. So we have to think more strategically about how we design our system.
Manage Secrets Very Carefully
The third thing is that we should not be lax when we secure these systems. There’s going to be all sorts of what in security call secrets laying around—your login ID, your password, encryption keys that allow you to encrypt and decrypt sensitive data. All that stuff needs to be protected because, you know, it’s great to have a lock on your house, but if you leave the key under the front mat, you didn’t really protect your house, right? Anybody could lift the mat up, find the key and open the door. So the in software, it’s the same thing. We’ve got to protect these keys somewhere and keep them protected at all time. We’re gonna need them to log into systems, we’re going to need them to decrypt information, we’re gonna have to make sure that we’re managing them very carefully. So we never let anybody spy or watch or try to access them or get access to those secrets, because once they get access to the secrets, all bets are off in terms of what they can do.
Analyze Third-Party and Partner Components
And the last thing I would mention Is that in your software supply chain, you probably don’t build all the software yourself. You probably use open source libraries, open source frameworks, you buy third party components from other companies, and you hook to third party API’s to other services. We use a lot of other partners or providers as we build software, and we have to make sure what they give us is secure. Because, again, the way we could be compromised in the supply chain is if one of our partners gets compromised, and maliciousness comes along for the ride right into our organization. So analyzing third party components, analyzing cloud services, analyzing open source frameworks, third-party applications and libraries we’re getting from others. If security is important, you’ve got to make sure those are secure, too.
Case Study: Securing a Software Supply Chain
Kim Trott: Coveros helps organizations improve their application security posture and DevSecOps capabilities. Can you talk a little about or give us an example of how we’ve helped organizations secure their software supply chain?
Jeff Payne: Yeah, yeah, we do. And as mentioned, we’ve always considered this part of our security solution when we’re helping people with application security—or today they call it DevSecOps. In securing your supply chain, there’s a couple of things that we do. Certainly, you want to use tools, there’s technologies and tools out there to manage secrets to encrypt and decrypt things and do other things to protect yourself. But in our experience, tools aren’t sufficient. If you don’t know how to use those tools properly, you’re gonna have problems. If it’s not designed into the overall system properly, you’re going to have problems as well. And so we usually start at the top.
Recently, we’re working with a large technology firm that produces software that mission-critical and security-critical organizations use in their infrastructure. So they’re really worried about security. We were brought in to put in place an end-to-end DevOps process for them, and increase the agility and flexibility of what they had in place already. As part of that, we were bringing in things like containers, Docker, and orchestration capabilities like Kubernetes, and other pretty advanced features, used in increasing flexibility and scalability and elasticity of environments and production systems.
Their security group was very worried about this, because all this was going to be automated. They had all their stuff locked down, it was all in their own data center. And all of a sudden, we were coming in telling them there’s this new way of doing it all, and it’s going to help you from a business perspective. They were like, “Well, how are we going to make sure this is secure?”
Read More—Request a Case Study
So the first thing we did when we started was sat down with them and we laid out a security architecture for them. We took the time to say, “Look, this is how this is all gonna work, how we’re going to automate your supply chain, your software supply chain, but here’s how we’re going to make sure every step, you’re still getting the security you need, the compliance you need.”
Based on that architecture, we implemented the various security capabilities and features—security management solutions to make sure we kept secrets secure, integrated security monitoring inside of containers so we could monitor and evaluate security as code set into the containers that were used throughout the process. And we looked at the open source components that they were using in their applications to make sure they didn’t have known vulnerabilities.
Now, that was just the first step. There’s other things now that are being done in that organization to take security to the next level, but that’s an example of the kinds of things that we typically do.
Kim Trott: That’s great info, Jeff. For anyone out there looking to assess how vulnerable they are to software supply chain attacks, please don’t hesitate to give us a call at Coveros. We have over a decade of experience addressing these issues, as well as overall AppSec issues and DevSecOps challenges. We’re here to help.
