Owen Gotimer
Hello everyone. My name is Owen Gotimer. I’m the community manager at TechWell. I’m joined today by Tom Stiehm the CTO at Coveros. Tom, thanks for joining me today.
Tom Stiehm
Thanks for having me.
Owen Gotimer
Another anti-pattern we can chat about is not having a production-like test environment. What are the symptoms of that, and some of the consequences you see from not having a production like test environment?
Tom Stiehm
So the idea is that any software that’s not tested in the environment it’s going to run in production, the best assumption is that it’s not going to work. So having a production-like test environment means that it, to some scale, it’s the same as production. So it’s the same OS, the same software, the same software configuration, the tiers are broken out the same way. What often happens is in development and in lower environments, you don’t traditionally configure things the same way as you do in production. So the developer on their development computer may have one server where they write and test all their code, and then they check that in all the way on the other side. In production it may be broken into four different servers that have to talk to each other across the network. In the development environment, the servers may be talking to each other through the like JVM or something. So they’re using in-process communication versus network communication in the production environment, because you’re distributing the different parts of the application and on different servers. And that in fact, happened in one of the projects I worked on where we had four distinct types of processing. In the development environments, they all ran on one developer’s laptop. They progressively broke out some in the testing environments, but there was none where all four were broken out like it was in production. So the first day we put this software in production, the software just didn’t work. And what we found out is that there were two types of processing that had to talk to each other a good deal, and everything that we processed had to use those two servers. And they had never been broken out into separate servers before; they’d always been on the same JVM. So naturally, that configuration was never tested. And in production, we found out by Murphy’s Law, anything that can go wrong, does. It didn’t work. And we spent a few days figuring that out and then fixing the code so it would work. And from that point on, one of the things we really focused on on that project was, let’s make sure we test in a production-like environment, because the amount of processing we were doing was large per every request. And we had to make sure that all these things broken out into different servers worked when the communication is over the network versus just over the in-process communication.
Owen Gotimer
So when you have situations where teams don’t already have production-like test environments, what are some initial steps you can take to move the team towards having those environments?
Tom Stiehm
One of the the positive things about virtualization and the cloud is that makes it easier to create production-like test environments. In the Java world, when I started programming, it was extremely common for all developers to work on Windows and all development environments to be Solaris or Linux. So you would go from I’m creating this on my laptop on Windows, and I’m putting in production in a Unix environment or Linux environment. There were always problems in doing that. So one of the things that we’ve recently done with the virtual machines, like Docker or something, is even though the developers’ laptops’ base-OS may be Windows, we can have a Linux virtual machine that is configured the same way that the production Linux servers are going to be configured. And that means that we’re actually running in this same environment. I’d like to go as far as to have all the same security patches and all the same things locked down. One of the common things to happen is that as you’re developing applications, you have some kind of network communication you have to do, and the developers will just pick a random port to do that network communication over. Of course, ust the way processes over a network communicate, the security people will tell them, we’re not going to open that port, we never will, you have to pick one of these other things for various security reasons. And because they never talked to the security people or they never reached an environment that’s locked down and will prevent someone from connecting that port until sometimes into production, they don’t know until they get into production, then all of a sudden, this doesn’t work again. So pulling production-like environment into the lower environments, and even as far as the developers’ laptops can really help with that. And we have a lot of technology that can make that easier for us to do now.
