The Engineer Your DevOps webinar series emphasizes application of software engineering principles to build DevOps solutions that are are reliable, scalable and sustainable. On June 17, 2015, special guest Max Saperstone joined Jeffery Payne and Richard Mills on the topic “Integrating Automated Testing Into DevOps.”
DevOps is not a tool or a process but a culture shift that values communication, collaboration, integration, and automation across an organization. The DevOps pipeline, however, is a collection of tools that enable delivery of quality software with high confidence. We must find effective ways to test out this pipeline as it’s being built. Otherwise, it is too easy to end up with a pile of spaghetti code.
We study the Agile Testing Pyramid and take a look at the various stages. As we move up the pyramid, we ask the simple question: is it worth going to the next level?
In the commit stage, unit testing and component tests run fast, so we do lots of them.
Builds are immediate and some kind of assessment is required daily. When things fail we go back to the beginning and make fixes.
In the QA stage, we go through a series of quality gates in order to gain more confidence in code quality. We’re checking the commits to make sure they’re behaving as we expect them to.
Testing gets increasingly expensive as we progress into pre-production environments. We have bigger test cases that typically run end to end, where software is actually running in some kind of model of what a production installation might look like. These tests may be automated and take many hours or even a day. Some manual testing is often required.
Achieving Continuous Integration
To achieve continuous integration, developers are encouraged to commit code on a daily basis, if not every few hours. The point is to assure that nothing introduced during the day breaks functionality. Some functional tests are done weekly or at the end of each sprint. It’s advisable to try to slice off a small bit of performance and load testing even testing at a high-cycle rate to see if things are trending in right direction.
Achieving Continuous Delivery
We are continuously delivering software to somebody, somewhere. This can mean into production, to the Product Owner or to the QA team. The software is installed in an area that’s representative of production. To ensure that the software works properly in the environment, we test it under different conditions. Often it’s best to use a nighttime window so that builds are not getting held up during the day.
Don’t ignore manual testing
Manual tests are always part of a good testing strategy. Let computers do what they’re good at: repetition. Reserve humans to do the thinking. This is especially true when it comes to exploratory and usability testing.
Components of an automated test framework
Unit Testing. Unit tests are mostly written in an object-oriented style. However, there are other options. For instance, with Test Anything Protocol, individual tests can communicate with previous test results to support integration. Test Local Fixtures make sure that a particular environment is set up for a particular test dependent on a particular library list. Group fixtures take one unit test and run it over many different environments, providing increased coverage.
Data Management. There is no one, single solution for data management. If many people are we working with one database, it’s important not to step on each other’s toes. All tests require different data, dependent upon the particular system. Data generators can control the way the test is run for each and every input data. 
Multi-levels of Interfaces. Performing tests at different interfaces shows how they’re going to be interacting with the system. If the test steps are generalized enough, it is possible to write one test that can address multiple tiers of the application. The same tests can be formatted a bit differently for each interface.
Traceability. We need traceability to be able to debug as well as inventory what has taken place. When we can trace results back to requirements, it helps to build confidence in the automated testing.
Parameterization. Parameterized testing allows us to take one test and extrapolate on it. Rerunning tests using multiple different inputs expands the test base and provides more coverage.
DevOps Testing Tips
- Pay attention to see what data is actually needed for your automated testing – make sure you have enough.
- Analyze your production data and make sure that it can properly provide you with your use cases, your happy path and bad path testing.
- To minimize risk, production data must always be sanitized properly. Sanitize, sanitize, sanitize.
- When you have concurrent users, make sure that you’re distributing your tests so they’re not interfering with each other.
- Make sure that you have some sort of data rollback system.
- Use simple scripting built into the devops pipeline to get data set up for your load.
- Test towards different interfaces, including how the client application interacts with the back end through APIs. You want your testing to be representative of your workflow.
- Save steps in documentation by using screen shots in your handoff materials.
- Have tests populate the data for you.
- Throughout your approach to testing in a DevOps environment, set up appropriate feedback loops to provide quality information as soon as possible.
You can hear the entire recording here: https://attendee.gotowebinar.com/recording/4619797482962359810
Next Webinar August 4 – register now for Dashboards & Quality Gates: Making Your DevOps Visible
