Whatever process you use to deliver software, your ultimate goal is to determine if you have a viable candidate for production. Whether your process is completely automated, completely manual, or (most likely) somewhere in between, there are a lot of opportunities to add tools and tests to your process – your pipeline – to help you build confidence in your software.
And since you can’t do everything first, nor wait until the end and make everything last, you’ll have to arrange your process to make sure that you are using your time wisely at each stage.
Do just enough of each type of testing early in the pipeline to determine if further testing is justified.
Balance early, rapid feedback with the goal of avoiding any surprises later on.
Here are some things to consider and some tools to help you if you don’t already do something similar in your pipeline.
Check and reduce your dependencies
Fewer dependencies mean less maintenance. Use a dependency management system to make sure you are actually using all the dependencies you are including and exclude any that you aren’t actually using.
- Apache Maven (Java)
- Maven Dependency Plugin
mvn dependency:treemvn dependency:analyze
- Maven Dependency Versions Check Plugin
mvn com.ning.maven.plugins:maven-dependency-versions-check-plugin
Keep your dependencies up-to-date
Once you’ve pared down your list of dependencies, make sure the versions you are using don’t have known security vulnerabilities. If they do, update them or make sure you are protected some other way from becoming a target (e.g., web application firewall, input sanitization, disable functionality).
Use static analysis tools
Poor-quality code is harder to maintain than clean code. Static analysis tools can make sure your source code follows best practices, honors code style conventions, and can even check for some security flaws (e.g., SQL injection, race conditions).
- SonarQube (many languages)
- PHP Mess Detector (PHP)
- PMD (Java)
- FindBugs (Java)
- Find Security Bugs (Java)
- Pylint (Python)
Use mutation testing to test your unit tests
Just because you have unit tests (you do have unit tests, don’t you?), it doesn’t mean your code is tested. Code coverage tools can tell you which lines of source code were executed during the test run. Mutation testing can show you what lines are actually tested and what scenarios and conditions you haven’t tested for.
- PIT (Java)
- Ninja Turtles (.NET)
- VisualMutator (.NET)
- MutPy (Python)
- mutant (Ruby)
- Stryker (JavaScript)
Functional testing can do a lot more
Functional testing is important, but it can be used for even more than just checking your features. Use functional tests to do user-role testing to make sure users can only do what they are supposed to do and can only see what they are supposed to see, and make sure that the “have-nots” can’t. Proxy your functional tests, automated and manual, through a passive proxy like OWASP ZAP to look for cross-site scripting (XSS), cross-site request forgery (XSRF), and SQL injection vulnerabilities.
- Selenium
- Selenified
- OWASP Zed Attack Proxy
- Blog post: Running Selenium Tests Through ZAP
- Blog post: Scripting Authenticated Login within ZAP Vulnerability Scanner
Test external services
If you use external services, don’t just assume everything will always work. Test the APIs and see how the services behave when you send an incomplete or incorrectly-formed message. Make sure you know what to expect. And then mock those services for testing on your side so you aren’t dependent on them always being available. Plus you can create error conditions in your easier-to-control mocked environment that might be harder to induce in the real world.
Watch what your users are doing
Monitor your users, your application, and your logs to gain insight into how your system is being used, where users are struggling and when, what is going on in production.
- Piwik
- Google Analytics
- Splunk
- ELK Stack (Elasticsearch + Logstash + Kibana)
Make sure you have repeatable, reliable deployments
Continuous delivery is so dependent on automated deployment that many people assume they are one and the same. But even if your pipeline isn’t ready for continuous deployment yet, being able to stand up a new system the same way, every time, is more than just a time-saver. You can bring up systems for demos, long-running performance or security testing, or even to recover from system corruption or failures.
Keep your system packages up-to-date
Just like checking your dependencies, once you start deploying your software you want to make sure the other packages on the system are up-to-date. Do regular updates and upgrades so that you can deal with the changes incrementally and proactively. And keep on top of the security findings that might be exposing your system to attackers.
Protect all systems from hackers
In the cloud, there’s no such thing as “just a development or test system” that doesn’t need to be protected. Just because you don’t track corporate secrets or nuclear launch codes doesn’t mean that hackers won’t try to gain access to your system. Protect yourself from brute force attacks and unauthorized file modifications. Ensure you are using best practices for configuring your systems. And audit yourself, even if it won’t replace an official audit, just so you gain confidence that you won’t run into any late surprises when that auditor does come around.
Test the database, too
Do some checks to make sure the web front-end doesn’t let attackers directly into your database back-end. Run tests with real or realistic data and make sure that the data you retrieve and/or generate is right. Monitor your database logs to see what queries your applications run most often, and then develop a baseline performance profile for those queries for you’ll know if things are slowing down.
Check performance and security
Performance and security testing are often pushed to the end of the pipeline and sometimes ignored completely. Even if you don’t have time or resources to shift those tests to earlier in the process, you can use open-source tools and quick tests to do just enough testing to give you confidence that the later tests are likely to pass.
Plan for failure
Things will break. And you’ll never recover gracefully from a system crash if the first time you try is with users crying and management screaming in your ear, or vice-versa. Practice failures so that recovering becomes second nature.
Test your pipeline
Lastly, remember that your pipeline, and the tools and systems that support it, is how software gets delivered. Test your pipeline. Make changes to a development pipeline before breaking everyone by rolling those changes out to the pipeline everyone is relying on. Don’t prevent anyone from releasing because you updated Jenkins and now no builds with work. Do failure testing, disaster planning. This is critical infrastructure. If it goes down, the software delivery teams grind to a halt.
- Webinar: Why DevOps Engineering is Important
Key takeaways
I hope these lists of tools are helpful. But even more so, I hope you will consider if there are some additional tests and tools you can incorporate into your delivery process to help making it better.
- The pipeline is about building confidence that you have a viable candidate for production. Add tests with that in mind.
- Do just enough of each type of testing early in the pipeline to determine if further testing is justified. Early, rapid feedback balanced with no late surprises.
- Use mutation testing to make sure your unit tests are actually covering what you need to be covered.
- Some testing is better no testing. Even just watching trends of some token tests can be valuable.
- A little better is still better. Keep improving.
I was fortunate to be able to briefly discuss these tools, tests, and opportunities at AgileDC 2017 and then in more detail at TestBash Philadelphia 2017.
