Continuous Testing with Self-Healing: Why Is It Important?

In agile methodologies, applications are built rapidly and are subject to frequent changes during the initial development. In continuous testing, automated tests are executed as part of the software delivery pipeline to provide instant feedback on business risks associated with the software release candidate. Continuous testing is vital in DevOps to improve quality, reduce costs, and accelerate releases.

As continuous testing happens as part of the CI/CD pipeline, issues are identified early during the code development. It assists developers in determining when and how to release new changes. This article will explore how continuous testing with self-healing capabilities can take testing to the next level. One can cut downtime on repetitive maintenance of automation code and utilize this time for activities that provide more value to the project, such as increasing automation coverage, exploratory testing, etc. Owing to the benefits of continuous testing, market research firms are predicting that the continuous testing market will grow from $1.15 Billion in 2018 to $2.41 Billion by 2023, at a compound annual growth rate of nearly 16%. 

What are the factors for continuous testing success?

Over the last decade, many organizations moved to the DevOps delivery model. One of the primary purposes of this model is to ensure quality for every code check-in by adapting continuous integration, continuous deployment, and continuous testing (Figure 1). Some of the critical factors (Figure 2) that contribute to the success of continuous testing are stable automated end-to-end tests (Web/UI), i.e., automated tests that do not fail randomly, higher automation coverage, and tests for every code change in the application.

Continuous testing should include tests for every level, right from Component to API to UI, to achieve higher test coverage and quality. Tests performed from an end-to-end perspective provide a few significant benefits over other forms of testing. In contrast to other tests that only touch small parts of your system, these tests simulate how your customers would interact with your application in normal circumstances. Having this kind of coverage helps us ensure third-party services that we don’t control work as intended with our application.

Continuous Testing Challenges

One of the primary challenges of moving to a continuous testing approach is that automated tests are often flaky in nature and fail frequently. These unstable E2E automated tests inhibit continuous integration in the CI/CD pipeline. According to a recent study, around 74% of automated tests fail due to brittle locators on the web page. These flaky tests are caused mainly because of 3 issues:

• Frequent code changes related to the web/UI object locators such as ID, Name, XPath, CSS locator, class, tag, link text, or partial link text, etc.
• Element rendering: Wait time/sync up issues in the web page.
• Automation code issues.

The flaky test failures are categorized as false positives because there is no impact on functionality, and it is working as expected, but due to brittle locators, the tests fail. Flaky tests are not uncommon. For example, Google reports that 16% of their tests show some degree of flakiness.

Discussing the quality engineer’s perspective on flaky tests is worthwhile here. From a quality engineer perspective, it is not encouraging as even with the best framework design in place, the tests tend to fail often. It becomes a mundane activity to fix the automation code whenever the build failure occurs due to flaky tests, which takes additional time for analysis and fixing. This is one of the major drawbacks of most automation tools as it is a common practice to use element locators to identify elements and perform actions in test automation code.

The Self-Healing Technique

In test automation, self-healing is a solution that helps solve major pain points such as test maintenance, where automation scripts break whenever a change is made to the object properties (Name, ID, XPath, CSS, etc.). Manually identifying and updating object changes in the object repository takes up significant time, increasing the overall testing effort.

With a self-healing solution, project teams can adopt a Shift-Left approach in the Agile Methodology, and the overall testing process will be more efficient with increased productivity and faster delivery. Many commercial and open-source tools are available that automatically detect these changes and fix them dynamically without human intervention by leveraging machine learning algorithms and other artificial intelligence (AI)-driven algorithms. Most commercial tools like Rapise, Functionize, Applitools, Mabl, Testim, Test.ai, Appvance.ai., Tosca, and others have a codeless automation ideology, and many of these tools have self-healing capabilities. Unfortunately, being commercial frameworks, additional details on the algorithms and solutions offered by such tools are often not disclosed. Open-source tools such as TestProject and Healenium also support self-healing.

The following process flow diagram (Figure 3) shows the end-to-end workflow of the self-healing technique, which is handled by AI-based test platforms.

In this process flow, when the AI engine discovers that an object property had changed and tests might break, it extracts the entire document object model (DOM) and the object properties. In order to extract the updated property, it uses dynamic location strategies for the object based on the weighted scoring system, which is developed based on a machine learning algorithm. It uses the new object properties to perform automated actions on the object during runtime. With this dynamic location strategy, the test scripts run seamlessly without the user knowing about the changes.

Self-Healing Automation: Selecting the Right Tool for the Job

In the world of test automation, there are an overwhelming number of test automation tools to choose from, each one with its own set of benefits and drawbacks. The following list outlines some of the key capabilities to consider when evaluating test automation tools, along with self-healing capabilities.

• CI/CD integration: Some examples are Jenkins, Azure DevOps, Github Actions, Gitlab
• Test Case Management integration: Examples include Zephyr, Xray, TestRail, SpiraTest
• Workflow / Collaboration integration: Examples are Jira, Slack, PagerDuty, SpiraTeam, SpiraPlan
• Cloud platform support
• UI and API level integration testing
• Cross-browser testing
• Mobile OS Testing: Android, iOS, Blackberry
• Parallel test execution

There are multiple factors to consider during tool evaluation, such as your current technology stack, current testing tools, tool cost, open-source or commercial tools, the need for codeless automation tools, team automation skillset for adopting test tools, organization tool policies, and if there is already a framework, whether it can be extended. One of the effective approaches would be to conduct brainstorming sessions to come up with shortlisted tools and then perform some proof of concepts during innovation sprints to come up with recommendations. Based on my experience I have listed a few important aspects below which can serve as good starting points for your tool evaluation:

• Understand and analyze current tools and the technology stack used in developing the application. For example, if the application is built using .NET, one can evaluate open-source and commercial tools. Open-source tools such as TestProject, or commercial tools like Mabl, Functionize, Tosca, Rapise etc. can be considered for evaluation.
• Understand the existing automation framework and evaluate if it can be extended to utilize self-healing tools or libraries. For example, if your current automation framework is Java Selenium-based, the team can evaluate and perform a proof of concept to extend the existing framework to leverage any self-healing supporting libraries such as Healenium.
• Identify appropriate tools that fit your project testing requirements based on the organization’s tool policies and project budget around commercial and open-source tools. For example, if your organization’s policies allow open-source tools, consider tools such as TestProject, Katalon, and Healenium for evaluation.
• Based on the team’s test automation skillset, identify the tools that fit the application testing needs. For example, if the team has subject matter expertise in the application domain but lacks the automation skills, then codeless, rapid test automation development tools such as TestProject, Functionize, Tosca, Mabl, and Rapise could be a good choice.

Final Thoughts

Self-healing in automated tests is here to stay and brings value to agile teams during implementation. With self-healing technique adoption in your testing practice, one can expect fewer tests to fail, prevent flaky tests, and save time and effort. It is one of those opportunities where the teams can gain significant benefits from the new software testing machine learning-driven innovations.