Introduction
The more I have gotten into developing testing frameworks based on Cucumber, the more I’ve seen how important writing effective cucumber tests are. The feature files provided earlier are not a good example of cucumber tests, and below we will cover why, and how to turn these into effectively written test cases.
A badly written test informs little of the system, cannot be tested against multiple interfaces, and most painfully requires more upkeep. A smartly formed test can not only be re-used, but can also directly inform anyone reading the test exactly what the system is capable of. Cucumber is a tool that supports Behavior Driven Development, BDD. People think that the only place where a system has behavior is in the user interface and especially in the graphical user interface. This is not the case. All systems have behavior at different places and different levels. Most applications have multiple interfaces, and each of these can be tested by examining their behavior. An important point is that interfaces generally behave the same. This demonstrates how Cucumber and BDD is not about testing GUIs. It is about systems behavior.
Cucumber is Declarative
BDD and Cucumber should be written in a declarative fashion. Define properties that should hold true for a particular solution. A set of boundaries the solution should stay within should be outlined; however the steps to arriving at the solution should not be included. You don’t know, and should not care, about the implementation when you define your feature files. The implementation comes later and may be someone else’s responsibility. These tests should be written from the point of view of an architect, not of a developer. The focus needs to stay on what, not how.
How To Write Tests
The whole point of BDD is that it is vitally important to write each test/feature step
- One at a time
- With a domain expert
- In plain Language
The use of plain language in the feature file is crucial to successful use of Cucumber. State the result that you wish to obtain while avoiding specifying how you expect to get it.
Test Step Amgibuity
Because there must only be one method (step) which matches the regular expression in the classpath, it is important to view the system at a high level. These methods are global; the reasoning being that they describe a part of the system, and just as a system with redundant systems might need a design, so should tests. If two different parts are described the same way, then they are the same and the same step definition can be used in both cases. If you have described two different parts of the system with the exact same wording, then you have an issue with ambiguity. Take some time and understand why you describe different parts of a system the same way. Chances are that they actually are the same thing and that you have some refactoring to do.
A Quick Example
When looking at a test to retrieve a quote from a website, a test statement should look like
When request a rate quote for 54481 for 1 month out
And not
Given I visit the page www.deltadentalcoversme.com When I enter 54481 in the zip field And I click GetQuote And I select 1 month out And I click GetQuote Then I see a rate quote
What is important about the difference in styles? The first example, When request a rate quote for 54481 for 1 month out, is a functional requirement. The second, much longer, example is a procedural reference. Functional requirements are features and procedures belong in the implementation details. You should concern yourselves with that which has to happen and not how you expect it to happen. That way when somebody later decides to change the rate workflow, or add another interface, you then simply change the implementation of the steps behind the scenes.
Fixing our Previous Cucumber Tests
When trying to write tests thinking about the what and not the how, it helps to envision the system as if you were performing black box testing. Determine the initial setup you require, look at the inputs you have, ignore what specifically happens to the data, and then again look at the expected output. Let’s take the example of some of our previous login tests.
Feature: Testing for login page Scenario: Login without password Given I want to use the browser Firefox When I type testuser1 in the username input field And I click the login button Then I see the login error message "Please provide a password." And I am on the login page Scenario: Login with bad password Given I want to use the browser Firefox When I type testuser1 in the username input field When I type testuser2 in the password input field And I click the login button Then I see the login error message "The password provided does not match the username entered." And I am on the login page Scenario Outline: Successful login Given I want to use the browser [browser] When I type testuser1 in the username input field When I type testuser1 in the password input field And I click the login button Then I am on the launcher page Examples: | browser | | Firefox | | Chrome | | InternetExplorer |
Let’s pull out our interested data into the below table
| Test Case | Prerequisites | Inputs | Outputs |
|---|---|---|---|
| No password | Browser=Firefox | username=testuser1 | URL=login, error message=Please provide a password. |
| Bad password | Browser=Firefox | username=testuser1, password=testuser2 | URL=login, error message=The password provided does not match the username entered |
| Successful | Browser=[variable] | username=testuser1, password=testuser1 | URL=launcher |
Using this data, let’s rebuild our test cases more generically
Feature: Testing for login page Scenario: Login without password Given I want to use the browser Firefox When I set the username to testuser1 When I login to CosmicComix Then I see the error message "Please provide a password." And I am on the login page Scenario: Login with bad password Given I want to use the browser Firefox When I set the username to testuser1 And I set the password to testuser2 When I login to CosmicComix Then I see the error message "The password provided does not match the username entered." And I am on the login page Scenario Outline: Successful login Given I want to use the browser [browser] When I set the username to testuser1 And I set the password to testuser1 When I login to CosmicComix Then I am on the launcher page Examples: | browser | | Firefox | | Chrome | | InternetExplorer |
We can see how these test steps are more generic guidelines for what to do to login to any system, and not at this level specific to our Cosmic Comix application design. If I changed the layout, or fields that were present on the login page, these tests would not have to change, just the back-end step controlling these test steps. For this small subset of three tests, that means changing one method, instead of changing three. The more tests we have, obviously the less rework will be required. Additionally, a user with little information about how our system behaves can understand these tests with more ease than the previous ones.
7 thoughts to “Writing Effective Cucumber Tests”
Great article – we’ve just begun writing BDD in SpecFlow and our tests embody every definition of “what not to do”, as we pretty much list every attribute of every entity that has anything to do with a particular test. Using your suggested approach makes a lot more sense, but I wonder what added challenges it introduces, if any? Is it more difficult for the developers to code? I’m merely the BA writing the tests, and our devs are all just as new to BDD as myself, so it’d be interesting to know. Also, I’ve attached a revised test based on what you suggest – the previous iteration basically had background data listing every attribute of every person linked to a particular client, as well as the attributes of each client and relationship with the person. This seems much simpler, but is it now too simple? It’s essentially testing UI, but given we have API’s plugging into the entire web app, this level of testing is required.
Scenario: Golden Path for Viewing Client Personnel
Given Client has linked Personnel
When View Client Personnel Requested
Then Linked Client Personnel Returned
But Inactive Linked Client Personnel Not Displayed
#Linked Personnel is a Manager, Contact Person or Partner linked to the Client within whose context the Personnel are being viewed.
The challenges that are introduced are mainly based on communication and coordination between the development and test team. By extending these tests further, changes in the code can often squeeze by unnoticed as they may not immediately effect any of the tests. However, a well coordinated team shouldn’t run into this issue, nor would a team where developers are in charge of maintaining the glue code. They would know that every change in code should indicate some change needs to be made in the test.
I’m curious what roles your testers are playing, and where the line is between the work being done.
Additionally, pulling the data out of the tests can introduce another problem of how and where to provide the data. It could be pulled in from property files, or spreadsheets, or possibly even stored somewhere else completely.
As for your tests, I do not think they are too simple, as long as the terms you are using have meaning. Often times we can use terms like Client and Personnel to take on some default or standard values, that can be understood across the team. For example, in some of my tests, when I refer to an Adult in a test step, I mean an individual, unmarried, who is exactly 30 years old, with no kids. If I then make reference to a user named a Child or Spouse, that changes the information of the Adult, and provides more/different context.
Having testers and those involved reading familiar with these terms can greatly reduce the information that needs to be provided into the tests. That said, sometimes I have found it difficult when additional information is needed. In these times, storing data in a separate file, and properly referencing it may be your best bet
Max, thanks for your prompt and comprehensive response. To answer your question, we have a rather unique mix of people writing the BDD scenarios, but each of us are predominantly the tester, the analyst, the subject matter expert, and the scenario developer. Once we have written our scenarios we then pass them on to the developers who get started on the associated step definitions and code. We are looking to move to a situation where the analysts can also write the method ‘shells’ and just leave the devs to work on actual code, but we’re a wee way off that at this stage. At the end of this month we have some training booked with someone called Gojko Adzic which will be interesting, and we’ll have both devs and analysts alongside one another for that.
And that’s a very good point you raise regarding pulling the data out of the tests – is it an option to instead keep the data in the tests, but perhaps only have the one scenario for describing what constitutes “Active” Client Personnel (e.g. Given fields x,y,z / When I save a New Personnel Record / Then an Active Client Personnel record is created, or something to this effect) and simply reference that as a comment each time I create a scenario that needs an active personnel record? That way, if another field is introduced to Personnel, we only need to go back to that one scenario and add the field to that.
Interested to know your thoughts.
That is definitely an option, and you may want to look into the Background capabilities of Cucumber to provide this information simply to multiple tests. I posted a link to my post going over some good practices for it.
The problem you may run into with defining your user once in a scenario (or even background), is setting up dependencies. If you want to use it in multiple scenarios, that one setup scenario (or feature if using background) always has to be run. Dependencies are non-trivial to setup within a Cucumber testing framework, and you may not want them, keeping each scenario independent to keep your automated tests lean and fast. That said, adding additional test steps to every scenario also doesn’t fit with keeping them lean
http://www.coveros.com/background-and-hooks-for-cucumber-jvm/
Very nice article Max… thank you.
I have a question : I am working in agile from last 5-6 years and this cucumber suites Agile a lot, as it can convert.convey business requirements to test scripts.
Now my question is Who should write Cucumber .feature files?
1.QA team 2. BA 3. who?
Hi Hari,
That really depends on your setup, but I would suggest getting your domain experts to write your feature files. Depending on your group, this may or may not be your BAs. I then would then recommend that the QA team review the scenarios written, ensure that they are good and testable, and to expand each scenario into a scenario outline. In this way you get the best of both worlds.
yes, that’s correct …thanks