Choosing a Test Case Management Solution for JIRA

Atlassian JIRA is one of the most popular tool for Agile project management. Many add-ons exist for test case management with this tool. In this article, Vladimir Belorusets presents a set of requirements for an efficient test case management process in JIRA that should help you select the best solution for the software testing needs of your organization.

Author: Vladimir Belorusets, PhD, Bluescape, https://www.bluescape.com/

Introduction

JIRA is one of the most popular issue and project tracking framework for software teams. A framework defines a common structure that organizes software artifacts, data, and supporting libraries. Framework functionality can be altered and/or expanded by installing special modules called add-ons or plug-ins. These add-ons are designed in accordance with the framework architecture in order to re-use all framework benefits.

Out of the box JIRA supports the following issue types for software projects:
* Requirements (epics & stories for agile methodology)
* Bugs
* Tasks & sub-tasks

Software testing represents an important part of the software development process. There are several test management add-ons available on the market that are integrated with JIRA to varying degrees. To help testing teams to select the right tool that fits their preferences and needs, I developed a set of requirements for the efficient test case management solution in JIRA. Initially, I presented these requirements along with my preferred plug-in synapseRT  at Atlassian Summit 2014. Time has proved that this approach is practical and valuable. Since then, I have successfully implemented test management process at SafeNet (acquired by Gemalto) and Bluescape.

Requirements to Test Case Management System (TCMS)

In general, any viable TCMS should act as a test IDE. It should support test creation, editing, execution (both manual and automated), and viewing test results. With respect to JIRA, it also must address the “traceability triangle” between the different issue types that is illustrated below.

Here ares my ten + one commandments:

1. TCMS should support, at least, the following test issues:
* Test cases
* Test suites
* Test plans

Figure 1. Traceability triangle.

Herein test suites allow logically grouping and extracting test cases belonging to the same functional area, and Test plans assemble test cases intended for execution.

2. Test issues should be an integral part of JIRA as show in figure 2. In other words, they just need to represent new JIRA issue types. There are some add-ons, which are connected to JIRA, but not really integrated with it. These add-ons are implemented as web applications that you need to open in a separate window, and your work will be linked with JIRA. This design has three major drawbacks:
* you cannot use all the power of JIRA, search engine, for example
* you need to learn a new interface that does not follow the JIRA design guidelines
* you need to constantly switch between JIRA and the TCMS.

Figure 2. Test issue types

3. Test artifacts should be manageable on the project agile boards: added to the sprints and prioritized in the Backlog. Developing and executing test cases takes time. That’s why the time spent on these activities should be counted during the Sprint planning meetings.

4. Test suites should provide the hierarchical grouping of test cases to match the functional decomposition model of the application under test. This feature will help to create easily test plans from the relevant test suites, and review if some test cases are missing for a functional area under test.

5. Test engineers should be able to associate test cases with user stories for test coverage. On the story’s screen you should immediately see what test cases cover that story. And vice versa, from the test case’s screen, you should be able to see what stories that test case verifies.

Figure 3. Test coverage for a story.

6. Engineers should be able to execute test cases from a test plan on multiple environments in parallel without overwriting test case results. Common use cases include running the same test plan on multiple browsers, for example, Firefox and Microsoft Edge, on different versions of the same browser, and on multiple operating systems as desired. To achieve this requirement, I came up with a concept of a test case in JIRA as a static document describing a set of actions and expected results to verify some feature. For analogy, think of a test case as a Class in object-oriented programming. You do not associate any test results directly with the test case. To convert that document to an executable item with test results, you need to create an independent instance of that test case. Back to analogy with object-oriented programming, think of the executable test as a Class Object. You combine all test cases intended for execution into a Test Plan as shown in figure 4. Then you create an instance of the Test Plan that includes, as a bulk operation, the instantiation of all test cases in that test plan. For example, synapseRT calls a named instance of the test plan as a test cycle.

Figure 4. Relationship between a test case as a document and one in the test plan.

7. Test plan should provide an aggregated view of all test plan executions. QA Managers should be able from one screen (test plan view) to evaluate quality of the application. It means viewing high-level test results for all environments and have links to the bugs found during each test plan execution. It is also very beneficial to display all stories covered by the test plan in a separate panel.

8. Besides the aggregated view for one test plan, there should be a detailed traceability matrix for all Test Plans. That matrix could be implemented as a table. It shows how quality of the application changes from sprint to sprint.

9. The TCMS should provide configurable gadgets on JIRA dashboard for high-level assessment of the product quality. These gadgets will help senior management to make an informative decision about application readiness for release. For example, I fond very useful to have a gadget that can show test results (In Progress, Passed, Failed, Can’t Test) for all Test Plans executed in the current sprint as shown below.

Figure 5. An example of the TCMS gadget.

10. The TCMS should provide test automation engineers with the REST API to update an instance of the test plan in JIRA programmatically. To update test case result in an automated script, you should identify an executable instance of the test case. It can be done, for example, by combining three parameters: test plan ID in JIRA, name of the executable instance of that test plan, and ID of the test case added to the test plan.

11. TCMS should provide capability to run automated tests and view their results directly from JIRA. Test plans include both automated and manual tests. The automated tests are usually executed via a CI server. It would be greatly beneficial if TCMS could trigger execution of the automated tests on the CI server, like Jenkins, from JIRA. This feature will give ability to run automated tests on demand to much a broader audience than the test automation engineers:
* anyone without knowledge of automation or programming
* testers who do not have direct permission to work with the CI server

I hope the requirements listed above for test management tools will provide QA Managers with the clear guidelines on how to select the right test case management solution add-on for JIRA for their organizations.

About the author

Dr. Vladimir Belorusets is a Senior QA Manager at Bluescape. He is a Certified Tester–Foundation Level and Certified ScrumMaster. Dr. Belorusets is the author of numerous articles on test automation and testing methodology published in Methods & Tools, Testing Experience, Agile Record, SoftwareTest & Quality Assurance, and other magazines. He was a speaker at Atlassian Summits, Software Test Professionals, HP Software Universe, and STARWEST. Dr. Belorusets holds his PhD in Control Systems from the Institute for Systems Analysis of Russian Academy of Sciences.