What Happens In The Case When You Are Testing?

Two mice dressed as detectives contemplating a question Image generated by Bing Image Creator

A simpler, plain-English definition of test cases

I have my own definition of test case that I believe to be more useful, more accurate, than the conventional and common definition.

A test case is a way of asking and answering the question “What happens in the case when…?” via observation and possible manipulation of something.

Observation in combination with asking and answering questions is what makes the definition about testing.

“…in the case when…“ utilizes the simple, plain English use of the word “case” in the same way people use the same word in regular conversation.

  • The definition is independent of specification, structure, or content of any given test case.
  • The definition is independent about when or how the test case is realized, perceived, or acted upon.
  • The definition is structured in a way that makes it compatible with, any reasonable testing methodology or practice.
  • The definition suggests that a test case always exists so long as one is testing independent of awareness of the test case or formal expression of it.
  • The definition establishes that test cases are differentiated by what follows “…in the case when…“, and that when we are learning we are seeking to understand how those differences affect what we are observing.

I find this definition an accurate description of what a tester truly does. It describes the unit of learning which generates the value and currency of testing. Whether a tester is following a precisely written script, or exploring ideas from a set of topics, or executing a set of tools to apply stochastic load patterns, or applying some other testing approach, they are still in some way or another learning “What happens in the case when…?”

I find this definition useful because it gets at the heart of what is interesting in testing - which comes down to figuring out what to do in order to learn the most important information necessary to describe issues that present risk to whatever product is underway. For a tester, the “what to do” is a plan for getting at different ways of asking “what happens in the case when…“

I find it distracting to conflate the definition of test cases with any notions of how a test case is specified, expressed, used, or executed. I see those as formal applications or manifestations of a test case, but not the same as a test case. The distraction comes when we insist test cases take a specific form, and that form runs counter to a certain testing methodology. Now, instead of focusing on the different ways asking, “what happens in the case when…“ we are asking “should I use test cases?” when we might REALLY be asking something more specific such as, “Is it necessary for me to follow IEEE 829-1989 to describe my tests?” While that may be a valid question to ask, it dodges a far more important point.

What are you going to do?

From this definition, the question “Should I use test cases?” becomes a nonsense question, because phrased as it is, test cases are inherently present once someone engages in the act of testing. One may anticipate and describe them fully up front. One may have some ideas of things to try, but not write down the steps and procedures explicitly. One might ad hoc exercise an application and after the fact realize that taking certain actions under certain conditions resulted in a certain outcome. In all these examples, and in everything else that is testing, the notion of “What happens in the case when…” is present. We may never see any expression of it, should we decide not to report on our testing. But if we do any kind of testing report, the test case will manifest in some sort of expression. If we say, “Executed operations using SSL authentication, and the following issue occurred…”, that is a “What happens in the case when…” account. A bug report is an output of a test case, one where the question has been answered (and the outcome is believed as possibly undesired).

Understanding test cases in this way allows one to step back from specifics of how they are expressed or utilized and focus first on what the test case is. It is the unit of understanding product behavior, what happens in the case when. We can ask ourselves what we need to know? What don’t we know? Based on what we know already, what concerns us that we need to know more about? Based on what we know already, what decisions must we make about what to do next? Do we feel good about what we know already, or do we feel as if we need to know more? All of these questions motivate us to consider, or sometimes disregard asking further rounds of “What happens in the case when…?”

Considering Some Examples

Consider the simple case of an application you have encountered for the first time. Imagine it is runnable, and you are presented with it. You do not know what it does. You could begin with a simple set of generic test cases: “What happens when I launch the application?”, “What happens when I try using each visible control in the UI?”, “What happens when I attempt to close the application?”

This might lead to findings such as:

  • When I launch the application, I am presented with a sign-in dialog but no way to create a login
  • When I try to close the application from the sign-in dialog via the X in the upper right corner, there is a beep, and the application does not close.

Even if we do not know what the application is or does, we can guess at a few test cases. Once we are testing, we find ourselves discovering test cases, such as “What happens in the case when a user is presented with a sign-in dialog and does not have an account.”, or “What happens in the case when the user tries to close the application during sign-in?”

Neither of these cases were planned. Knowing nothing about the application, there was no way to plan them. They don’t need to be planned, but the moment the tester takes action and makes observations, the cases are apparent and emergent.

Consider now a very short test report:

Launched application, but did not have a login to use for further testing after sign-in confirmed sign-in with made up login did not let me through ISSUE: sign-in dialog does not present means to create a sign-in id ISSUE: attempt to close application at sign-in beeps and does not close: how is user to leave the app in this case?

Even with a very simple, low level of formality (note the absence of Jira tickets - we are running a small operation here), the test report indicates a great deal of information. First, the application remains very much not tested due to issues of missing information, missing product capabilities. Second, you can see that at least three test cases are implied in the report, so the product team has some small measure of coverage, however informally established.

The same story could be told using an explicit, detailed, step by step, pre-planned set of test cases with a case-by-case pass/fail/blocked report. Either way, the definition of test case presented here still works. The only difference being in one story the cases are fully articulated and detailed up front with stepwise instructions, and in another they are considered in brief in combination with those learned on the way. Either version has the same relationship to “test case,” it does not change based on the approach. The same value and utility exists. We can ask what we ought to look for, what we have seen, what we do not know yet. We can try to think of as many things up front as we can, or rely on our ability to adapt and learn to find things we did not anticipate.

All of that falls apart if we say test cases must take a specific form. We lose track of what we really get from the test case, what relationship they have to the act of testing, and we begin analyzing information relative to the specific form (e.g. “what % of the cases pass/fail” or “how fast do we execute the cases?”) and not to the actual value of the case (e.g. “what have we learned? what have we not learned?”). Not that some of these other questions are devoid of purpose, but they represent an entirely different type of analysis and I would argue are about test cases expressed in a specific form within a specific methodology, and therefore ought to have some other word or modifier to describe them (James Bach brilliantly utilizes the term ‘documented narrative test procedure’ as a way to describe test cases in the traditionally understood, formal sense). If we eschew test cases altogether - at least the idea of them (we can substitute different words if we like and not commit the crime of eschewing - but I assert the phrase ‘test case’ makes the most sense in its plain English sense) - then we miss out on the idea of anticipating, or contemplating what can happen prior to us acting on it, or accounting after the fact for what we have done. I have seen no serious formal practitioner of any methodology commit that particular crime, but I have worked with testers who were entirely incapable or unwilling to articulate intent or activity in any sense - forcing their partners to shrug their shoulders wondering what they have been doing during ambiguous hours of seemingly busy activity beyond the bugs they reported. It is for these reasons I hold to the term “test case,” but insist on my definition as presented here. It is the basis of how we understand and discuss what tests we have done, and escapes the trappings of specific form or methodology.