Broad assertion cuts out too large a chunk of output and side effects for bit-to-bit comparison, which makes it harmful due to the resulting brittleness. The assertion fails if any small detail changes, regardless of whether that change is relevant to the interests of this particular test. In this article, based on chapter 4 of Unit Testing in Java, author Lasse Koskela explains how to refactor the tests that suffer from broad assertion.
Author: Lasse Koskela, http://lassekoskela.com/
The is an excerpt from Unit Testing in Java by Lasse Koskela to release Fall 2011. It is being reproduced here with the permission of Manning Publications. MEAP (Manning Early Access Program) ebooks allow the reader to learn with the authors as the book is conceived and produced with online forums for feedback. Compliments of Manning Publications is a 41% discount on Unit Testing in Java. Enter promotional code methtools41 at checkout at Manning.com and receive 41% off the MEAP/ebook and pbook. Visit the book’s page for more information: http://www.manning.com/koskela2/
A broad assertion is one that is so scrupulous in nailing down every little detail about the behavior it is checking that it becomes brittle and hides its intent under its overwhelming breadth and depth. When you encounter a broad assertion, it’s hard to say what exactly is it supposed check and, when you step back to observe, that test is probably breaking far more frequently than the average because it’s so picky that any change whatsoever will cause a difference in the expected output.
Let’s make this discussion a bit more concrete again by looking at an example test that suffers from this
condition.
Example
The following example is my very own doing. I wrote it some years back as part of a sales presentation tracking system for a medical corporation. The corporation wanted to gather data on how the various sales presentations were carried out by the sales fleet that visited doctors to push their products. Essentially, they wanted a log of which salesman showed which slide of which presentation for how many seconds before moving on.
The solution involved a number of components. There was a little plug-in in the actual presentation file, triggering events when starting a new slide show, entering a slide, and so forth—each with a timestamp to signify when that particular event happened. Those events were pushed to a background application that appended them into a log file. Before synchronizing that log file with the centralized server, however, we transformed the log file into another format, preprocessing it a bit to make it easier for the centralized server to chomp the log file and dump the numbers into a central database. Essentially, we calculated the slide durations from timestamps.
The object responsible for this transformation was called a LogFileTransformer and, being test-infected as I was, I had written some tests for it. Listing 1 presents one of those tests—the one that suffered from broad assertion—along with the relevant setup. Have a look at it and see if you can detect the broad assertion.
Listing 1. Broad assertion makes a test brittle and opaque
public class LogFileTransformerTest {
private String expectedOutput;
private String logFile;
@Before
public void setUpBuildLogFile() {
StringBuilder lines = new StringBuilder();
appendTo(lines, “[2005-05-23 21:20:33] LAUNCHED”);
appendTo(lines, “[2005-05-23 21:20:33] session-id###SID”);
appendTo(lines, “[2005-05-23 21:20:33] user-id###UID”);
appendTo(lines, “[2005-05-23 21:20:33] presentation-id###PID”);
appendTo(lines, “[2005-05-23 21:20:35] screen1”);
appendTo(lines, “[2005-05-23 21:20:36] screen2”);
appendTo(lines, “[2005-05-23 21:21:36] screen3”);
appendTo(lines, “[2005-05-23 21:21:36] screen4”);
appendTo(lines, “[2005-05-23 21:22:00] screen5”);
appendTo(lines, “[2005-05-23 21:22:48] STOPPED”);
logFile = lines.toString();
}
@Before
public void setUpBuildTransformedFile() {
StringBuilder file = new StringBuilder();
appendTo(file, “session-id###SID”);
appendTo(file, “presentation-id###PID”);
appendTo(file, “user-id###UID”);
appendTo(file, “started###2005-05-23 21:20:33”);
appendTo(file, “screen1###1”);
appendTo(file, “screen2###60”);
appendTo(file, “screen3###0”);
appendTo(file, “screen4###24”);
appendTo(file, “screen5###48”);
appendTo(file, “finished###2005-05-23 21:22:48”);
expectedOutput = file.toString();
}
@Test
public void transformationGeneratesRightStuffIntoTheRightFile()
throws Exception {
TempFile input = TempFile.withSuffix(“.src.log”).append(logFile);
TempFile output = TempFile.withSuffix(“.dest.log”);
new LogFileTransformer().transform(input.file(), output.file());
assertTrue(“Destination file was not created”, output.exists());
assertEquals(expectedOutput, output.content());
}
// rest omitted for clarity
}
Did you see it? Did you see the broad assertion? You probably did – there are only two assertions in there. But, which of the two is the culprit here, and what makes it too broad?
The first assertion checks that the destination file was indeed created. The second assertion checks that the destination file’s content is what’s expected. Now, the value of the first assertion is questionable and it
should probably be deleted. However, it’s the second assertion that’s our main concern—the broad assertion: assertEquals(expectedOutput, output.content());
This is quite a relevant assertion in the sense that it verifies exactly what the name of the test implies—that the right stuff ended up in the right file. The problem is really that the test is too broad, resulting in the assertion’s being a wholesale comparison of the whole log file. It’s a thick safety net, that’s for sure, as even the tiniest of changes in the output will fail the assertion. And therein lies the problem.
A test that has never failed is of little value—it’s probably not testing anything. In the other end of the spectrum, a test that always fails is a mere nuisance. What we’re looking for is a test that has failed in the past, proving that it is able to catch a deviation from the desired behavior of the code it’s testing and that it will break again if we make such a change to the code it’s testing.
The test in our example fails to fulfill this criterion by failing too easily, making it brittle and fragile. But that’s only a symptom of a more fundamental issue—the problem of being a broad assertion. The various small changes in the log file’s format or content that would break this test are valid reasons to fail the test. There’s nothing intrinsically wrong about the assertion. The problem lies in the test’s violation of a fundamental guiding principle for what constitutes a good test:
A test should have only one reason to fail.
If that principle seems familiar, it’s a variation of a well-known object-oriented design principle, the Single Responsibility Principle, which says, “A class should have one, and only one, reason to change.” [1] Now let’s clarify why the principle of having only one reason to fail is so important.
Catching many kinds of changes in the generated output is good. However, when the test does fail, we want to know why.
In our example it’s quite difficult to tell what happened if this test transformationGeneratesRightStuffIntoTheRightFile, suddenly breaks. In practice, we’ll always have to look at the details to figure out what had changed and, consequently, broke the test. If the assertion is too broad, many of those details that break the test are in fact irrelevant. How should we go about improving this test, then?
What to do about it?
The first order of action when encountering an overly broad assertion is to identify irrelevant details and remove them from the test. In our example, we might look at the log file being transformed and try to reduce the number of lines.
We want it to represent a valid log file and be elaborate enough for the purposes of the test. For example, our log file has timings for five screens. Maybe two or three would be enough? Could we get by with just one?
This question brings us to the next improvement to consider—splitting the test. Asking ourselves how few lines in the log file we could get by with quickly leads to concerns about the test no longer testing this and that. Listing 2 presents one possible solution where each aspect of the log file and its transformation are extracted into separate tests.
Listing 2 More relaxed, semantics-oriented assertions reduce brittleness and improve readability
public class LogFileTransformerTest {
private static final String END = “2005-05-23 21:21:37”;
private static final String START = “2005-05-23 21:20:33”;
private LogFile logFile;
@Before
public void setUp() {
logFile = new LogFile(START, END);
}
@Test #1
public void overallFileStructureIsCorrect()
throws Exception {
StringBuilder expected = new StringBuilder();
appendTo(expected, “session-id###SID”);
appendTo(expected, “presentation-id###PID”);
appendTo(expected, “user-id###UID”);
appendTo(expected, “started###2005-05-23 21:20:33”);
appendTo(expected, “finished###2005-05-23 21:21:37”);
assertEquals(expected.toString(), transform(logFile.toString()));
}
@Test #2
public void screenDurationsGoBetweenStartedAndFinished()
throws Exception {
logFile.addContent(“[2005-05-23 21:20:35] screen1”); String out =
transform(logFile.toString());
assertTrue(out.indexOf(“started”) < out.indexOf(“screen1”));
assertTrue(out.indexOf(“screen1”) < out.indexOf(“finished”));
}
@Test #3
public void screenDurationsAreRenderedInSeconds()
throws Exception {
logFile.addContent(“[2005-05-23 21:20:35] screen1”);
logFile.addContent(“[2005-05-23 21:20:35] screen2”);
logFile.addContent(“[2005-05-23 21:21:36] screen3”);
String output = transform(logFile.toString());
assertTrue(output.contains(“screen1###0”));
assertTrue(output.contains(“screen2###61”));
assertTrue(output.contains(“screen3###1”));
}
// rest omitted for brevity
private String transform(String log) { ... }
private void appendTo(StringBuilder buffer, String string) { ... }
private class LogFile { ... }
}
#1 Checks that common headers are placed correctly
#2 Checks screen durations’ place in the log
#3 Checks screen duration calculations
The solution above introduces a test helper class, LogFile, which establishes the standard “envelope”— the header and footer—for the log file being transformed based on the given starting and ending timestamps. This allows the second and the third test, screenDurationsGoBetweenStartedAndFinished and screenDurationsAreRenderedInSeconds, to append just the screen durations to the log, making the test more focused and easier to grasp. In other words, we delegate some of the responsibility for constructing the complete log file to LogFile. In order to ensure that that responsibility receives due diligence, the overall file structure is verified by the first test, overallFileStructureIsCorrect, in the context of the simplest possible scenario—an otherwise empty log file.
This refactoring has given us more focus by hiding the details from each test that are irrelevant for that particular test. That is also the downside of this approach—some of the details are hidden. In applying this technique, we must ask ourselves what we value more—being able to see the whole in one place or being able to see the essence of a test quickly.
I suggest that, most of the time, when speaking of unit tests, the latter is more desirable as the fine-grained, focused tests point us quickly to the root of the problem in case of a test failure. With all tests making assertions against the whole transformed log file, for example, a small change in the file syntax could easily break all of our tests making it more difficult to figure out what exactly broken—where’s the problem?
Summary
We can shoot ourselves in the proverbial foot by making too broad assertions. A broad assertion cuts out too large a chunk of output and side effects for bit-to-bit comparison, which makes it harmful due to the esulting brittleness—the assertion fails if any small detail changes, regardless of whether that change is relevant to the interests of this particular test.
A broad assertion also makes it difficult for the programmer to identify the intent and essence of the test. When you see a test that seems to bite off a lot, ask yourself what exactly do you want to verify? Then, try to formulate your assertion in those terms.
References
[1] Agile Software Development; Martin, Robert C., Addison-Wesley