Unit Testing in Java: A Sleeping Snail

In the context of Java unit testing, a sleeping snail is a test that’s sluggish and takes (relatively speaking) forever to run because it relies on Thread#sleep and arbitrarily long waits to allow threads to execute before performing assertions or continuing with the workflow under test. In this article, based on chapter 5 of Unit Testing in Java, author Lasse Koskela explains this code smell and the appropriate deodorant with an example

Author: Lasse Koskela, http://lassekoskela.com/

This article is based on Unit Testing in Java, to be published in April 2012. It is being reproduced here by permission from Manning Publications. Manning early access books and ebooks are sold exclusively through Manning. Visit the book’s page for more information.

Working with file I/O is dreadfully slow compared to working with in-memory data. Slow tests are a major drag to maintainability because programmers need to run the test suite again and again as they work on changes whether it’s about adding new functionality or fixing something that’s broken. File I/O isn’t the only source of slow tests, however. Often a much bigger impact to a test suite s execution time stems from the army of Thread#sleep calls put in place to allow other threads to finish their work before asserting on the expected outcomes and side-effects. This is a smell that’s remarkably easy to spot—just look out for calls to Thread#sleep and keep an eye out for exceptionally slow tests. Unfortunately, getting rid of the stench may not be quite that straightforward. Having said that, it’s very much doable and most definitely worth the trouble.

Let’s dig into this code smell and the appropriate deodorant with an example.

Example

Since threads often bump up the complexity of our code quite much, let s pick an example that’s simple but involves testing behavior in a concurrent access scenario. Listing 1 presents a test for a Counter object responsible for generating unique numbers. Obviously, we want these numbers to be unique even if multiple threads invoke our Counter simultaneously.

Listing 1 Testing multi-threaded access to a counter

@Test
public void concurrentAccessFromMultipleThreads() throws Exception {
final Counter counter = new Counter();

final int callsPerThread = 100;
final Set values = new HashSet(); #1
Runnable runnable = new Runnable() { #1
@Override #1
public void run() { #1
for (int i = 0; i < callsPerThread; i++) { #1
values.add(counter.getAndIncrement()); #1
} #1
} #1
}; #1

int threads = 10;
for (int i = 0; i < threads; i++) { #2
new Thread(runnable).start(); #2
} #2

Thread.sleep(500); #3

int expectedNoOfValues = threads * callsPerThread;
assertEquals(expectedNoOfValues, values.size()); #4
}

#1 Threads increment counter in a loop
#2 Start threads
#3 Wait for threads to finish
#4 Check values’ uniqueness

The test in listing 1 sets up several threads to repeatedly access the counter and starts all the threads. Since the test has no way of knowing how the threads are scheduled and when they finish we’ve added a call to Thread#sleep, making the test wait for half a second before continuing with the assertion. While 500 milliseconds (ms) might sound exaggerated; it’s not, really. Even with a sleep of 300 ms, the test would fail roughly 10 percent of the time on my computer—and that’s flaky with a capital F!

Now, that 500 ms sleep is still not totally reliable—sometimes it might take longer for all threads to complete their job. In fact, as soon as the test fails because that 500 ms sleep is long enough, the programmer is likely to increase the sleep. After all, we don’t like flaky tests.

Flakiness is most definitely a big issue. However, there’s another problem with a test sleeping like we do in listing 1 as it’s pretty slow. While 500 ms doesn’t sound like a lot, it does add up, especially if our code base has multiple tests with Thread#sleeps sprinkled all over.

So let’s see what we can do about those sleepy tests.

What to do about it?

The reason why Thread#sleep is slow is because there’s a delay between the time when the worker threads finish and when the test thread knows that they’ve finished. Even though some of the time the threads would all be finished in 10 ms, sometimes it takes several hundred milliseconds (or more) and, therefore, we have to wait for hundreds of milliseconds every time, regardless of how much of that waiting is unnecessary.

Listing 2 presents a better approach that replaces the wasteful and unreliable Thread#sleep. The fundamental difference is that we make use of Java’s synchronization objects from the java.util.concurrent package to make the test thread immediately aware of when the worker threads have completed their work.

Listing 2 Testing multithreaded access without sleeping

@Test
public void concurrentAccessFromMultipleThreads() throws Exception {
final Counter counter = new Counter();

final CountDownLatch allThreadsComplete = #1
new CountDownLatch(threads); #1

final int callsPerThread = 100;
final Set values = new HashSet();
Runnable runnable = new Runnable() {
@Override
public void run() {
for (int i = 0; i < callsPerThread; i++) {
values.add(counter.getAndIncrement());
}
allThreadsComplete.countDown(); #2
}
};

int threads = 10;
for (int i = 0; i < threads; i++) {
new Thread(runnable).start();
}

allThreadsComplete.await(10, TimeUnit.SECONDS); #3

int expectedNoOfValues = threads * callsPerThread;
assertEquals(expectedNoOfValues, values.size());
}

#1 Synchronization latch
#2 Mark thread completed
#3 Wait for threads to complete

We’ve highlighted the key differences in listing 2. The central piece in this solution is the synchronization object we use for coordinating between the worker threads and the test thread. A CountDownLatch is initialized to count down from the number of worker threads, and each worker thread ticks the latch once when they finish their work. After starting up all worker threads, our test proceeds to wait on the latch. The call to CountDownLatch#await blocks until either all threads have notified about their completion or the given timeout is reached.

With this mechanism our test, thread wakes up to perform the assertion immediately when all worker threads have completed. No unnecessary sleeping and no uncertainty about whether 500 ms is long enough for all threads to finish.

Summary

A sleeping snail is a test that’s sluggish and takes (relatively speaking) forever to run because it relies on Thread#sleep and arbitrarily long waits to allow threads to execute before performing assertions or continuing with the workflow under test. This smells because thread scheduling and variance in execution speed implies that those sleeps must be much, much longer than what the threads would typically need to finish their work. All of those seconds add up, making our test suite slower and slower with every single sleep.

The solution to getting rid of this smell is based on the simple idea that the test thread should be notified when each worker thread actually completes. As long as the test thread has this knowledge, the test can resume itself as soon as it makes sense, essentially avoiding the unnecessary wait associated with the Thread#sleep hack.

The solution in listing 2 builds on one of the standard synchronization objects in the Java API, CountDownLatch. The CountDownLatch is useful for ensuing test execution only once the required number of other threads has passed the latch. If you work with a system dealing with threads, do yourself a favor and dive into the utilities provided by the java.util.concurrent package.