Written by Stephen Cleary | November 2014 | Get the Code
Unit testing is a cornerstone of modern development. The benefits of unit testing for a project are pretty well understood: Unit testing decreases the number of bugs, reduces time to market and discourages overly coupled design. Those are all nice benefits, but there are further advantages more directly relevant to developers. When I write unit tests, I can have much greater confidence in the code. It’s easier to add features or fix bugs in tested code, because the unit tests act as a safety net while the code is changing.
Writing unit tests for asynchronous code brings a few unique challenges. Furthermore, the current state of async support in unit test and mocking frameworks varies and is still evolving. This article will consider MSTest, NUnit and xUnit, but the general principles apply to any unit testing framework. Most of the examples in this article will use MSTest syntax, but I’ll point out any differences in behavior along the way. The code download contains examples for all three frameworks.
Before diving into the specifics, I’ll briefly review a conceptual model of how the async and await keywords work.
Async and Await in a Nutshell
The async keyword does two things: it enables the await keyword within that method, and it transforms the method into a state machine (similar to how the yield keyword transforms iterator blocks into state machines). Async methods should return Task or Task<T> when possible. It’s permissible for an async method to return void, but it’s not recommended because it’s very difficult to consume (or test) an async void method.
The task instance returned from an async method is managed by the state machine. The state machine will create the task instance to return, and will later complete that task.
An async method begins executing synchronously. It’s only when the async method reaches an await operator that the method may become asynchronous. The await operator takes a single argument, an “awaitable” such as a Task instance. First, the await operator will check the awaitable to see if it has already completed; if it has, the method continues (synchronously). If the awaitable isn’t yet complete, the await operator will “pause” the method and resume when the awaitable completes. The second thing the await operator does is retrieve any results from the awaitable, raising exceptions if the awaitable completed with an error.
The Task or Task<T> returned by the async method conceptually represents the execution of that method. The task will complete when the method completes. If the method returns a value, the task is completed with that value as its result. If the method throws an exception (and doesn’t catch it), then the task is completed with that exception.
There are two immediate lessons to draw from this brief overview. First, when testing the results of an asynchronous method, the important bit is the Task it returns. The async method uses its Task to report completion, results and exceptions. The second lesson is that the await operator has special behavior when its awaitable is already complete. I’ll discuss this later when considering asynchronous stubs.
The Incorrectly Passing Unit Test
In free-market economics, losses are just as important as profits; it’s the failures of companies that force them to produce what people will buy and encourage optimum resource allocation within the system as a whole. Similarly, the failures of unit tests are just as important as their successes. You must be sure the unit test will fail when it should, or its success won’t mean anything.
A unit test that’s supposed to fail will (incorrectly) succeed when it’s testing the wrong thing. This is why test-driven development (TDD) makes heavy use of the red/green/refactor loop: the “red” part of the loop ensures the unit test will fail when the code is incorrect. At first, testing code that you know to be wrong sounds ludicrous, but it’s actually quite important because you must be sure the tests will fail when they need to. The red part of the TDD loop is actually testing the tests.
With this in mind, consider the following asynchronous method to test:
public sealed class SystemUnderTest
{
public static async Task SimpleAsync()
{
await Task.Delay(10);
}
}
Newcomers to async unit testing will often make a test like this as a first attempt:
// Warning: bad code!
[TestMethod]
public void IncorrectlyPassingTest()
{
SystemUnderTest.SimpleAsync();
}
Unfortunately, this unit test doesn’t actually test the asynchronous method correctly. If I modify the code under test to fail, the unit test will still pass:
public sealed class SystemUnderTest
{
public static async Task SimpleAsync()
{
await Task.Delay(10);
throw new Exception("Should fail.");
}
}
This illustrates the first lesson from the async/await conceptual model: To test an asynchronous method’s behavior, you must observe the task it returns. The best way to do this is to await the task returned from the method under test. This example also illustrates the benefit of the red/green/refactor testing development cycle; you must ensure the tests will fail when the code under test fails.
Most modern unit test frameworks support Task-returning asynchronous unit tests. The IncorrectlyPassingTest method will cause compiler warning CS4014, which recommends using await to consume the task returned from SimpleAsync. When the unit test method is changed to await the task, the most natural approach is to change the test method to be an async Task method. This ensures the test method will (correctly) fail:
[TestMethod]
public async Task CorrectlyFailingTest()
{
await SystemUnderTest.FailAsync();
}
Avoiding Async Void Unit Tests
Experienced users of async know to avoid async void. I described the problems with async void in my March 2013 article, “Best Practices in Asynchronous Programming” (bit.ly/1ulDCiI). Async void unit test methods don’t provide an easy way for their unit test framework to retrieve the results of the test. In spite of this difficulty, some unit test frameworks do support async void unit tests by providing their own SynchronizationContext in which its unit tests are executed.
Providing a SynchronizationContext is somewhat controversial, because it does change the environment in which the tests run. In particular, when an async method awaits a task, by default it will resume that async method on the current SynchronizationContext. So the presence or absence of a SynchronizationContext will indirectly change the behavior of the system under test. If you’re curious about the details of SynchronizationContext, see my MSDN Magazine article on the subject atbit.ly/1hIar1p.
MSTest doesn’t provide a SynchronizationContext. In fact, when MSBuild is discovering tests in a project that uses async void unit tests, it will detect this and issue warning UTA007, notifying the user that the unit test method should return Task instead of void. MSBuild won’t run async void unit tests.
NUnit does support async void unit tests, as of version 2.6.2. The next major update of NUnit, version 2.9.6, supports async void unit tests, but the developers have already decided to remove support in version 2.9.7. NUnit provides a SynchronizationContext only for async void unit tests.
As of this writing, xUnit is planning to add support for async void unit tests with version 2.0.0. Unlike NUnit, xUnit provides a SynchronizationContext for all of its test methods, even synchronous ones. However, with MSTest not supporting async void unit tests, and with NUnit reversing its earlier decision and removing support, I wouldn’t be surprised if xUnit also chooses to drop async void unit test support before version 2 is released.
The bottom line is that async void unit tests are complicated for frameworks to support, require changes in the test execution environment, and bring no benefit over async Task unit tests. Moreover, support for async void unit tests varies across frameworks, and even framework versions. For these reasons, it’s best to avoid async void unit tests.
Async Task Unit Tests
Async unit tests that return Task have none of the problems of async unit tests that return void. Async unit tests that return Task enjoy wide support from almost all unit test frameworks. MSTest added support in Visual Studio 2012, NUnit in versions 2.6.2 and 2.9.6, and xUnit in version 1.9. So, as long as your unit testing framework is less than 3 years old, async task unit tests should just work.
Unfortunately, outdated unit test frameworks don’t understand async task unit tests. As of this writing, there’s one major platform that doesn’t support them: Xamarin. Xamarin uses a customized older version of NUnitLite, and it doesn’t currently support async task unit tests. I expect that support will be added in the near future. In the meantime, I use a workaround that’s inefficient but works: Execute the async test logic on a different thread pool thread, and then (synchronously) block the unit test method until the actual test completes. The workaround code uses GetAwaiter().GetResult() instead of Wait because Wait will wrap any exceptions inside an AggregateException:
[Test]
public void XamarinExampleTest()
{
// This workaround is necessary on Xamarin,
// which doesn't support async unit test methods.
Task.Run(async () =>
{
// Actual test code here.
}).GetAwaiter().GetResult();
}
Testing Exceptions
When testing, it’s natural to test the successful scenario; for example, a user can update his own profile. However, testing exceptions is also very important; for example, a user shouldn’t be able to update someone else’s profile. Exceptions are part of an API surface just as much as method parameters are. Therefore, it’s important to have unit tests for code when it’s expected to fail.
Originally, the ExpectedExceptionAttribute was placed on a unit test method to indicate that the unit test was expected to fail. However, there were a few problems with ExpectedExceptionAttribute. The first was that it could only expect a unit test to fail as a whole; there was no way to indicate that only a particular part of the test was expected to fail. This isn’t a problem with very simple tests, but can have misleading results when the tests grow longer. The second problem with ExpectedExceptionAttribute is that it’s limited to checking the type of the exception; there’s no way to check other attributes, such as error codes or messages.
For these reasons, in recent years there has been a shift toward using something more like Assert.ThrowsException, which takes the important part of the code as a delegate and returns the exception that was thrown. This solves the shortcomings of ExpectedExceptionAttribute. The desktop MSTest framework supports only ExpectedExceptionAttribute, while the newer MSTest framework used for Windows Store unit test projects supports only Assert.ThrowsException. xUnit supports only Assert.Throws, and NUnit supports both approaches. Figure 1 is an example of both kinds of tests, using MSTest syntax.
Figure 1 Testing Exceptions with Synchronous Test Methods
// Old style; only works on desktop.
[TestMethod]
[ExpectedException(typeof(Exception))]
public void ExampleExpectedExceptionTest()
{
SystemUnderTest.Fail();
}
// New style; only works on Windows Store.
[TestMethod]
public void ExampleThrowsExceptionTest()
{
var ex = Assert.ThrowsException<Exception>(()
=> { SystemUnderTest.Fail(); });
}
But what about asynchronous code? Async task unit tests work perfectly well with ExpectedExceptionAttribute on both MSTest and NUnit (xUnit doesn’t support ExpectedExceptionAttribute at all). However, the support for an async-ready ThrowsException is less uniform. MSTest does support an async ThrowsException, but only for Windows Store unit test projects. xUnit has introduced an async ThrowsAsync in the prerelease builds of xUnit 2.0.0.
NUnit is more complex. As of this writing, NUnit supports asynchronous code in its verification methods such as Assert.Throws. However, in order to get this to work, NUnit provides a SynchronizationContext, which introduces the same problems as async void unit tests. Also, the syntax is currently brittle, as the example in Figure 2 shows. NUnit is already planning to drop support for async void unit tests, and I wouldn’t be surprised if this support is dropped at the same time. In summary: I recommend you do not use this approach.
Figure 2 Brittle NUnit Exception Testing
[Test]
public void FailureTest_AssertThrows()
{
// This works, though it actually implements a nested loop,
// synchronously blocking the Assert.Throws call until the asynchronous
// FailAsync call completes.
Assert.Throws<Exception>(async () => await SystemUnderTest.FailAsync());
}
// Does NOT pass.
[Test]
public void BadFailureTest_AssertThrows()
{
Assert.Throws<Exception>(() => SystemUnderTest.FailAsync());
}
So, the current support for an async-ready ThrowsException/Throws isn’t great. In my own unit testing code, I use a type very similar to the AssertEx in Figure 3. This type is rather simplistic in that it just throws bare Exception objects instead of doing assertions, but this same code works in all major unit testing frameworks.
Figure 3 The AssertEx Class for Testing Exceptions Asynchronously
using System;
using System.Threading.Tasks;
public static class AssertEx
{
public static async Task<TException>
ThrowsAsync<TException>(Func<Task> action,
bool allowDerivedTypes = true) where TException : Exception
{
try
{
await action();
}
catch (Exception ex)
{
if (allowDerivedTypes && !(ex is TException))
throw new Exception("Delegate threw exception of type " +
ex.GetType().Name + ", but " + typeof(TException).Name +
" or a derived type was expected.", ex);
if (!allowDerivedTypes && ex.GetType() != typeof(TException))
throw new Exception("Delegate threw exception of type " +
ex.GetType().Name + ", but " + typeof(TException).Name +
" was expected.", ex);
return (TException)ex;
}
throw new Exception("Delegate did not throw expected exception " +
typeof(TException).Name + ".");
}
public static Task<Exception> ThrowsAsync(Func<Task> action)
{
return ThrowsAsync<Exception>(action, true);
}
}
This allows async task unit tests to use a more modern ThrowsAsync instead of the ExpectedExceptionAttribute, like this:
[TestMethod]
public async Task FailureTest_AssertEx()
{
var ex = await AssertEx.ThrowsAsync(()
=> SystemUnderTest.FailAsync());
}
Async Stubs and Mocks
In my opinion, only the simplest of code can be tested without some kind of stub, mock, fake or other such device. In this introductory article, I’ll just refer to all of these testing assistants as mocks. When using mocks, it’s helpful to program to interfaces rather than implementations. Asynchronous methods work perfectly well with interfaces; the code in Figure 4 shows how code can consume an interface with an asynchronous method.
Figure 4 Using an Asynchronous Method from an Interface
public interface IMyService
{
Task<int> GetAsync();
}
public sealed class SystemUnderTest
{
private readonly IMyService _service;
public SystemUnderTest(IMyService service)
{
_service = service;
}
public async Task<int> RetrieveValueAsync()
{
return 42 + await _service.GetAsync();
}
}
With this code, it’s easy enough to create a test implementation of the interface and pass it to the system under test. Figure 5 shows how to test the three major stub cases: asynchronous success, asynchronous failure and synchronous success. Asynchronous success and failure are the primary two scenarios for testing asynchronous code, but it’s also important to test the synchronous case. This is because the await operator behaves differently if its awaitable is already completed. The code in Figure 5 uses the Moq mocking framework to generate the stub implementations.
Figure 5 Stub Implementations for Asynchronous Code
[TestMethod]
public async Task RetrieveValue_SynchronousSuccess_Adds42()
{
var service = new Mock<IMyService>();
service.Setup(x => x.GetAsync()).Returns(() => Task.FromResult(5));
// Or: service.Setup(x => x.GetAsync()).ReturnsAsync(5);
var system = new SystemUnderTest(service.Object);
var result = await system.RetrieveValueAsync();
Assert.AreEqual(47, result);
}
[TestMethod]
public async Task RetrieveValue_AsynchronousSuccess_Adds42()
{
var service = new Mock<IMyService>();
service.Setup(x => x.GetAsync()).Returns(async () =>
{
await Task.Yield();
return 5;
});
var system = new SystemUnderTest(service.Object);
var result = await system.RetrieveValueAsync();
Assert.AreEqual(47, result);
}
[TestMethod]
public async Task RetrieveValue_AsynchronousFailure_Throws()
{
var service = new Mock<IMyService>();
service.Setup(x => x.GetAsync()).Returns(async () =>
{
await Task.Yield();
throw new Exception();
});
var system = new SystemUnderTest(service.Object);
await AssertEx.ThrowsAsync(system.RetrieveValueAsync);
}
Speaking of mocking frameworks, there’s a bit of support they can give to asynchronous unit testing, as well. Consider for a moment what the default behavior of a method should be, if no behavior was specified. Some mocking frameworks (such as Microsoft Stubs) will default to throwing an exception; others (such as Moq) will return a default value. When an asynchronous method returns a Task<T>, a naïve default behavior would be to return default(Task<T>), in other words, a null task, which will cause a NullReferenceException.
This behavior is undesirable. A more reasonable default behavior for asynchronous methods would be to return Task.FromResult(default(T))—that is, a task that’s completed with the default value of T. This enables the system under test to use the returned task. Moq implemented this style of default behavior for asynchronous methods in Moq version 4.2. To my knowledge, as of this writing, it’s the only mocking library that uses async-friendly defaults like that.
Wrapping Up
Async and await have been around since the introduction of Visual Studio 2012, long enough for some best practices to emerge. Unit test frameworks and helper components such as mocking libraries are converging toward consistent async-friendly support. Asynchronous unit testing today is already a reality, and it will get even better in the future. If you haven’t done so recently, now is a good time to update your unit test frameworks and mocking libraries to ensure you have the best async support.
Unit test frameworks are converging away from async void unit tests and toward async task unit tests. If you have any async void unit tests, I recommend you change them today to async task unit tests.
I expect over the next couple years you’ll see much better support for testing failure cases in async unit tests. Until your unit test framework has good support, I suggest you use the AssertEx type mentioned in this article, or something similar that’s more tailored to your particular unit test framework.
Proper asynchronous unit testing is an important part of the async story, and I’m excited to see these frameworks and libraries adopt async. One of my first lightning talks was about async unit testing a few years ago when async was still in community technology preview, and it’s so much easier to do these days!
Stephen Cleary is a husband, father and programmer living in northern Michigan. He has worked with multithreading and asynchronous programming for 16 years and has used async support in the Microsoft .NET Framework since the first community technology preview. He is the author of “Concurrency in C# Cookbook” (O’Reilly Media, 2014). His homepage, including his blog, is at stephencleary.com.
Thanks to the following Microsoft technical expert for reviewing this article: James McCaffrey
No comments :
Post a Comment