The best bug is a bug that does not enter the software, and according to our experience a proper first level of testing (aka Unit Testing) will keep the bugs out. The earlier you can detect bugs on this level, the less time you loose with debugging, especially in the final stages before the release. Over the years we have developed numerous unit tests and would, based on our experience, recommend to watch out for the following criteria. This will make sure that the unit tests are tight enough on the one hand, and that you do not overtest at the same time, which would also results in waste. Also we would encourage you to keep your code easily testable, so you can find defects fast and effortlessly.
Here is an overview over the guidelines for testable code and the criteria for good unit tests that we have established in our company and in our clients’ projects. We mostly use Java to develop solid code so the examples are written in Java but they should be easily transferable to any other imperative programming language like C#, GoLang, …
Writing Testable Code
The first question that is always important to answer is: is the code testable. Whether you’re writing a new software from scratch or if you’re just about to add some functionality to a an existing project, you should make sure that the code that you’re writing doesn’t make it more cumbersome than necessary to being tested.
The following requirements are important if you want to write properly testable code:
Separation of Concerns
You might have heard this before, but the most powerful tool in our arsenal is modularization. This not only allows us to keep us from duplicating and spreading code all over the place, but this also makes the code concise, readable, maintainable and easy to test!
Each and every Unit should take care of just one thing. So avoid methods like cleanStringAndPersist() but rather have two separate methods cleanString() and persist(). Or even better have a Utility class for the cleanString() and a separate repository class to handle the persisting of data.
Furthermore we need to make sure that classes (aka units) describe all external dependencies that they have. The best way to do this is to simply state all external dependencies that the unit requires in the constructor:
1
2
3
4
5
6
7
8
9
public class Unit {
private final Repository repository;
private final SomeService someService;
public Unit(Repository repository, SomeService someService) {
this.repository = repository;
this.someService = someService;
}
}
Instead of having the unit search for the dependency itself:
1
2
3
4
5
6
7
8
9
public class Unit {
private final Repository repository;
private final SomeService someService;
public Unit(Context context) {
repository = context.findRepository();
someService = context.getServices().getSomeService();
}
}
Letting the unit search for its dependencies in some context object makes it hard for us to create a test later on, as we need to know about the internal workings of the unit. Ideally you always want to create a test that is not dependent on the internal workings of the unit that you are testing. That way the test stays the same even if the implementation gets revised or changes completely.
Also make sure that you have no hidden dependencies in your code as they are even harder to identify and mock in a test. So avoid things like this:
1
2
3
4
5
6
7
8
9
public class Unit {
// ...
public createNewUser(UserDto userDto) {
UserEntity userEntity = translator.translate(userDto);
DB db = DB.getInstance();
db.persist(userEntity);
}
}
The external dependency to the DB instance is completely hidden as we acquire it using the Singleton Pattern. If you need to ensure that there is only one instance of a particular class available in your code, you should instead consider using the singleton scope within a dependency injection framework.
Public Methods
In most modern programming languages you have visibility modifiers to limit who can see and therefore access your API. In Java those are package, protected, private and public. For testing the most important ones are methods marked with the latter one: public. You should always treat those methods as contracts between you and a potential user of your method. Ideally all public methods need to be tested because those are the ones that other parts of your software or even other software might use (with or without you knowing about it). This is especially true when you’re writing a library.
Make sure to document all your publicly accessible methods well (by either a speaking method name and/or by adding a proper JavaDoc comment). It should be crystal clear what the method does, what the method expects you to provide and what it will return (or that it returns nothing). A public method is a contract and a proper unit test should then cover all these scenarios that are described in that contract.
Dependency Injection
A lot of modern languages offer some way of dependency injection, most of the time through third party libraries. In Java the Spring framework for example offers the most well known implementation of dependency injection. It is always good idea to use dependency injection and to make code properly testable you should favor dependency injection via constructor like this:
1
2
3
4
5
6
7
8
9
@Service
public class Unit {
private final Repository repository;
@Autowired
public Unit(Repository repository) {
this.repository = repository;
}
}
In comparison to field injection, this has the advantage that in our unit test we don’t need to use more complicated mocking techniques like the @Inject annotation that requires the full dependency injection framework to start up. Instead we simply can construct the Unit with the mocks directly by instantiating the Unit ourselves and providing the mocks in the constructor:
1
2
3
4
5
6
7
8
9
10
11
12
13
class UnitTest {
private Unit unit;
private Repository repositoryMock;
@BeforeEach
void init() {
repositoryMock = Mockito.mock(Repository.class);
unit = new Unit(repositoryMock);
}
// individual tests ...
}
Criteria for proper unit tests
1. Proper Initialization & Cleanup
Before any test can happen, you need to initialize the test. Most of the time it is a good idea to set up the test by instantiating the unit that should be tested and by wiring all the mocks:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
class UnitTest {
private Unit unit;
private Repository repositoryMock;
private Path tmpFile;
@BeforeEach
void init() throws IOException {
tmpFile = Files.createTempFile("unit-test", ".dat");
repositoryMock = Mockito.mock(Repository.class);
unit = new Unit(tmpFile, repositoryMock);
}
// individual tests ...
@AfterEach
void cleanup() {
// make sure to clean up after each test
tmpFile.getFile().delete();
}
}
2. One Unit & Behavior per Test Case
For unit tests avoid writing test cases that test more than one behavior at a time or that even test more than one unit (those are called Integration Tests and are a different breed).
Testing multiple things at the same time leads to problems when a bug was introduced and one test is falling. If you separate your tests properly, then you’ll know instantly from what test is failing what behavior broke. In case you are testing multiple things in one test you have no idea which scenario did not work anymore.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
class UnitTest {
private Unit unit;
@BeforeEach
void init() {
unit = new Unit(/* ... */);
}
@Test
void cleanString_stringWithNonValidCharsAndLeadingSpaces_expectCleanedString() {
String result = unit.cleanString("Hello-Wörld! ");
assertEquals("Hello Wrld", result);
}
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
class UnitTest {
private Unit unit;
@BeforeEach
void init() {
unit = new Unit(/* ... */);
}
@Test
void cleanString_stringWithLeadingSpaces_expectReturnsStringWithoutLeadingSpaces() {
String result = unit.cleanString(" Hello World")
assertEquals("Hello World", result);
}
@Test
void cleanString_stringWithNonAcceptedChars_expectReturnsStringWithoutNonExceptedChars() {
String result = unit.cleanString("Hello Wörld!!!!")
assertEquals("Hello Wrld", result);
}
}
3. Mocking of All Dependencies
As we’ve seen before in the section about how to write testable code: writing the code in a way so that it adheres to the Separation of Concerns helps us tremendously when writing the unit tests. We know all external dependencies of the unit and can easily mock them:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
class UnitTest {
private Unit unit;
private Repository repositoryMock;
private OtherService otherServiceMock;
@BeforeEach
void init() throws IOException {
repositoryMock = Mockito.mock(Repository.class);
otherServiceMock = Mockito.mock(OtherService.class);
unit = new Unit(repositoryMock, otherService);
}
// individual tests ...
}
Try to avoid instantiating the dependencies yourself, as this effectively makes your test an Integration Test.
1
2
3
4
5
6
7
8
9
10
11
class UnitTest {
private Unit unit;
@BeforeEach
void init() throws IOException {
unit = new Unit(new Repository(/* ... */), new OtherService(/* ... */));
}
// individual tests ...
}
If you initialize the dependencies yourself you have no control over what is happening and in case of a failed test you won’t be able to tell where the problem resides.
4. Use Speaking Names For Test Cases
When running test cases the results usually get compiled into a nice report. To be able to identify the problematic unit at first glance, it is advisable to give the test methods descriptive names.
What naming scheme works best for you and your team is often times a matter of taste, but there are two possibilities I want to show here:
- Use a functional description (after Roy Osherove1) like
[UnitOfWork_StateUnderTest_ExpectedBehavior]to name your tests. So e.g.multiply_twoNumbers_expectReturnsProduct()orcleanString_null_expectThrowsIllegalArgumentException(). - Human readable description - use a name in which you describe exactly what you try to do.So e.g.
post_comment_stores_command_in_db(),send_valid_data_package()orcleaning_null_string_is_not_possible(). This is the more modern approach and has the advantage that even non-programmers are able to understand what went wrong when reading the unit test report.
5. Cover All Happy & Failing Paths
Of all the criteria for good unit tests this is definitely one of the most important. You should always test all happy paths and all failing paths. I’ve seen a lot of tests where only the happy paths got tested and sometimes even just a few of them and not all of them. Remember what we discussed in the paragraph about Public Methods? Public methods are contracts and you should treat them as such when testing them.
Consider the following unit where you want to test the cleanString() method of:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
public class Unit {
/**
* Cleans the string by trimming it and removing
* all characters that are not part of A-Z, a-z, 0-9
* and space
*
* @param string the string to clean
* @return the cleaned string
* @throws IllegalArgumentException in case the argument is null
*/
public String cleanString(String string) {
if (string == null) throw new IllegalArgumentException("provided string is null");
return string.trim().replaceAll("[^A-Z^a-z^0-9^\\s]+", "");
}
}
In your unit test you should test all of these scenarios:
- does the method return a string that has no spaces in the beginning or the end
- does the method return a string that does not contain any other characters apart from
A-Z,a-z,0-9andspace - does the method throw an
IllegalArgumentExceptionin case the string isnull
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
class UnitTest {
private Unit unit;
@BeforeEach
void init() throws IOException {
unit = new Unit();
}
@Test
void cleanString_stringWithLeadingSpaces_expectReturnsStringWithoutLeadingSpaces() {
String result = unit.cleanString(" Hello World ")
assertEquals("Hello World", result);
}
@Test
void cleanString_stringWithNonAcceptedChars_expectReturnsStringWithoutNonExceptedChars() {
String result = unit.cleanString("Hello Wörld!!!!")
assertEquals("Hello Wrld", result);
}
@Test
void cleanString_nullString_expectThrowsIllegalArgumentException() {
assertThrows(IllegalArgumentException.class, () -> unit.cleanString(null));
}
}
6. Cover the Full Range of Parameter Values
We partially touched base on this already in #5 but this is too important so it got its own criteria. Make sure that you always test all possible parameter values. Of course you should not literally try to check all the values for all parameters in your tests (e.g. for certain types this might not even be possible - e.g. for Strings or non-primitive data types). But imagine you have a method that takes an int as parameter - make sure to also check what happens if you hand over a negative number, or Integer.MAX_VALUE. With this you try to find the corner cases because experience taught us that if something fails its most of the time the cases that the developers did not think of.
Don’t just write one test case that covers a typical value the user would provide.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
class UnitTest {
private Unit unit;
@BeforeEach
void init() throws IOException {
unit = new Unit();
}
@Test
void fibonacci_for10_expectReturns55() {
int result = unit.fibonacci(10);
assertEquals(55, result);
}
}
Better also cover all corner cases that value could assume (like 0 and negative values e.g.):
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
class UnitTest {
private Unit unit;
@BeforeEach
void init() throws IOException {
unit = new Unit();
}
@Test
void fibonacci_for10_expectReturns55() {
int result = unit.fibonacci(10);
assertEquals(55, result);
}
@Test
void fibonacci_for0_expectReturns0() {
int result = unit.fibonacci(0);
assertEquals(0, result);
}
@Test
void fibonacci_forNegativeValue_expectThrowsIllegalArgumentException() {
assertThrows(IllegalArgumentException.class, () -> unit.fibonacci(-1));
}
}
7. Check If Exceptions Get Thrown
Again I need to refer back to the section about Public Methods: public methods should be considered contracts! That means that if you specify that a methods throws an exception under a certain condition, you also need to check that it does.
Consider you have this unit that you want to write test cases for:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
public class Unit {
/**
* Cleans the string by trimming it and removing
* all characters that are not part of A-Z, a-z, 0-9
* and space
*
* @param string the string to clean
* @return the cleaned string
* @throws IllegalArgumentException in case the argument is null
*/
public String cleanString(String string) {
if (string == null) throw new IllegalArgumentException("provided string is null");
return string.trim().replaceAll("[^A-Z^a-z^0-9^\\s]+", "");
}
}
Make sure to include a test that checks that the defined IllegalArgumentException is actually thrown under the condition that is specified in the documentation (in this case when a null value is passed as parameter):
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
class UnitTest {
private Unit unit;
@BeforeEach
void init() throws IOException {
unit = new Unit();
}
@Test
void cleanString_nullString_expectThrowsIllegalArgumentException() {
assertThrows(IllegalArgumentException.class, () -> unit.cleanString(null));
}
// other tests ...
}
8. Use Assertions & Verifications
To most developer this shouldn’t come as a surprise: you should not only test that there was not exception thrown in your unit test but you should also check that the result is what you’d expect. And for whitebox tests you can also check that certain mocked methods have been called. You wouldn’t believe how often tests are written just to get the code coverage up without and assert or verify statement.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
class UnitTest {
private Unit unit;
@BeforeEach
void init() throws IOException {
unit = new Unit();
}
@Test
void cleanString_stringWithLeadingSpaces_expectReturnsStringWithoutLeadingSpaces() {
unit.cleanString(" Hello World");
// if we get this far we can be sure that the method works
}
// other tests ...
}
Don’t just call the method and check if it didn’t throw an exception. Instead assert that the correct result was returned:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
class UnitTest {
private Unit unit;
@BeforeEach
void init() throws IOException {
unit = new Unit();
}
@Test
void cleanString_stringWithLeadingSpaces_expectReturnsStringWithoutLeadingSpaces() {
String result = unit.cleanString(" Hello World")
assertEquals("Hello World", result);
}
// other tests ...
}
9. Keep Tests Repeatable
Nothing is more annoying than a test that sometimes fails and sometimes doesn’t. While it is a good idea to test multiple parameter value permutations using a randome generator, make sure to initialize it with a constant seed. That way the random numbers it creates during a test case will always stay the same.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
class UnitTest {
private Unit unit;
@BeforeEach
void init() throws IOException {
unit = new Unit();
}
@Test
void createUser_1000randomNames_expectUsersBeingCreated() {
var random = new Random(); // <- no seed - no good!
for (int i = 0; i < 1000; ++i) {
String randomFirstName = createRandomName(random);
String randomLastName = createRandomName(random);
User user = unit.createUser(new User(randomFirstName, randomLastName));
assertEquals(i + 1, user.getId());
assertEquals(randomFirstName, user.getFirstName());
assertEquals(randomLastName, user.getLastName());
}
}
// other tests ...
}
Assume now that there is a bug in createUser() that throws an exception when a user’s first name starts contains an umlaut. This will lead to this bug sometimes appearing and sometimes not. These things make software developers believe in ghosts. Better use a fixed seed so that the set of names generated will always be the same!
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
class UnitTest {
private Unit unit;
@BeforeEach
void init() throws IOException {
unit = new Unit();
}
@Test
void createUser_1000randomNames_expectUsersBeingCreated() {
var random = new Random(1234L); // fixed seed = fixed random numbers (names)!
for (int i = 0; i < 1000; ++i) {
String randomFirstName = createRandomName(random);
String randomLastName = createRandomName(random);
User user = unit.createUser(new User(randomFirstName, randomLastName));
assertEquals(i + 1, user.getId());
assertEquals(randomFirstName, user.getFirstName());
assertEquals(randomLastName, user.getLastName());
}
}
// other tests ...
}
10. Keep Runtime of Test In Check
Nowadays most software development projects make use of CI/CD pipelines to make sure that a new feature does not break existing functionality. This means that all unit tests will be executed each and every time you push a change, open a merge request, something gets merged, or before the software gets deployed. Therefore it is advisable to try to keep the time complexity in check and in general not to waste a lot of time for a test run when developing test cases.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
class UnitTest {
private Unit unit;
@BeforeEach
void init() throws IOException {
unit = new Unit();
}
@Test
void createUser_1000randomNames_expectUsersBeingCreated() {
var random = new Random(1234L);
for (int i = 0; i < 1_000_000; ++i) { // <- one million iterations might not really be required
String randomFirstName = createRandomName(random);
String randomLastName = createRandomName(random);
User user = unit.createUser(new User(randomFirstName, randomLastName));
assertEquals(i + 1, user.getId());
assertEquals(randomFirstName, user.getFirstName());
assertEquals(randomLastName, user.getLastName());
}
}
// other tests ...
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
class UnitTest {
private Unit unit;
@BeforeEach
void init() throws IOException {
unit = new Unit();
}
@Test
void createUser_1000randomNames_expectUsersBeingCreated() {
var random = new Random(1234L);
for (int i = 0; i < 1_000; ++i) { // <- one thousand iterations is enough
String randomFirstName = createRandomName(random);
String randomLastName = createRandomName(random);
User user = unit.createUser(new User(randomFirstName, randomLastName));
assertEquals(i + 1, user.getId());
assertEquals(randomFirstName, user.getFirstName());
assertEquals(randomLastName, user.getLastName());
}
}
// other tests ...
}
Further Reading
This is just an overview over the best practice that we’ve established in our company. We used our experience and input from the following books to compile this document. If you need more input or want to dive deeper into the subject, here are our sources:
-
https://osherove.com/blog/2005/4/3/naming-standards-for-unit-tests.html ↩
-
Petar Tahchiev, Felipe Leme, Vincent Massol & Gary Gregory: JUnit in Action ↩
-
Jeff Langr, Andy Hunt, Dave Thomas: Pragmatic Unit Testing in Java 8 with JUnit ↩
-
Lee Copeland: A Practitioner’s Guide to Software Test Design ↩