JUnit is probably the most popular Java testing framework. Every JUnit test uses a Runner. It’s an abstract class, which is most of all responsible for invoking test methods. Runners are pretty straightforward to be customized.

This article is not about how JUnit runners work internally – there’s plenty of great articles about that on the web already. It’s about a huge variety of Runners which are already implemented and ready to be used out of the box. I will focus only on Runners that can be declared directly inside @RunWith annotation.

I’ve prepared a simple Spring Boot application, which provides a simple REST endpoint. It displays a number of milliseconds from the epoch of 1970-01-01T00:00:00Z to the date provided by the user. If date param is not provided, it’s displaying epoch of current time taken from external api – worldclockapi.com. This application is pretty simple, but it has some implementation pitfalls, which make it a little bit more difficult to test in some cases. For testing, we will use JUnit to run tests and assertions, and Mocktito with PowerMock to mock objects and methods.

Spring Boot Application in practice

Now… Let’s begin with our app.

The most important classes are:






You’ve probably already noticed that WorldClockDateClient has a public static method instead of being declared as Spring bean… That’s correct – this is our pitfall. 🙂 Now it’s time for some testing. Let’s start with something simple – we have our DateValidator class with two static methods to check if date and time have the correct format.

Let’s create a simple unit test:

1. JUnit Runners: BlockJUnit4ClassRunner

As you can see, there is no @RunWith annotation. In fact, we are implicitly using JUnit BlockJUnit4ClassRunner located in org.junit.runners package, which is a default Runner class used for single test classes. It’s responsible for running all methods annotated with @Test, @Before, @After etc. in the correct order.

2. JUnit Runners: Parametrized

Our test is working, but there’s much more to be tested, especially in case of negative scenarios – there may be many more cases. Adding another test method for each case doesn’t make any sense. We can create a list of inputs and run assertions in a loop – better, but JUnit has a much more elegant way to create such a test. Here comes Parameterized runner:

We can prepare a list of inputs and expected results, then mark them with @Parameterized. Parameters annotation and Parameterized Runner will execute each scenario for us. This runner is internally using BlockJUnit4ClassRunnerWithParameters, which can’t be used directly out of the box (as @RunWith) – it’s created by ParametersRunnerFactory inside Parameterized runner class. This Runner can do much, much more and it’s also improved in JUnit 5. It’s just a short example to keep in mind that there is such a Runner and it’s useful for tests with more cases.

There’s also an alternative to Parameterized Runner – it’s an external library called JUnitParams. It has some great features and makes parameterized tests even simpler. You can find it (and a manual) here: https://github.com/Pragmatists/JUnitParams.

3. JUnit Runners: Suite
In our app we have one more component, which can be tested separately – DateConverter. I’ve created a simple unit test for this component:

Now we have 2 simple unit tests: DateValidatorParameterizedTest (we don’t need DateValidatorTest anymore) and DateConverterTest. If we want to make sure that those two tests are always run together, we can use suites and SuiteRunner:

The first question that comes to my mind is… WHY? Does it have any use cases? The best answer is here: https://stackoverflow.com/questions/36901680/why-use-junit-test-suites. The second question is – will Maven or Gradle run both single test classes and test suite, so there will be duplicated tests running? The answer is NO, both Maven and Gradle are ignoring tests with Suite runner by default.

3. JUnit Runners: Enclosed
We have now our DateValidatorParameterizedTest and DateConverterTest running together in suite. Both tests are using DateUtils service, and both may use the same date object for tests. It would be great if we could re-use our logic for both tests with inheritance.

There is also one problem – one of these tests is a basic BlockJUnit4ClassRunner test, the second one is using Parameterized runner. If we want to bind these tests together, Suite runner won’t be helpful. We need Enclosed runner. You can find this runner inside org.junit.experimental.runners package. It extends Suite runner and allows running our tests inside static inner classes. The main difference between Enclosed and Suite runner is that with the second one you can extend base class and share its members and init() method. The first one is simply binding classes together.

Let’s put our DateValidatorParameterizedTest and DateConverterTest inside one Enclosed test class:

As you can see, we have one base class called AbstractBaseTest and other test classes are extending this class. We can even use different Runner for each class, and the init() method and all members of our AbstractBaseTest class will be shared between other test classes.

You must remember that static classes have some limitations, for example I can’t create our testDate from AbstractBaseTest inside init() method, or make it non static. Making it non static won’t even compile, because of @Parameterized. Parameters method used inside DateValidatorParameterizedTest must be static. Changing our testDate to static and initializing it inside @ init() method will cause NullPointerException, because our static dates() method from DateValidatorParameterizedTest will be executed before non static init() method from AbstractBaseTest which initializes our testDate object.

4. Mockito Runners: MockitoJUnitRunner
Now it’s time to test one of our Spring beans, called DateToEpochMilliConvertingService with Mockito. MockitoJUnitRunner is the default Mockito runner, which allows us to run tests with mocked objects.

Everything goes well, except the last test: convertWorldClockDateToEpochMilliTest(). We are testing DateToEpochMilliConvertingService, which is doing a real call to external API:

Unit tests shouldn’t be dependent on external API. The second problem is that WorldClockDateClient.getWorldClockDate() is a static method, which we can’t mock with the Mockito framework. So what now?

5. PowerMock Runners: PowerMockRunner
PowerMock is a framework which helps with cases where other mocking frameworks, like Mockito, fail. It mocks static methods and does much more:

PowerMock uses a custom classloader and bytecode manipulation to enable mocking of static methods, constructors, final classes and methods, private methods, removal of static initializers and more.

The first question that comes to my mind is: OK, let’s use PowerMock then, but our test is already using MockitoJUnitRunner. We can’t declare two runners at the same time, so what now?

The solution is very easy: PowerMock has @PowerMockRunnerDelegate annotation. You can use it to delegate PowerMock to any other runner you want, but before that PowerMock will use it’s own runner to do it’s own mocking magic. Here’s the code:

As you can see now we’ve got rid of call to external API from convertWorldClockDateToEpochMilliTest by mocking static method:

The rest of the tests remain the same. We can also test what happens with our service when external API is offline by mocking our WorldClockDateClient to throw an exception, just like with Mockito.The only problem that you will probably face is that PowerMock combined with Mockito sometimes has compatibility issues. It may throw some exceptions and report missing internal classes etc. You need to test which version of Mockito is compatible with which version of PowerMock.

6. Spring Runners: SpringJUnit4ClassRunner aka SpringRunner
Our application is Spring Boot based, so let’s use Spring Runners for our tests. SpringRunner is the base Spring framework Runner. It extends SpringJUnit4ClassRunner, but that’s all – it’s just an alias:

Spring framework developers try to make things as simple as possible, so probably that was the reason to change the name of the Runner from SpringJUnit4ClassRunner to SpringRunner with backward compatibility. Here’s our test:

We already know how to handle static methods, so I’ve simplified our test, and it has only two simple tests just to explain how SpringRunner works. @SpringBootTest annotation is used to start Spring ApplicationContext, and @MockBean annotation is used to create Mockito mocks and put them into a Spring ApplicationContext container before our test starts.

We can also use @Autowire to inject beans. Now our DateUtils class is a bean located in Spring ApplicationContext and it’s injected into our DateToEpochMilliConvertingService bean just like it would happen when we run our application. Now we can test our components as a part of the Spring Boot application. This kind of Runner is better for integration tests than unit tests, because it’s running the whole Spring container, with all other beans, so it’s more memory consuming and takes much more time to launch the test.

Spring provides one SpringRunner, which can be combined with a few more annotations, which will automatically run different test scenarios, depending on what we want to test: @SpringBootTest, @JsonTest, @WebMvcTest, @DataJpaTest etc. You can read more about it here: https://docs.spring.io/spring-boot/docs/1.5.2.RELEASE/reference/html/boot-features-testing.html.

7. Let’s combine our runners together
We have tested most of the common runners provided by JUnit, Mockito, PowerMock and Spring framework. Now it’s time for a final “end to end” test. We will test the whole application, by making requests directly to our REST API.

The idea is to create one Enclosed test with two static test classes:

  • EndToEndSpringRunnerWithParameterizedRunnerTest: it will test our API with the “date” parameter as parameterized test. It’s using a Parameterized runner combined with SpringRunner.
  • EndToEndSPowerMockRunnerWithSpringRunnerTest: it will call our API without the “date” parameter, which results in calling external world clock api.

This test will have two methods: one will let our api to call external API, the second one will mock it. Here we have PowerMockRunner, delegated to SpringRunner.Now we have a huge mix of runners: main EndToEndTest class annotated with @RunWith(Enclosed.class), which contains an abstract class AbstractBaseTest annotated with @SpringBootTest. Then we have two classes extending AbstractBaseTest using PowerMockRunner and Parameterized runner. So here’s the code:

When I was writing this test, I was pretty sure it wouldn’t work like this. There are many different frameworks combined together, we also have inheritance, but… it worked. 🙂

There were some problems, but all of them turned out to be solvable. First test class was executed successfully, but then the second one was failing, because TomcatURLStreamHandlerFactory has a method setURLStreamHandlerFactory, which can be executed only once. Otherwise it will throw an exception. This can be easily fixed, by disabling the default TomcatURLStreamHandlerFactory before starting the Tomcat server instance:

Another problem was with EndToEndSpringRunnerWithParameterizedRunnerTest. Enclosed runner test, with the SpringBootTest and Parameterized combined all together was too much. We can use only one @RunWith annotation and here we had to use SpringRunner and Parameterized runner together. Solution was simple – we can use both @ClassRule and @Rule to use SpringRunner inside Parameterized runner.

The last problem was with EndToEndSPowerMockRunnerWithSpringRunnerTest. PowerMockRunner delegated to SpringRunner caused Spring ApplicationContext fail to load, because of:

java.lang.LinkageError: loader constraint violation: when resolving method ‘javax.management.MBeanServer java.lang.management.ManagementFactory.getPlatformMBeanServer()’ the class loader org.powermock.core.classloader.javassist.JavassistMockClassLoader @4148db48 of the current class, org/apache/tomcat/util/modeler/Registry, and the class loader ‘bootstrap’ for the method’s defining class, java/lang/management/ManagementFactory, have different Class objects for the type javax/management/MBeanServer used in the signature (org.apache.tomcat.util.modeler.Registry is in unnamed module of loader org.powermock.core.classloader.javassist.JavassistMockClassLoader @4148db48, parent loader ‘app’; java.lang.management.ManagementFactory is in module java.management of loader ‘bootstrap’)

Sounds like a huge problem, but it turned out to be easy to fix. There was some mismatch between PowerMock and Spring dependencies. We can simply tell PowerMock to ignore this dependencies with this annotation:

After fixing all these issues I could run all my tests together, with use of all runners that could be helpful in this case. Mixing runners turned out to be possible, even with inheritance from annotated classes. The only problems you may face in future is a mismatch between dependencies used in different versions of testing frameworks.

You can find the whole project here: https://gitlab.codete.com/ryszard.kusnierczyk/junit-runners-test.


Java Software Engineer at Codete, who really likes testing, mocking and using reflection.