6.2. JUnit Platform Test Kit

The junit-platform-testkit artifact provides support for executing a test plan on the JUnit Platform and then verifying the expected results. As of JUnit Platform 1.4, this support is limited to the execution of a single TestEngine (see Engine Test Kit).

Although the Test Kit is currently an experimental feature, the JUnit Team invites you to try it out and provide feedback to help improve the Test Kit APIs and eventually promote this feature.

6.2.1. Engine Test Kit

The [org.junit.platform.testkit.engine](https://junit.org/junit5/docs/current/api/org.junit.platform.testkit/org/junit/platform/testkit/engine/package-summary.html) package provides support for executing a [TestPlan](https://junit.org/junit5/docs/current/api/org.junit.platform.launcher/org/junit/platform/launcher/TestPlan.html) for a given [TestEngine](https://junit.org/junit5/docs/current/api/org.junit.platform.engine/org/junit/platform/engine/TestEngine.html) running on the JUnit Platform and then accessing the results via a fluent API to verify the expected results. The key entry point into this API is the [EngineTestKit](https://junit.org/junit5/docs/current/api/org.junit.platform.testkit/org/junit/platform/testkit/engine/EngineTestKit.html) which provides static factory methods named engine() and execute(). It is recommended that you select one of the engine() variants to benefit from the fluent API for building a LauncherDiscoveryRequest.

If you prefer to use the LauncherDiscoveryRequestBuilder from the Launcher API to build your LauncherDiscoveryRequest, you must use one of the execute() variants in EngineTestKit.

The following test class written using JUnit Jupiter will be used in subsequent examples.

  1. import static org.junit.jupiter.api.Assertions.assertEquals;
  2. import static org.junit.jupiter.api.Assumptions.assumeTrue;
  4. import example.util.Calculator;
  6. import org.junit.jupiter.api.Disabled;
  7. import org.junit.jupiter.api.MethodOrderer.OrderAnnotation;
  8. import org.junit.jupiter.api.Order;
  9. import org.junit.jupiter.api.Test;
  10. import org.junit.jupiter.api.TestMethodOrder;
  12. @TestMethodOrder(OrderAnnotation.class)
  13. public class ExampleTestCase {
  15.     private final Calculator calculator = new Calculator();
  17.     @Test
  18.     @Disabled("for demonstration purposes")
  19.     @Order(1)
  20.     void skippedTest() {
  21.         // skipped ...
  22.     }
  24.     @Test
  25.     @Order(2)
  26.     void succeedingTest() {
  27.         assertEquals(42, calculator.multiply(6, 7));
  28.     }
  30.     @Test
  31.     @Order(3)
  32.     void abortedTest() {
  33.         assumeTrue("abc".contains("Z"), "abc does not contain Z");
  34.         // aborted ...
  35.     }
  37.     @Test
  38.     @Order(4)
  39.     void failingTest() {
  40.         // The following throws an ArithmeticException: "/ by zero"
  41.         calculator.divide(1, 0);
  42.     }
  44. }

For the sake of brevity, the following sections demonstrate how to test JUnit’s own JupiterTestEngine whose unique engine ID is "junit-jupiter". If you want to test your own TestEngine implementation, you need to use its unique engine ID. Alternatively, you may test your own TestEngine by supplying an instance of it to the EngineTestKit.engine(TestEngine) static factory method.

6.2.2. Asserting Statistics

One of the most common features of the Test Kit is the ability to assert statistics against events fired during the execution of a TestPlan. The following tests demonstrate how to assert statistics for containers and tests in the JUnit Jupiter TestEngine. For details on what statistics are available, consult the Javadoc for [EventStatistics](https://junit.org/junit5/docs/current/api/org.junit.platform.testkit/org/junit/platform/testkit/engine/EventStatistics.html).

  1. import static org.junit.platform.engine.discovery.DiscoverySelectors.selectClass;
  3. import example.ExampleTestCase;
  5. import org.junit.jupiter.api.Test;
  6. import org.junit.platform.testkit.engine.EngineTestKit;
  8. class EngineTestKitStatisticsDemo {
  10.     @Test
  11.     void verifyJupiterContainerStats() {
  12.         EngineTestKit
  13.             .engine("junit-jupiter") (1)
  14.             .selectors(selectClass(ExampleTestCase.class)) (2)
  15.             .execute() (3)
  16.             .containerEvents() (4)
  17.             .assertStatistics(stats -> stats.started(2).succeeded(2)); (5)
  18.     }
  20.     @Test
  21.     void verifyJupiterTestStats() {
  22.         EngineTestKit
  23.             .engine("junit-jupiter") (1)
  24.             .selectors(selectClass(ExampleTestCase.class)) (2)
  25.             .execute() (3)
  26.             .testEvents() (6)
  27.             .assertStatistics(stats ->
  28.                 stats.skipped(1).started(3).succeeded(1).aborted(1).failed(1)); (7)
  29.     }
  31. }
1Select the JUnit Jupiter TestEngine.
2Select the ExampleTestCase test class.
3Execute the TestPlan.
4Filter by container events.
5Assert statistics for container events.
6Filter by test events.
7Assert statistics for test events.
In the verifyJupiterContainerStats() test method, the counts for the started and succeeded statistics are 2 since the JupiterTestEngine and the ExampleTestCase class are both considered containers.

6.2.3. Asserting Events

If you find that asserting statistics alone is insufficient for verifying the expected behavior of test execution, you can work directly with the recorded [Event](https://junit.org/junit5/docs/current/api/org.junit.platform.testkit/org/junit/platform/testkit/engine/Event.html) elements and perform assertions against them.

For example, if you want to verify the reason that the skippedTest() method in ExampleTestCase was skipped, you can do that as follows.

The assertThatEvents() method in the following example is a shortcut for org.assertj.core.api.Assertions.assertThat(events.list()) from the AssertJ assertion library.

For details on what conditions are available for use with AssertJ assertions against events, consult the Javadoc for EventConditions.

  1. import static org.junit.platform.engine.discovery.DiscoverySelectors.selectMethod;
  2. import static org.junit.platform.testkit.engine.EventConditions.event;
  3. import static org.junit.platform.testkit.engine.EventConditions.skippedWithReason;
  4. import static org.junit.platform.testkit.engine.EventConditions.test;
  6. import example.ExampleTestCase;
  8. import org.junit.jupiter.api.Test;
  9. import org.junit.platform.testkit.engine.EngineTestKit;
  10. import org.junit.platform.testkit.engine.Events;
  12. class EngineTestKitSkippedMethodDemo {
  14.     @Test
  15.     void verifyJupiterMethodWasSkipped() {
  16.         String methodName = "skippedTest";
  18.         Events testEvents = EngineTestKit (5)
  19.             .engine("junit-jupiter") (1)
  20.             .selectors(selectMethod(ExampleTestCase.class, methodName)) (2)
  21.             .execute() (3)
  22.             .testEvents(); (4)
  24.         testEvents.assertStatistics(stats -> stats.skipped(1)); (6)
  26.         testEvents.assertThatEvents() (7)
  27.             .haveExactly(1, event(test(methodName),
  28.                 skippedWithReason("for demonstration purposes")));
  29.     }
  31. }
1Select the JUnit Jupiter TestEngine.
2Select the skippedTest() method in the ExampleTestCase test class.
3Execute the TestPlan.
4Filter by test events.
5Save the test Events to a local variable.
6Optionally assert the expected statistics.
7Assert that the recorded test events contain exactly one skipped test named skippedTest with “for demonstration purposes” as the reason.

If you want to verify the type of exception thrown from the failingTest() method in ExampleTestCase, you can do that as follows.

For details on what conditions are available for use with AssertJ assertions against events and execution results, consult the Javadoc for EventConditions and TestExecutionResultConditions, respectively.

  1. import static org.junit.platform.engine.discovery.DiscoverySelectors.selectClass;
  2. import static org.junit.platform.testkit.engine.EventConditions.event;
  3. import static org.junit.platform.testkit.engine.EventConditions.finishedWithFailure;
  4. import static org.junit.platform.testkit.engine.EventConditions.test;
  5. import static org.junit.platform.testkit.engine.TestExecutionResultConditions.instanceOf;
  6. import static org.junit.platform.testkit.engine.TestExecutionResultConditions.message;
  8. import example.ExampleTestCase;
  10. import org.junit.jupiter.api.Test;
  11. import org.junit.platform.testkit.engine.EngineTestKit;
  13. class EngineTestKitFailedMethodDemo {
  15.     @Test
  16.     void verifyJupiterMethodFailed() {
  17.         EngineTestKit.engine("junit-jupiter") (1)
  18.             .selectors(selectClass(ExampleTestCase.class)) (2)
  19.             .execute() (3)
  20.             .testEvents() (4)
  21.             .assertThatEvents().haveExactly(1, (5)
  22.                 event(test("failingTest"),
  23.                     finishedWithFailure(
  24.                         instanceOf(ArithmeticException.class), message("/ by zero"))));
  25.     }
  27. }
1Select the JUnit Jupiter TestEngine.
2Select the ExampleTestCase test class.
3Execute the TestPlan.
4Filter by test events.
5Assert that the recorded test events contain exactly one failing test named failingTest with an exception of type ArithmeticException and an error message equal to “/ by zero”.

Although typically unnecessary, there are times when you need to verify all of the events fired during the execution of a TestPlan. The following test demonstrates how to achieve this via the assertEventsMatchExactly() method in the EngineTestKit API.

Since assertEventsMatchExactly() matches conditions exactly in the order in which the events were fired, ExampleTestCase has been annotated with @TestMethodOrder(OrderAnnotation.class) and each test method has been annotated with @Order(…​). This allows us to enforce the order in which the test methods are executed, which in turn allows our verifyAllJupiterEvents() test to be reliable.

If you want to do a partial match with or without ordering requirements, you can use the methods assertEventsMatchLooselyInOrder() and assertEventsMatchLoosely(), respectively.

  1. import static org.junit.platform.engine.discovery.DiscoverySelectors.selectClass;
  2. import static org.junit.platform.testkit.engine.EventConditions.abortedWithReason;
  3. import static org.junit.platform.testkit.engine.EventConditions.container;
  4. import static org.junit.platform.testkit.engine.EventConditions.engine;
  5. import static org.junit.platform.testkit.engine.EventConditions.event;
  6. import static org.junit.platform.testkit.engine.EventConditions.finishedSuccessfully;
  7. import static org.junit.platform.testkit.engine.EventConditions.finishedWithFailure;
  8. import static org.junit.platform.testkit.engine.EventConditions.skippedWithReason;
  9. import static org.junit.platform.testkit.engine.EventConditions.started;
  10. import static org.junit.platform.testkit.engine.EventConditions.test;
  11. import static org.junit.platform.testkit.engine.TestExecutionResultConditions.instanceOf;
  12. import static org.junit.platform.testkit.engine.TestExecutionResultConditions.message;
  14. import java.io.StringWriter;
  15. import java.io.Writer;
  17. import example.ExampleTestCase;
  19. import org.junit.jupiter.api.Test;
  20. import org.junit.platform.testkit.engine.EngineTestKit;
  21. import org.opentest4j.TestAbortedException;
  23. class EngineTestKitAllEventsDemo {
  25.     @Test
  26.     void verifyAllJupiterEvents() {
  27.         Writer writer = // create a java.io.Writer for debug output
  29.         EngineTestKit.engine("junit-jupiter") (1)
  30.             .selectors(selectClass(ExampleTestCase.class)) (2)
  31.             .execute() (3)
  32.             .allEvents() (4)
  33.             .debug(writer) (5)
  34.             .assertEventsMatchExactly( (6)
  35.                 event(engine(), started()),
  36.                 event(container(ExampleTestCase.class), started()),
  37.                 event(test("skippedTest"), skippedWithReason("for demonstration purposes")),
  38.                 event(test("succeedingTest"), started()),
  39.                 event(test("succeedingTest"), finishedSuccessfully()),
  40.                 event(test("abortedTest"), started()),
  41.                 event(test("abortedTest"),
  42.                     abortedWithReason(instanceOf(TestAbortedException.class),
  43.                         message(m -> m.contains("abc does not contain Z")))),
  44.                 event(test("failingTest"), started()),
  45.                 event(test("failingTest"), finishedWithFailure(
  46.                     instanceOf(ArithmeticException.class), message("/ by zero"))),
  47.                 event(container(ExampleTestCase.class), finishedSuccessfully()),
  48.                 event(engine(), finishedSuccessfully()));
  49.     }
  51. }
1Select the JUnit Jupiter TestEngine.
2Select the ExampleTestCase test class.
3Execute the TestPlan.
4Filter by all events.
5Print all events to the supplied writer for debugging purposes. Debug information can also be written to an OutputStream such as System.out or System.err.
6Assert all events in exactly the order in which they were fired by the test engine.

The debug() invocation from the preceding example results in output similar to the following.

  1. All Events:
  2.     Event [type = STARTED, testDescriptor = JupiterEngineDescriptor: [engine:junit-jupiter], timestamp = 2018-12-14T12:45:14.082280Z, payload = null]
  3.     Event [type = STARTED, testDescriptor = ClassTestDescriptor: [engine:junit-jupiter]/[class:example.ExampleTestCase], timestamp = 2018-12-14T12:45:14.089339Z, payload = null]
  4.     Event [type = SKIPPED, testDescriptor = TestMethodTestDescriptor: [engine:junit-jupiter]/[class:example.ExampleTestCase]/[method:skippedTest()], timestamp = 2018-12-14T12:45:14.094314Z, payload = 'for demonstration purposes']
  5.     Event [type = STARTED, testDescriptor = TestMethodTestDescriptor: [engine:junit-jupiter]/[class:example.ExampleTestCase]/[method:succeedingTest()], timestamp = 2018-12-14T12:45:14.095182Z, payload = null]
  6.     Event [type = FINISHED, testDescriptor = TestMethodTestDescriptor: [engine:junit-jupiter]/[class:example.ExampleTestCase]/[method:succeedingTest()], timestamp = 2018-12-14T12:45:14.104922Z, payload = TestExecutionResult [status = SUCCESSFUL, throwable = null]]
  7.     Event [type = STARTED, testDescriptor = TestMethodTestDescriptor: [engine:junit-jupiter]/[class:example.ExampleTestCase]/[method:abortedTest()], timestamp = 2018-12-14T12:45:14.106121Z, payload = null]
  8.     Event [type = FINISHED, testDescriptor = TestMethodTestDescriptor: [engine:junit-jupiter]/[class:example.ExampleTestCase]/[method:abortedTest()], timestamp = 2018-12-14T12:45:14.109956Z, payload = TestExecutionResult [status = ABORTED, throwable = org.opentest4j.TestAbortedException: Assumption failed: abc does not contain Z]]
  9.     Event [type = STARTED, testDescriptor = TestMethodTestDescriptor: [engine:junit-jupiter]/[class:example.ExampleTestCase]/[method:failingTest()], timestamp = 2018-12-14T12:45:14.110680Z, payload = null]
  10.     Event [type = FINISHED, testDescriptor = TestMethodTestDescriptor: [engine:junit-jupiter]/[class:example.ExampleTestCase]/[method:failingTest()], timestamp = 2018-12-14T12:45:14.111217Z, payload = TestExecutionResult [status = FAILED, throwable = java.lang.ArithmeticException: / by zero]]
  11.     Event [type = FINISHED, testDescriptor = ClassTestDescriptor: [engine:junit-jupiter]/[class:example.ExampleTestCase], timestamp = 2018-12-14T12:45:14.113731Z, payload = TestExecutionResult [status = SUCCESSFUL, throwable = null]]
  12.     Event [type = FINISHED, testDescriptor = JupiterEngineDescriptor: [engine:junit-jupiter], timestamp = 2018-12-14T12:45:14.113806Z, payload = TestExecutionResult [status = SUCCESSFUL, throwable = null]]