Quarkus - Measuring the coverage of your tests

Learn how to measure the test coverage of your application. This guide covers:

  • Measuring the coverage of your Unit Tests

  • Measuring the coverage of your Integration Tests

  • Separating the execution of your Unit Tests and Integration Tests

  • Consolidating the coverage for all your tests

Please note that code coverage is not supported in native mode.

1. Prerequisites

To complete this guide, you need:

  • less than 15 minutes

  • an IDE

  • JDK 1.8+ installed with JAVA_HOME configured appropriately

  • Apache Maven 3.6.2+

  • Having completed the Testing your application guide

2. Architecture

The application built in this guide is just a JAX-RS endpoint (hello world) that relies on dependency injection to use a service. The service will be tested with JUnit 5 and the endpoint will be annotated via a @QuarkusTest annotation.

3. Solution

We recommend that you follow the instructions in the next sections and create the application step by step. However, you can go right to the completed example. Clone the Git repository: git clone [https://github.com/quarkusio/quarkus-quickstarts.git](https://github.com/quarkusio/quarkus-quickstarts.git), or download an archive.

The solution is located in the tests-with-coverage-quickstart directory.

4. Starting from a simple project and two tests

Let’s start from an empty application created with the Quarkus Maven plugin:

  1. mvn io.quarkus:quarkus-maven-plugin:create \
  2.     -DprojectGroupId=org.acme \
  3.     -DprojectArtifactId=tests-with-coverage-quickstart
  4. cd tests-with-coverage-quickstart

Now we’ll be adding all the elements necessary to have an application that is properly covered with tests.

First, an application serving a hello endpoint:

  1. package org.acme.testcoverage;
  3. import javax.inject.Inject;
  4. import javax.ws.rs.GET;
  5. import javax.ws.rs.Path;
  6. import javax.ws.rs.PathParam;
  7. import javax.ws.rs.Produces;
  8. import javax.ws.rs.core.MediaType;
  10. @Path("/hello")
  11. public class GreetingResource {
  13.     private final GreetingService service;
  15.     @Inject
  16.     public GreetingResource(GreetingService service) {
  17.         this.service = service;
  18.     }
  20.     @GET
  21.     @Produces(MediaType.TEXT_PLAIN)
  22.     @Path("/greeting/{name}")
  23.     public String greeting(@PathParam("name") String name) {
  24.         return service.greeting(name);
  25.     }
  27.     @GET
  28.     @Produces(MediaType.TEXT_PLAIN)
  29.     public String hello() {
  30.         return "hello";
  31.     }
  32. }

This endpoint uses a greeting service:

  1. package org.acme.testcoverage;
  3. import javax.enterprise.context.ApplicationScoped;
  5. @ApplicationScoped
  6. public class GreetingService {
  8.     public String greeting(String name) {
  9.         return "hello " + name;
  10.     }
  12. }

The project will also need some tests. First a simple JUnit:

  1. package org.acme.testcoverage;
  3. import org.junit.jupiter.api.Assertions;
  4. import org.junit.jupiter.api.Test;
  6. public class GreetingServiceTest {
  8.     @Test
  9.     public void testGreetingService() {
  10.         GreetingService service = new GreetingService();
  11.         Assertions.assertEquals("hello Quarkus", service.greeting("Quarkus"));
  12.     }
  13. }

But also a @QuarkusTest:

  1. package org.acme.testcoverage;
  3. import io.quarkus.test.junit.QuarkusTest;
  4. import org.junit.jupiter.api.Test;
  5. import org.junit.jupiter.api.Tag;
  7. import java.util.UUID;
  9. import static io.restassured.RestAssured.given;
  10. import static org.hamcrest.CoreMatchers.is;
  12. @QuarkusTest
  13. @Tag("integration")
  14. public class GreetingResourceTest {
  16.     @Test
  17.     public void testHelloEndpoint() {
  18.         given()
  19.           .when().get("/hello")
  20.           .then()
  21.              .statusCode(200)
  22.              .body(is("hello"));
  23.     }
  25.     @Test
  26.     public void testGreetingEndpoint() {
  27.         String uuid = UUID.randomUUID().toString();
  28.         given()
  29.           .pathParam("name", uuid)
  30.           .when().get("/hello/greeting/{name}")
  31.           .then()
  32.             .statusCode(200)
  33.             .body(is("hello " + uuid));
  34.     }
  35. }

The first one will be our example of a Unit Test and the second one will be our example of Integration Test.

5. Separating executions of Unit Tests and Integration Tests

You may want to consider that JUnits and QuarkusTests are two different kind of tests and that they deserve to be separated. This way you could run them separately, in different cases or some more often than the others. In order to do so, we’ll use a feature of JUnit 5 that allows us to tag some tests. Let’s tag GreetingResourceTest.java and specify that it is an Integration Test:

  1. import org.junit.jupiter.api.Tag;
  2. ...
  4. @QuarkusTest
  5. @Tag("integration")
  6. public class GreetingResourceTest {
  7.     ...
  8. }

We’re now able to distinguish unit tests and integration tests. Now, let’s bind them to different Maven lifecycle phases. Let’s use surefire to bind unit tests to the test phase and the integration tests to the integration-test phase.

  1. <project>
  2.     ...
  3.     <build>
  4.         <plugins>
  5.             <plugin>
  6.                 <artifactId>maven-surefire-plugin</artifactId>
  7.                 <version>${surefire-plugin.version}</version>
  8.                 <configuration>
  9.                     <excludedGroups>integration</excludedGroups>
  10.                     <systemPropertyVariables>
  11.                         <java.util.logging.manager>org.jboss.logmanager.LogManager</java.util.logging.manager>
  12.                         <maven.home>${maven.home}</maven.home>
  13.                     </systemPropertyVariables>
  14.                 </configuration>
  15.                 <executions>
  16.                     <execution>
  17.                         <id>integration-tests</id>
  18.                         <phase>integration-test</phase>
  19.                         <goals>
  20.                             <goal>test</goal>
  21.                         </goals>
  22.                         <configuration>
  23.                             <excludedGroups>!integration</excludedGroups>
  24.                             <groups>integration</groups>
  25.                         </configuration>
  26.                     </execution>
  27.                 </executions>
  28.             </plugin>
  29.         </plugins>
  30.     </build>
  31.     ...
  32. </project>
This way, the QuarkusTest instances will be executed as part of the integration-test build phase while the other JUnit tests will still be ran during the test phase. You can run all the tests with the command ./mvnw clean verify (and you will notice that two tests are ran in different phases).

6. Measuring the coverage of JUnit tests using JaCoCo

It is now time to introduce JaCoCo to measure the coverage. The straightforward way to add JaCoCo to your build is to reference the plugin in your pom.xml.

  1.     <properties>
  2.     ...
  3.         <jacoco.version>0.8.4</jacoco.version>
  4.     </properties>
  6.     <build>
  7.         <plugins>
  8.         ...
  9.             <plugin>
  10.                 <groupId>org.jacoco</groupId>
  11.                 <artifactId>jacoco-maven-plugin</artifactId>
  12.                 <version>${jacoco.version}</version>
  13.                 <executions>
  14.                     <execution>
  15.                         <id>default-prepare-agent</id>
  16.                         <goals>
  17.                             <goal>prepare-agent</goal>
  18.                         </goals>
  19.                     </execution>
  20.                     <execution>
  21.                         <id>default-report</id>
  22.                         <goals>
  23.                             <goal>report</goal>
  24.                         </goals>
  25.                         <configuration>
  26.                             <dataFile>${project.build.directory}/jacoco.exec</dataFile>
  27.                             <outputDirectory>${project.reporting.outputDirectory}/jacoco</outputDirectory>
  28.                         </configuration>
  29.                     </execution>
  30.                 </executions>
  31.             </plugin>
  32.         </plugins>
  33.     </build>
If you run ./mvnw clean test the coverage information will be collected during the execution of the unit tests in the file jacoco.exec.

7. Measuring separately the coverage of each test type

It is not strictly necessary, but let’s distinguish the coverage brought by each test type. To do so, we`ll just output the coverage info in two different files, one in jacoco-ut.exec and one in jacoco-it.exec. We also need to generate a separate report for each test execution. Let’s adjust the Jacoco configuration for that:

  1.     <properties>
  2.     ...
  3.         <jacoco.version>0.8.4</jacoco.version>
  4.     </properties>
  6.     <build>
  7.         <plugins>
  8.         ...
  9.             <plugin>
  10.                 <groupId>org.jacoco</groupId>
  11.                 <artifactId>jacoco-maven-plugin</artifactId>
  12.                 <version>${jacoco.version}</version>
  13.                 <executions>
  14.                     <execution>
  15.                         <id>prepare-agent-ut</id>
  16.                         <goals>
  17.                             <goal>prepare-agent</goal>
  18.                         </goals>
  19.                         <configuration>
  20.                             <destFile>${project.build.directory}/jacoco-ut.exec</destFile>
  21.                         </configuration>
  22.                     </execution>
  23.                     <execution>
  24.                         <id>prepare-agent-it</id>
  25.                         <phase>pre-integration-test</phase>
  26.                         <goals>
  27.                             <goal>prepare-agent</goal>
  28.                         </goals>
  29.                         <configuration>
  30.                             <destFile>${project.build.directory}/jacoco-it.exec</destFile>
  31.                         </configuration>
  32.                     </execution>
  33.                     <execution>
  34.                         <id>report-ut</id>
  35.                         <goals>
  36.                             <goal>report</goal>
  37.                         </goals>
  38.                         <configuration>
  39.                             <dataFile>${project.build.directory}/jacoco-ut.exec</dataFile>
  40.                             <outputDirectory>${project.reporting.outputDirectory}/jacoco-ut</outputDirectory>
  41.                         </configuration>
  42.                     </execution>
  43.                     <execution>
  44.                         <id>report-it</id>
  45.                         <phase>post-integration-test</phase>
  46.                         <goals>
  47.                             <goal>report</goal>
  48.                         </goals>
  49.                         <configuration>
  50.                             <dataFile>${project.build.directory}/jacoco-it.exec</dataFile>
  51.                             <outputDirectory>${project.reporting.outputDirectory}/jacoco-it</outputDirectory>
  52.                         </configuration>
  53.                     </execution>
  54.                 </executions>
  55.             </plugin>
  56.         </plugins>
  57.     </build>

8. The coverage does not seem to correspond to the reality

You can now run the tests: ./mvnw clean verify As explained earlier, it will run the unit tests first, then the integration tests. And finally, it will generate two separate reports. First a report of the coverage of the unit tests in target/site/jacoco-ut then a report of the coverage of the integration tests in target/site/jacoco-it.

Given the content of GreetingResourceTest, GreetingResource should have been covered. But when we open the report target/site/jacoco-it/index.html, the class GreetingResource is reported with 0% of coverage. But the fact that GreetingService is reported as covered shows that the test execution was actually recorded. How come?

During the report generation, you may have noticed a warning:

  1. [WARNING] Classes in bundle '***' do no match with execution data. For report generation the same class files must be used as at runtime.
  2. [WARNING] Execution data for class org/acme/testcoverage/GreetingResource does not match.

It seems that Quarkus and JaCoCo step on each other’s toes. What happens is that Quarkus transforms the JAX-RS resources (and also the Panache files). You may have noticed that GreetingResource was not written in the simplest way like:

  1. ...
  2. @Path("/hello")
  3. public class GreetingResource {
  5.     @Inject
  6.     GreetingService service;
  8.     @GET
  9.     @Produces(MediaType.TEXT_PLAIN)
  10.     @Path("/greeting/{name}")
  11.     public String greeting(@PathParam("name") String name) {
  12.         return service.greeting(name);
  13.     }
  15.     @GET
  16.     @Produces(MediaType.TEXT_PLAIN)
  17.     public String hello() {
  18.         return "hello";
  19.     }
  20. }

Above, the constructor is implicit and we use injection to have an instance of GreetingService. Note that, with this code relying on an implicit constructor, the coverage would have been reported properly by JaCoCo. Instead, we introduced a constructor based injection:

  1. ...
  2. @Path("/hello")
  3. public class GreetingResource {
  5.     private final GreetingService service;
  7.     @Inject
  8.     public GreetingResource(GreetingService service) {
  9.         this.service = service;
  10.     }
  11. ...
  12. }

Some might say that this approach is preferable since the field can be final like this. Anyway, in some cases you might need an explicit constructor. And, in that case, the coverage is not reported properly by JaCoCo. This is because Quarkus generates a constructor without any parameter and does some bycode manipulations in order to add it to the class. That is what happened here, just before the execution of the integration tests:

  1. [INFO] --- quarkus-maven-plugin:0.16.0:build (default) @ getting-started-testing ---
  2. [INFO] [io.quarkus.deployment.QuarkusAugmentor] Beginning quarkus augmentation
  3. ...

As a consequence, JaCoCo does not recognize the classes when it wants to create its report. But wait…​ there is a solution.

9. Instrumenting the classes instead

JaCoCo has two modes. The first one is based on an agent and instruments classes on-the-fly. Unfortunately, this is incompatible with the dynamic classfile transformations that Quarkus does. The second mode is called offline instrumentation. Classes are pre-instrumented in advance via the jacoco:instrument Maven goal and during their usage (when the tests are ran), jacocoagent.jar must be added to the classpath. Once the tests have been executed, it is recommended to restore the original classes using the jacoco:restore-instrumented-classes Maven goal.

Let’s first add the dependency on jacocoagent.jar:

  1. <project>
  2.     ...
  3.     <dependencies>
  4.         ...
  5.         <dependency>
  6.             <groupId>org.jacoco</groupId>
  7.             <artifactId>org.jacoco.agent</artifactId>
  8.             <classifier>runtime</classifier>
  9.             <scope>test</scope>
  10.             <version>${jacoco.version}</version>
  11.         </dependency>
  12.     </dependencies>
  13. </project>

Then let’s configure three jacoco plugin goals for unit tests:

  • One to instrument the classes during the process-classes phase

  • One to restore the original classes during the prepare-package phase (after the tests are ran)

  • One to generate the report during the verify phase (the report generation requires the original classes to have been restored)

and a similar setup for the integration tests too:

  1. <project>
  2.     ...
  3.     <build>
  4.         <plugins>
  5.             ...
  6.             <plugin>
  7.                 <groupId>org.jacoco</groupId>
  8.                 <artifactId>jacoco-maven-plugin</artifactId>
  9.                 <version>${jacoco.version}</version>
  10.                 <executions>
  11.                     <execution>
  12.                         <id>instrument-ut</id>
  13.                         <goals>
  14.                             <goal>instrument</goal>
  15.                         </goals>
  16.                     </execution>
  17.                     <execution>
  18.                         <id>restore-ut</id>
  19.                         <goals>
  20.                             <goal>restore-instrumented-classes</goal>
  21.                         </goals>
  22.                     </execution>
  23.                     <execution>
  24.                         <id>report-ut</id>
  25.                         <goals>
  26.                             <goal>report</goal>
  27.                         </goals>
  28.                         <configuration>
  29.                             <dataFile>${project.build.directory}/jacoco-ut.exec</dataFile>
  30.                             <outputDirectory>${project.reporting.outputDirectory}/jacoco-ut</outputDirectory>
  31.                         </configuration>
  32.                     </execution>
  33.                     <execution>
  34.                         <id>instrument-it</id>
  35.                         <phase>pre-integration-test</phase>
  36.                         <goals>
  37.                             <goal>instrument</goal>
  38.                         </goals>
  39.                     </execution>
  40.                     <execution>
  41.                         <id>restore-it</id>
  42.                         <phase>post-integration-test</phase>
  43.                         <goals>
  44.                             <goal>restore-instrumented-classes</goal>
  45.                         </goals>
  46.                     </execution>
  47.                     <execution>
  48.                         <id>report-it</id>
  49.                         <phase>post-integration-test</phase>
  50.                         <goals>
  51.                             <goal>report</goal>
  52.                         </goals>
  53.                         <configuration>
  54.                             <dataFile>${project.build.directory}/jacoco-it.exec</dataFile>
  55.                             <outputDirectory>${project.reporting.outputDirectory}/jacoco-it</outputDirectory>
  56.                         </configuration>
  57.                     </execution>
  58.                 </executions>
  59.             </plugin>
  60.         </plugins>
  61.     </build>
  62. </project>

It also requires a small change in the Surefire configuration. Note below that we specified jacoco-agent.destfile as a system property in the default case (unit tests) and for the integration tests.

  1. <project>
  2.     ...
  3.     <build>
  4.         <plugins>
  5.             ...
  6.             <plugin>
  7.                 <artifactId>maven-surefire-plugin</artifactId>
  8.                 <version>${surefire-plugin.version}</version>
  9.                 <configuration>
  10.                     <excludedGroups>integration</excludedGroups>
  11.                     <systemPropertyVariables>
  12.                         <jacoco-agent.destfile>${project.build.directory}/jacoco-ut.exec</jacoco-agent.destfile>
  13.                         <java.util.logging.manager>org.jboss.logmanager.LogManager</java.util.logging.manager>
  14.                         <maven.home>${maven.home}</maven.home>
  15.                     </systemPropertyVariables>
  16.                 </configuration>
  17.                 <executions>
  18.                     <execution>
  19.                         <id>integration-tests</id>
  20.                         <phase>integration-test</phase>
  21.                         <goals>
  22.                             <goal>test</goal>
  23.                         </goals>
  24.                         <configuration>
  25.                             <excludedGroups>!integration</excludedGroups>
  26.                             <groups>integration</groups>
  27.                             <systemPropertyVariables>
  28.                                 <jacoco-agent.destfile>${project.build.directory}/jacoco-it.exec</jacoco-agent.destfile>
  29.                             </systemPropertyVariables>
  30.                         </configuration>
  31.                     </execution>
  32.                 </executions>
  33.             </plugin>
  34.         </plugins>
  35.     </build>
  36. </project>

Let’s now check the generated report that can be found in target/site/jacoco-it/index.html. The report now shows that GreetingResource is actually properly covered! Yay!

10. Bonus: Building a consolidated report for Unit Tests and Integration Tests

So, finally, let’s improve the setup even further and let’s merge the two execution files (jacoco-ut.exec and jacoco-it.exec) into one consolidated report and generate a consolidated report that will show the coverage of all your tests combined.

You should end up with something like this (note the addition of the merge-results and post-merge-report executions):

  1. <project>
  2.     ...
  3.     <build>
  4.         <plugins>
  5.             <plugin>
  6.                 <artifactId>maven-surefire-plugin</artifactId>
  7.                 <version>${surefire-plugin.version}</version>
  8.                 <configuration>
  9.                     <excludedGroups>integration</excludedGroups>
  10.                     <systemPropertyVariables>
  11.                         <jacoco-agent.destfile>${project.build.directory}/jacoco-ut.exec</jacoco-agent.destfile>
  12.                         <java.util.logging.manager>org.jboss.logmanager.LogManager</java.util.logging.manager>
  13.                         <maven.home>${maven.home}</maven.home>
  14.                     </systemPropertyVariables>
  15.                 </configuration>
  16.                 <executions>
  17.                     <execution>
  18.                         <id>integration-tests</id>
  19.                         <phase>integration-test</phase>
  20.                         <goals>
  21.                             <goal>test</goal>
  22.                         </goals>
  23.                         <configuration>
  24.                             <excludedGroups>!integration</excludedGroups>
  25.                             <groups>integration</groups>
  26.                             <systemPropertyVariables>
  27.                                 <jacoco-agent.destfile>${project.build.directory}/jacoco-it.exec</jacoco-agent.destfile>
  28.                             </systemPropertyVariables>
  29.                         </configuration>
  30.                     </execution>
  31.                 </executions>
  32.             </plugin>
  33.             ...
  34.             <plugin>
  35.                 <groupId>org.jacoco</groupId>
  36.                 <artifactId>jacoco-maven-plugin</artifactId>
  37.                 <version>${jacoco.version}</version>
  38.                 <executions>
  39.                     <execution>
  40.                         <id>instrument-ut</id>
  41.                         <goals>
  42.                             <goal>instrument</goal>
  43.                         </goals>
  44.                     </execution>
  45.                     <execution>
  46.                         <id>restore-ut</id>
  47.                         <goals>
  48.                             <goal>restore-instrumented-classes</goal>
  49.                         </goals>
  50.                     </execution>
  51.                     <execution>
  52.                         <id>report-ut</id>
  53.                         <goals>
  54.                             <goal>report</goal>
  55.                         </goals>
  56.                         <configuration>
  57.                             <dataFile>${project.build.directory}/jacoco-ut.exec</dataFile>
  58.                             <outputDirectory>${project.reporting.outputDirectory}/jacoco-ut</outputDirectory>
  59.                         </configuration>
  60.                     </execution>
  61.                     <execution>
  62.                         <id>instrument-it</id>
  63.                         <phase>pre-integration-test</phase>
  64.                         <goals>
  65.                             <goal>instrument</goal>
  66.                         </goals>
  67.                     </execution>
  68.                     <execution>
  69.                         <id>restore-it</id>
  70.                         <phase>post-integration-test</phase>
  71.                         <goals>
  72.                             <goal>restore-instrumented-classes</goal>
  73.                         </goals>
  74.                     </execution>
  75.                     <execution>
  76.                         <id>report-it</id>
  77.                         <phase>post-integration-test</phase>
  78.                         <goals>
  79.                             <goal>report</goal>
  80.                         </goals>
  81.                         <configuration>
  82.                             <dataFile>${project.build.directory}/jacoco-it.exec</dataFile>
  83.                             <outputDirectory>${project.reporting.outputDirectory}/jacoco-it</outputDirectory>
  84.                         </configuration>
  85.                     </execution>
  86.                     <execution>
  87.                         <id>merge-results</id>
  88.                         <phase>verify</phase>
  89.                         <goals>
  90.                             <goal>merge</goal>
  91.                         </goals>
  92.                         <configuration>
  93.                             <fileSets>
  94.                                 <fileSet>
  95.                                     <directory>${project.build.directory}</directory>
  96.                                     <includes>
  97.                                         <include>*.exec</include>
  98.                                     </includes>
  99.                                 </fileSet>
  100.                             </fileSets>
  101.                             <destFile>${project.build.directory}/jacoco.exec</destFile>
  102.                         </configuration>
  103.                     </execution>
  104.                     <execution>
  105.                         <id>post-merge-report</id>
  106.                         <phase>verify</phase>
  107.                         <goals>
  108.                             <goal>report</goal>
  109.                         </goals>
  110.                         <configuration>
  111.                             <dataFile>${project.build.directory}/jacoco.exec</dataFile>
  112.                             <outputDirectory>${project.reporting.outputDirectory}/jacoco</outputDirectory>
  113.                         </configuration>
  114.                     </execution>
  115.                 </executions>
  116.             </plugin>
  117.         </plugins>
  118.     </build>
  119.     ...
  120.     <dependencies>
  121.         ...
  122.         <dependency>
  123.             <groupId>org.jacoco</groupId>
  124.             <artifactId>org.jacoco.agent</artifactId>
  125.             <classifier>runtime</classifier>
  126.             <scope>test</scope>
  127.             <version>${jacoco.version}</version>
  128.         </dependency>
  129.     </dependencies>
  130. </project>

11. Conclusion

You now have all the information you need to study the coverage of your tests! But remember, some code that is not covered is certinaly not well tested. But some code that is covered is not necessarily well tested. Make sure to write good tests!