9. Code Coverage Analysis

Wikipedia:

In computer science, code coverage is a measure used to describe the degree to which the source code of a program is tested by a particular test suite. A program with high code coverage has been more thoroughly tested and has a lower chance of containing software bugs than a program with low code coverage.

In this chapter you will learn all about PHPUnit’s code coverage functionality that provides an insight into what parts of the production code are executed when the tests are run. It makes use of the php-code-coverage component, which in turn leverages the code coverage functionality provided by the Xdebug or PCOV extensions for PHP or by PHPDBG.

Note

If you see a warning while running tests that no code coverage driver is available, it means that you are using the PHP CLI binary (php) and do not have Xdebug loaded. The Xdebug installation guide explains how Xdebug can be installed and configured. Alternatively, you may use the PHPDBG binary (phpdbg) instead of the PHP CLI one.

PHPUnit can generate an HTML-based code coverage report as well as XML-based logfiles with code coverage information in various formats (Clover, Crap4J, PHPUnit). Code coverage information can also be reported as text (and printed to STDOUT) and exported as PHP code for further processing.

Please refer to The Command-Line Test Runner for a list of command line switches that control code coverage functionality as well as The <logging> Element for the relevant configuration settings.

Software Metrics for Code Coverage

Various software metrics exist to measure code coverage:

Line Coverage

The Line Coverage software metric measures whether each executable line was executed.

Function and Method Coverage

The Function and Method Coverage software metric measures whether each function or method has been invoked. php-code-coverage only considers a function or method as covered when all of its executable lines are covered.

Class and Trait Coverage

The Class and Trait Coverage software metric measures whether each method of a class or trait is covered. php-code-coverage only considers a class or trait as covered when all of its methods are covered.

Opcode Coverage

The Opcode Coverage software metric measures whether each opcode of a function or method has been executed while running the test suite. A line of code usually compiles into more than one opcode. Line Coverage regards a line of code as covered as soon as one of its opcodes is executed.

Branch Coverage

The Branch Coverage software metric measures whether the boolean expression of each control structure evaluated to both true and false while running the test suite.

Path Coverage

The Path Coverage software metric measures whether each of the possible execution paths in a function or method has been followed while running the test suite. An execution path is a unique sequence of branches from the entry of the function or method to its exit.

Change Risk Anti-Patterns (CRAP) Index

The Change Risk Anti-Patterns (CRAP) Index is calculated based on the cyclomatic complexity and code coverage of a unit of code. Code that is not too complex and has an adequate test coverage will have a low CRAP index. The CRAP index can be lowered by writing tests and by refactoring the code to lower its complexity.

Note

The Opcode Coverage, Branch Coverage, and Path Coverage software metrics are not yet supported by php-code-coverage.

Whitelisting Files

It is mandatory to configure a whitelist for telling PHPUnit which sourcecode files to include in the code coverage report. This can either be done using the --whitelist command line option or via the configuration file (see The <filter> Element).

The addUncoveredFilesFromWhitelist and processUncoveredFilesFromWhitelist configuration settings are available to configure how the whitelist is used:

  • addUncoveredFilesFromWhitelist="false" means that only whitelisted files that have at least one line of executed code are included in the code coverage report
  • addUncoveredFilesFromWhitelist="true" (default) means that all whitelisted files are included in the code coverage report even if not a single line of code of such a file is executed
  • processUncoveredFilesFromWhitelist="false" (default) means that a whitelisted file that has no executed lines of code will be added to the code coverage report (if addUncoveredFilesFromWhitelist="true" is set) but it will not be loaded by PHPUnit and it will therefore not be analysed for correct executable lines of code information
  • processUncoveredFilesFromWhitelist="true" means that a whitelisted file that has no executed lines of code will be loaded by PHPUnit so that it can be analysed for correct executable lines of code information

Note

Please note that the loading of sourcecode files that is performed when processUncoveredFilesFromWhitelist="true" is set can cause problems when a sourcecode file contains code outside the scope of a class or function, for instance.

Ignoring Code Blocks

Sometimes you have blocks of code that you cannot test and that you may want to ignore during code coverage analysis. PHPUnit lets you do this using the @codeCoverageIgnore, @codeCoverageIgnoreStart and @codeCoverageIgnoreEnd annotations as shown in Example 9.1.

Example 9.1 Using the @codeCoverageIgnore, @codeCoverageIgnoreStart and @codeCoverageIgnoreEnd annotations

  1. <?php declare(strict_types=1);
  2. use PHPUnit\Framework\TestCase;
  4. /**
  5.  * @codeCoverageIgnore
  6.  */
  7. final class Foo
  8. {
  9.     public function bar(): void
  10.     {
  11.     }
  12. }
  14. final class Bar
  15. {
  16.     /**
  17.      * @codeCoverageIgnore
  18.      */
  19.     public function foo(): void
  20.     {
  21.     }
  22. }
  24. if (false) {
  25.     // @codeCoverageIgnoreStart
  26.     print '*';
  27.     // @codeCoverageIgnoreEnd
  28. }
  30. exit; // @codeCoverageIgnore

The ignored lines of code (marked as ignored using the annotations) are counted as executed (if they are executable) and will not be highlighted.

Specifying Covered Code Parts

The @covers annotation (see the annotation documentation) can be used in the test code to specify which code parts a test class (or test method) wants to test. If provided, this effectively filters the code coverage report to include executed code from the referenced code parts only. Example 9.2 shows an example.

Note

If a method is specificed with the @covers annotation, only the referenced method will be considered as covered, but not methods called by this method. Hence, when a covered method is refactored using the extract method refactoring, corresponding @covers annotations need to be added. This is the reason it is recommended to use this annotation with class scope, not with method scope.

Example 9.2 Test class that specifies which class it wants to cover

  1. <?php declare(strict_types=1);
  2. use PHPUnit\Framework\TestCase;
  4. /**
  5.  * @covers \Invoice
  6.  * @uses \Money
  7.  */
  8. final class InvoiceTest extends TestCase
  9. {
  10.     private $invoice;
  12.     protected function setUp(): void
  13.     {
  14.         $this->invoice = new Invoice;
  15.     }
  17.     public function testAmountInitiallyIsEmpty(): void
  18.     {
  19.         $this->assertEquals(new Money, $this->invoice->getAmount());
  20.     }
  21. }

Example 9.3 Tests that specify which method they want to cover

  1. <?php declare(strict_types=1);
  2. use PHPUnit\Framework\TestCase;
  4. final class BankAccountTest extends TestCase
  5. {
  6.     private $ba;
  8.     protected function setUp(): void
  9.     {
  10.         $this->ba = new BankAccount;
  11.     }
  13.     /**
  14.      * @covers \BankAccount::getBalance
  15.      */
  16.     public function testBalanceIsInitiallyZero(): void
  17.     {
  18.         $this->assertSame(0, $this->ba->getBalance());
  19.     }
  21.     /**
  22.      * @covers \BankAccount::withdrawMoney
  23.      */
  24.     public function testBalanceCannotBecomeNegative(): void
  25.     {
  26.         try {
  27.             $this->ba->withdrawMoney(1);
  28.         }
  30.         catch (BankAccountException $e) {
  31.             $this->assertSame(0, $this->ba->getBalance());
  33.             return;
  34.         }
  36.         $this->fail();
  37.     }
  39.     /**
  40.      * @covers \BankAccount::depositMoney
  41.      */
  42.     public function testBalanceCannotBecomeNegative2(): void
  43.     {
  44.         try {
  45.             $this->ba->depositMoney(-1);
  46.         }
  48.         catch (BankAccountException $e) {
  49.             $this->assertSame(0, $this->ba->getBalance());
  51.             return;
  52.         }
  54.         $this->fail();
  55.     }
  57.     /**
  58.      * @covers \BankAccount::getBalance
  59.      * @covers \BankAccount::depositMoney
  60.      * @covers \BankAccount::withdrawMoney
  61.      */
  62.     public function testDepositWithdrawMoney(): void
  63.     {
  64.         $this->assertSame(0, $this->ba->getBalance());
  65.         $this->ba->depositMoney(1);
  66.         $this->assertSame(1, $this->ba->getBalance());
  67.         $this->ba->withdrawMoney(1);
  68.         $this->assertSame(0, $this->ba->getBalance());
  69.     }
  70. }

It is also possible to specify that a test should not cover any method by using the @coversNothing annotation (see @coversNothing). This can be helpful when writing integration tests to make sure you only generate code coverage with unit tests.

Example 9.4 A test that specifies that no method should be covered

  1. <?php declare(strict_types=1);
  2. use PHPUnit\DbUnit\TestCase
  4. final class GuestbookIntegrationTest extends TestCase
  5. {
  6.     /**
  7.      * @coversNothing
  8.      */
  9.     public function testAddEntry(): void
  10.     {
  11.         $guestbook = new Guestbook();
  12.         $guestbook->addEntry("suzy", "Hello world!");
  14.         $queryTable = $this->getConnection()->createQueryTable(
  15.             'guestbook', 'SELECT * FROM guestbook'
  16.         );
  18.         $expectedTable = $this->createFlatXmlDataSet("expectedBook.xml")
  19.                               ->getTable("guestbook");
  21.         $this->assertTablesEqual($expectedTable, $queryTable);
  22.     }
  23. }

Edge Cases

This section shows noteworthy edge cases that lead to confusing code coverage information.

  1. <?php declare(strict_types=1);
  2. use PHPUnit\Framework\TestCase;
  4. // Because it is "line based" and not statement base coverage
  5. // one line will always have one coverage status
  6. if (false) this_function_call_shows_up_as_covered();
  8. // Due to how code coverage works internally these two lines are special.
  9. // This line will show up as non executable
  10. if (false)
  11.     // This line will show up as covered because it is actually the
  12.     // coverage of the if statement in the line above that gets shown here!
  13.     will_also_show_up_as_covered();
  15. // To avoid this it is necessary that braces are used
  16. if (false) {
  17.     this_call_will_never_show_up_as_covered();
  18. }

Speeding Up Code Coverage with Xdebug

The performance of code coverage data collection with Xdebug 2.6 (and later) can be significantly improved by delegating whitelist filtering to Xdebug.

In order to do this, the first step is to generate the filter script for Xdebug using the --dump-xdebug-filter option:

  1. $ phpunit --dump-xdebug-filter build/xdebug-filter.php
  2. PHPUnit 7.4.0 by Sebastian Bergmann and contributors.
  4. Runtime:       PHP 7.2.11 with Xdebug 2.6.1
  5. Configuration: /workspace/project/phpunit.xml
  7. Wrote Xdebug filter script to build/xdebug-filter.php

Now we can use the --prepend option to load the Xdebug filter script as early as possible when we want to generate a code coverage report:

  1. $ phpunit --prepend build/xdebug-filter.php --coverage-html build/coverage-report