Introduction to working on test coverage

When tests check whether a particular method is working as expected, they do that by executing a certain code path through the method. Most methods have various code paths (branching on if/else and switch/case statements, branching on the multi-element conditions in the if/else conditions, etc.) and a through test suite should check all possible (or at least all sane) combinations of the code branches. The simplest way to monitor this is by using code coverage report when working on tests. CiviCRM code coverage reports are available on our test server: http://tests.dev.civicrm.org/trunk/coverage/ (for current development version) and http://tests.dev.civicrm.org/stable/coverage/ (for current stable version). You can also generate them on your local machine (see: Setting up your personal testing sandbox HOWTO).

Let's describe working on code coverage on a specific, real life example. We will be working on increasing the test coverage for the api/v2/CustomGroup.php file. Tests for it are located in tests/phpunit/api/v2/CustomGroupTest.php.

1. Go to http://tests.dev.civicrm.org/trunk/coverage/, click on "api" link, than "v2", and than find "CustomGroup.php" in the list and click it. You will see code coverage report for api/v2/CustomGroup.php file. Scroll down, you will see something like on the picture below:

Lines highlighted in green are those which has been executed by tests at least once, those highlighted in orange are the ones that haven't been executed - these are the interesting ones. At the beginning of each line you will see the line number (the very beginning, outside of the highlight); the second number, right before the colon, tells you how many times this line has been executed by all the tests. The colon is followed by the actual code of that line.

Example 1

So we want to make the orange lines green. Let's look at lines 69,70,71. 70 is orange.

Line 70 is not tested. It's enclosed in a condition - and the fact that it's red mean that the condition is always false (i.e., in our tests $params is always an array). We are not checking for the scenario in which civicrm_custom_group_create is called with wrong params type. So let's add following test method in api_v2_CustomGroupTest class:

There you go. Line number 70 done, we can move to the next orange spot.

Example 2

As we go through orange lines and write test methods for them, we hit line 196. A bunch of lines in civicrm_custom_field_create are not tested here, and again, they are enclosed in condition:

This means that when this code is executed by our tests this condition is always false. Here's how it looks like in broader context (don't mind the earlier orange lines in this method, I'm assuming we fixed them already ):

So we need another test method, which will render the condition true and execute enclosed lines. So something like this:

Since it's a compound condition (made of two params joined with an AND), we also might want to test a few scenarios. I'm taking a look at the tests that has been written so far. It seems that this condition was returning false based on the first part being false -

$params['option_values']

was never set (so

isset( $params['option_values'] )

was false, and therefore the whole condition returned false). I might want to check what happens if the first part of the condition returns true, but the second one doesn't. In this specific case, it will require fixing of the API - and we don't want to do this at the moment, since we don't have API backward compatibility rules agreed upon yet. However - the lesson from this example is that every time you see a compound condition, remember that it's not possible to check the code path with a single test (although there might be cases when checking an obscure code path is not feasible, especially at this point in time and our current goals to get as broad 'general' test coverage as fast as possible).