Introduction

Automated testing is a fundamental part of developing high-quality software. In CiviCRM, automated tests are divided into two categories: unit-tests and web-tests. Unit-tests run on the command-line and ensure the quality of CiviCRM APIs, functions, and classes. Web-tests run in the context of a simulated web browser, and are equivalent to clicking on buttons and filling out fields – web-tests ensure that the pieces of CiviCRM fit together to form a cohesive system. Both unit-tests and web-tests build on top of PHPUnit.

Web tests are developed using a tool called "Selenium", so the terms "web test" and "Selenium test" refer to the same thing: tests which either run within the context of the browser (Selenium IDE) or simulate the browser's actions at the command line (Selenium Server).

In software engineering, web tests may sometimes be called "integration tests".

 

Setup Unit Tests

Create a test database

  1. Create a database named "civicrm_tests_dev" in MySQL. This database will be used for a very minimal CiviCRM install on which to run tests. It should be separate from your main CiviCRM database because it will be frequently dropped, truncated and/or corrupted. You will need the username and password for a user with SUPER privileges. (Note that SUPER is a MySQL global privilege.)
  2. Populate the database schema and metadata by loading these two files: "civicrm/sql/civicrm.mysql" and "civicrm/sql/civicrm_generated.mysql"

 

cd /path-to/civicrm
echo "create database civicrm_tests_dev" | mysql -u user -p -hhostname
mysql -u user -p -hhostname civicrm_tests_dev < sql/civicrm.mysql
mysql -u user -p -hhostname civicrm_tests_dev < sql/civicrm_generated.mysql

It is helpful but not strictly necessary to name the database "civicrm_tests_dev" – subsequent steps will be easier if you use this default name. However, if you are an advanced developer who works concurrently on multiple versions of CiviCRM, then you may want to create separate databases with different names..

Create a database connection

When executing unit-tests on the command-line, the tests will need to connect to the MySQL database – by providing a username, password, hostname, and password. There are several options for configuring the database connection. (Note: In all cases, the user must have "SUPER" privileges.) Choose any one of the following:

Run unit-tests

To run a single test – for example, "api_v3_ContactTest" – execute following command:

cd /path-to/civicrm/tools
scripts/phpunit api_v3_ContactTest

If you have chosen to pass the MySQL connection as parameters (rather than putting the connection info in a file), then pass user/password/host/database like so:

cd /path-to/civicrm/tools
scripts/phpunit -udb_username -pdb_password -hdb_host -bdb_name api_v3_ContactTest

## Notes:
##  * There is no space between -u, -p, -h, -b and the corresponding parameters.
##  * The db_username must have super privileges.
##  * On Windows you will get errors about "stty", but it works anyway.

You may get unexpected errors or warnings if using the wrong version of PHP. For example, OS X users who run MAMP may receive warnings about timezone settings. This happens because the system includes multiple copies of PHP – one distributed with OS X (which is poorly configured/built), and a better version distributed with MAMP (which is better configured/built). To address this, see Setup Command-Line PHP.

To run the whole suite of tests, change the word "api_v3_ContactTest" to "AllTests". The full suite of tests can take a number of hours to complete, so it may be best to run this overnight.

cd /path-to/civicrm/tools
scripts/phpunit AllTests

To run a single test (one method in the test class), add the --filter parameter.

  1. cd /path-to/civicrm/tools
    scripts/phpunit --filter test_method_name$ api_v3_ContactTest
    
  2. --filter is evaluated as a regular expression and compared to test method names.
    EXAMPLE: To run all methods which contain the string "testCreate":

    cd /path-to/civicrm/tools
    scripts/phpunit --filter testCreate api_v3_ContactTest
    

    EXAMPLE: To run ONLY the "testCreateEmptyContact" method:

    cd /path-to/civicrm/tools
    scripts/phpunit --filter "/\btestCreateEmptyContact\b/"
    
  3. Test results will be printed on the screen ("F" for failure, "E" for exceptions, "." for success), in addition to that, files with simple reports will be saved to <REPO>/tools/tests.

Sometimes, when you're running your tests, phpunit might quit silently and it's hard to identify which specific test does it. Please use --tap parameter in such cases, it will give you more descriptive output, e.g.

scripts/phpunit --tap api_v2_ContactTest
scripts/phpunit --tap WebTest_Contact_AddTest

Running Unit Tests for Extensions

To run unit tests for an extension, see the relevant documentation.

Setup Web Tests

The process of using web-tests is generally the same as using unit-tests – one must create a test database, create a database connection, and run "phpunit" commands. However, there is a key difference: while the unit-tests perform all data operations on an isolated database (civicrm_tests_dev), the web-tests require a fully-functioning CiviCRM-Drupal installation, and they require extra information for simulating web-requests. To configure these, setup CiviSeleniumSettings.php:

cd /path-to/civicrm/tests/phpunit/CiviTest
cp CiviSeleniumSettings.php.txt CiviSeleniumSettings.php

Edit the civicrm.settings.local.php that was created in the PHPUnit setup and add or adjust CIVICRM_TEMPLATE_COMPILEDIR to a full path that is writable for the user who will be running tests (note that using "~" in the path may not work - use full paths).

If you want to run Selenium tests, go to Setting yourself up to work with Selenium tests page for details.

(REWRITE) Optional Steps

Optional:

  1. Without Xdebug extension, code coverage report generation will fail, but it won't infuence test execution.
  2. If you have Phing installed, you can go to <REPO>/tools/scripts/ and execute "phing" command. It will build nice results report and put it in <REPO>/tools/tests/results directory.
  3. You can avoid re-initialising the database at every run by adding $populateOnce = true in tests/phpunit/CiviTest/CiviUnitTestCase.php. Warning: it might/will create side effects where the db isn't in a known clean side. Do try to clean up everything in the setUp/tearDown methods.