E-mail
Thanks due to my secondary school chemistry teacher who taught me how to write up a science experiment...
Background and motivation
It seems like this might be a great way...
1) for implementers to test client upgrades - if they can run tests on 'test upgrades' (i.e. copies of sites that they are upgrading to make sure everything is working OK before doing the live upgrade) they'll be more confident that things are working as expected. If, post upgrade, they find a bug, they'll be set up with some infrastructure that will enable them to submit a (db independent) bug report.
2) automate testing of alpha and beta releases - some bugs are only exposed when CiviCRM gets used in the real world. They are not caught by the test suite running on demo data. Having a simple method that implementers can use to run the testing framework on alphas, betas and release candidates means that to it much easier for people to participate in the alpha and beta testing phase for each release.
3) increase quality of testing framework we've historically had some trouble getting people to engage with the testing framework, write and submit tests, etc. Getting more people running tests should mean more tests and more people finding and fixing errors in existing tests, so should hopefully improve the quality of web tests.
Aim
Work out if and how we can use the WebTests framework to test for bugs in real world installations.
Method
IMPORTANT NOTE: We're using a testing environment to run webtests on a remote environment, i.e. we can't use DB assert. We'd like to change DB assert to API Assert where possible so that webtests can be run remotley.
Get some real world data on a latest version (the later the better as we make improvements to how easy it is to set up testing).
Ensure that the test suite you are running is the same / as close to the site that you are testing.
Disable outbound email and any other external on the real world data that you are testing
Run a few example test classes and see how easy they are to run and see what changes we would need to make to make them DB independent.
Results
We ran this on http://svn.civicrm.org/civicrm revision 36931.
We first tested the tests on the standard demo data to ensure they were working. If they weren't working, we fixed them. We then tested on the custom data.
WebTest_Contact_AddTest
There are four tests here. We encountered a number of assumptions / problems.
testIndividualAdd
- presumes that the prefix with id 3 is present (i.e. Mr) and that it can be selected
- presumes that the USA is enabled as a country and that we have turned on State Province and Geocoding fields
- presumes that the following are enabled in admin/setting/preferences/display?reset=1
- instant messenger
- open id
- communication preferences
- demographics
testHouseholdAdd
- (maybe) presumes the above and
- phone option value 4 was not present
testOrganizationAdd
- (maybe) presumes the above and
- presumes that the prefix with id 3 is present (i.e. Mr) and that it can be selected
testIndividualAdWithSharedAddress
- this is an interesting one - because the random data was being added at the end of the contact and in this installation we changed the dupe matching rule to just match on the first few fields, this caused a duplicate matching warning to be shown.
WebTest_Contact_TagAddTest
Ran first time with no errors
AllTests real vs. demo
I thought it would be worth running the entire test suite on demo and real data to see how they compare.
A few caveats. This is the trunk (around revision 36931) running on trunk demo data and some 4.07 real data
Real
Demo
Conclusions / ways forward
Obviously others may encounter more assumptions as we progress with these tests.
Create helper functions to make the tests DB independent. e.g. pick an existing name_prefix rather than a default name_prefix (which might have been disabled in this particular DB).
Write a wrapper that
- looks at what components are enabled what option groups/values exist, etc. and runs tests dependent on these values
- allows you to run tests against any version of CiviCRM (by getting a copy of the relevant tests from SVN)
Write some guidelines on writing tests that could run on any installation.
Create some classes that create entities that you would expect to find in these kinds of DBs so they can be used in any tests.
