It's pretty well known that manually testing software changes is better than not testing changes at all, but it's even better to have a set of code tests that can be run automatically. In the Drupal world, this means writing tests using the Simpletest APIs, or on Drupal 8, PHPUnit. Having tests in code will mean that any time someone uploads a patch the drupal.org automated systems will kick in and, hopefully, confirm there are no regressions (i.e. bugs) from the proposed changes in the patch. When a module has a good set of tests this can really make everyone's lives easier - the maintainers can be sure that changes they make aren't going to break things for existing sites, and contributors will be informed if their proposed changes will need to be further improved.
The problem
One problem with drupal.org's tests system that I've run into a few times is related to adding new dependencies to a module. An example I ran into recently was on a patch by well known contributor Stefan "stefan.r" Ruijsenaars which was adding Search API integration to Metatag. Part of the patch was a new test file that would help ensure the integration was working correctly. When the patch was uploaded the testbots kept showing this error:
fail: [Other] Line 48 of sites/all/modules/metatag/tests/metatag.search_api.test:
Enabled modules: entity, search_api, metatag, metatag_search_testThis error corresponded to the following line of code from the new metatag.search_api.test file:
public function setUp() {
parent::setUp('entity', 'search_api', 'metatag', 'metatag_search_test');This simple line enables Metatag, the Entity API and Search API modules, and a tiny new module that was used to help the tests work correctly. The fact that it fails for this line simply means that the testbot was unable to enable these modules.
Rather strange, right?
Digging in
The testbot only makes modules available that have been downloaded. It knows what modules to download based upon the various info files found in the current branch. That means that it will determine which modules to download for the Metatag module's tests based upon the information in the main metatag.info file, i.e. the dependency lines.
A normal Drupal 7 module can indicate dependencies by adding a line that begins with "dependencies[] =" and then the name of the module it needs. A simple example is the following line from metatag.info:
dependencies[] = tokenThat informs Drupal, and the testbots, that the Metatag module requires the Token module, and if that module is not available then the Metatag module can't be installed.
This is starting to get close to the root of the problem.
Sometimes, however, a module isn't strictly needed, but it can be useful to test integration with it if it happens to be available. Stefan's patch didn't make Metatag *require* the Search API module, but it still needed to indicate to the testbots that it (and Entity API) should be downloaded anyway so the tests could run. Thankfully there's a way of handling this.
The testbots can be told to download additional modules by using a "test_dependencies" line in the info file. To make the testbots download Entity API and Search API modules Stefan added the following lines to the metatag.info file:
; These are required for the Search API integration.
test_dependencies[] = entity
test_dependencies[] = search_apiWhen Stefan (and I) ran the tests locally it worked just fine, but the drupal.org testbots continued to fail saying that the modules could not be enabled. So what gives?
The problem stems from something I mentioned above - the testbots download the current codebase and use the module's info files to identify the other modules to download. This means that when the testbots are running their Metatag tests they look at metatag.info and then download CTools, Token, Devel, Imagecache Token, Entity Translation, i18n, Context, Panels, Views, File Entity and Media. However, they won't download Entity API or Search API because the current 7.x-1.x codebase doesn't tell them to!
When Stefan and I ran the tests we already had Entity API and Search API downloaded to our local codebases, but the testbots do not.
And this is the root of the problem.
So here's how to fix the problem.
The solution
The way to get new test dependencies added so that a module can expand its tests is to first commit a small patch that adds the new dependencies, and then upload a patch that includes all of the remaining functionality.
For the changes Stefan was trying to add, we had to first split off the test_dependencies lines into a new patch and commit just that piece, and then work on the rest of the patch.
As can be seen from the next patch, once the testbot knew to download the two new modules the tests could run successfully. Tadaa!
Incidentally there is an ongoing discussion on changing this behaviour. Because the problem is somewhat deeply ingrained in the testbot architecture it might not be an easy fix, but maybe someone out there might have the time to dig into it.