This is not exactly an unexplained test "failure", it's more of an unexplained test "bypass". The tests I have added to my project do pass, but one case is skipped without any explanation in the testbot log - https://qa.drupal.org/pifr/test/664058. The test in question is of the class JuiceboxFileEntityCase and it gets picked-up and runs just fine locally, but there is almost no trace of it from the perspective of the testbot.

According to the log, the test class's file is found, and it's included the in the ./scripts/run-tests.sh command parameters, but the "test to be run" list simply omits it. There is no further explanation/error that I can find as to why this may be.

The only thing I can see that my be unique for this case is that it has a contrib dependency. This in itself is not odd (as I have other tests in the project with contrib dependencies that run just fine with the testbot), but perhaps something about the specific dependency versioning may be related? The dependency is File Entity (file_entity), which currently does not have a stable release (it sits at 7.x-2.0-alpha3), though I assume that should not be an issue. I did have a look at the file_entity.info file as I understand the info files are parsed ahead of time for testbot processing, and I did find something strange in there. Even in the release tagged 7.x-2.0-alpha3 the following line appears at the end of file_entity.info:

; We have to add a fake version so Git checkouts do not fail Media dependencies
version = 7.x-2.x-dev

Could be related? If there is an issue with the dependency for a test, can that lead to this kind of "silent" bypassing of a test?

I hope I'm not missing something obvious here.

Comments

rjacobs’s picture

rjacobs commented
Status: Active » Closed (works as designed)

Alright, never-mind.

It seems I was initially led to believe that adding 'dependencies' => array('file_entity') to getInfo() was a way to declare a test case dependency, but I guess that only tells simpletest to skip the test if the dependency is not already declared by other means.

I now added this to my module's main .info file:

test_dependencies[] = file_entity

And thing seem to work.

rfay’s picture

rfay
he,him
Primary language English
Location Palisade, CO, USA
commented

Sorry I didn't ever get to analyzing this one, but glad you got it!

FYI there's an FAQ on http://drupal.org/project/testbot about what would be different about tests locally vs on the testbot. So often people have the modules installed that they need locally... not knowing that the testbot has to check out an exact set.

rjacobs’s picture

rjacobs commented

Thanks Randy,

The thing that mostly unraveled this for was a read-through this old issue: #698932: "test_dependencies" for dependencies / integration tests, which I stumbled upon somewhat randomly.

It might be worth adding some notes to the FAQ that help clarify the different significances of:

  • Adding 'dependencies' to the info array in getInfo(), e.g., this seems to just disable the test if the dependencies aren't there.
  • Adding a hard dependency in the module's info file, e.g., this seems to pull in the dependency automatically for all tests.
  • Adding a hard dependency to a "helper" module which is then included with setUp(), e.g., this seems to ensure the dependency is pulled-in for all tests as all modules listed with the project seems to have their .info files scanned by testbot.
  • Adding test_dependencies[] entries to the main module file, e.g., this seems to have the same effect as the "helper" module idea above.

Sorting those things out was a little tricky for me, and may be for some others. Just a thought.

Also, on the topic of the FAQ (and unrelated to dependencies)... it's probably worth expanding on this point:

The testbot has clean URLs turned off by default (as does Drupal)

Possibly by noting that when testing locally the current "clean_urls" setting (from the environment calling the tests) is set within the test environment. testbot on the other hand always disables clean_urls. It took me a while to figure this out. I actually understood why my tests failed on d.o., but I was totally vexed as to how they ever passed locally (because the last part of the quote above seemed to imply that simpletest always used clean_urls when spinning-up a test environment). I understand that this is a characteristic of local tests (the testing module), and not a testbot, but more clarity here could help people understand such discrepancies.

Some of these points could possibly go into a docs page (and I know I can edit those, and may make some additions), but I figure those FAQs might get more traffic, so some of this info may be valid there.