Problem/Motivation
I posted this support request to get some help from the experts in understanding how to debug Drupal's automated tests, based upon what I perceived as erratic behaviour from some automated tests I had written some tests for my project Anonymous Publishing.
[Background: For local testing, I navigate to Administration » Configuration » Development » Testing, check the box for the module I want to test, and press "Run tests". For testing on Drupal.org, I queue a test using the project page "Automated testing" tab.]
When I run the tests on a local test instance, I get the result: "214 passes, 0 fails, 0 exceptions, and 99 debug messages".
When I run the tests on the same version using the Drupal.org test instance, I get: "10 pass, 5 fail".
Not only are the numbers different, but test cases that pass on the local test instance fail when run on Drupal.org.
I initially had no idea about how to go about debugging this, hence this cry for help.
After using a copious amount of debug
statements I finally discovered that the URLs appearing in my local test instance looked like this:
http://EXAMPLE.ORG/node/2/verify?akey=SOME-ACTIVATION-KEY
While those produced by the Drupal.org test instance looked like this.
http://localhost/checkout/?q=node/2/verify&akey=SOME-ACTIVATION-KEY
So it turns that there was a problem with my code after all. My too-crude URL parsing assumes "clean URLs" and do not work on a site that uses the ?q=
construct to indicate the path to a node.
(So Hurrah! for automated testing.)
Proposed resolution
Fix module being tested so that URL-parsing works whether clean URLs are enabled or not.
Remaining tasks
None (closing issue).
Comments
Comment #2
hestenetI am no expert in the test system, but one thing I might suggest is looking at the more detailed results on the Jenkins dispatcher:
https://dispatcher.drupalci.org/job/drupal_d7/9469/
If you drill down into the 'Build Artifacts' directory you may be able to find additional error logs.
You may also be able to use: https://dispatcher.drupalci.org/job/drupal_d7/9469/artifact/jenkins-drup... as a build definition file to re-run the exact test locally.
Also - how are you running the tests locally? Are you using the vagrant box? It's not very well documented yet, but you can find some detail on that here: https://www.drupal.org/node/2848494 <-- and maybe using that to run your local tests will be more consistent with what you see on the test bot.
Comment #3
gisleAdded description of how I run local tests to the issue summary.
Comment #4
drummComment #5
gisleFigured it out!
Updated summary to show solution, and moving to "Fixed".