Problem/Motivation

I posted this support request to get some help from the experts in understanding how to debug Drupal's automated tests, based upon what I perceived as erratic behaviour from some automated tests I had written some tests for my project Anonymous Publishing.

[Background: For local testing, I navigate to Administration » Configuration » Development » Testing, check the box for the module I want to test, and press "Run tests". For testing on Drupal.org, I queue a test using the project page "Automated testing" tab.]

When I run the tests on a local test instance, I get the result: "214 passes, 0 fails, 0 exceptions, and 99 debug messages".
When I run the tests on the same version using the Drupal.org test instance, I get: "10 pass, 5 fail".

Not only are the numbers different, but test cases that pass on the local test instance fail when run on Drupal.org.

I initially had no idea about how to go about debugging this, hence this cry for help.

After using a copious amount of debug statements I finally discovered that the URLs appearing in my local test instance looked like this:

http://EXAMPLE.ORG/node/2/verify?akey=SOME-ACTIVATION-KEY

While those produced by the Drupal.org test instance looked like this.

http://localhost/checkout/?q=node/2/verify&akey=SOME-ACTIVATION-KEY

So it turns that there was a problem with my code after all. My too-crude URL parsing assumes "clean URLs" and do not work on a site that uses the ?q= construct to indicate the path to a node.

(So Hurrah! for automated testing.)

Proposed resolution

Fix module being tested so that URL-parsing works whether clean URLs are enabled or not.

Remaining tasks

None (closing issue).

CommentFileSizeAuthor
z.png16.8 KBgisle
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

gisle created an issue. See original summary.

hestenet’s picture

I am no expert in the test system, but one thing I might suggest is looking at the more detailed results on the Jenkins dispatcher:
https://dispatcher.drupalci.org/job/drupal_d7/9469/

If you drill down into the 'Build Artifacts' directory you may be able to find additional error logs.

You may also be able to use: https://dispatcher.drupalci.org/job/drupal_d7/9469/artifact/jenkins-drup... as a build definition file to re-run the exact test locally.

Also - how are you running the tests locally? Are you using the vagrant box? It's not very well documented yet, but you can find some detail on that here: https://www.drupal.org/node/2848494 <-- and maybe using that to run your local tests will be more consistent with what you see on the test bot.

gisle’s picture

Issue summary: View changes

Added description of how I run local tests to the issue summary.

drumm’s picture

Project: Drupal.org infrastructure » DrupalCI: Drupal.org Testing Infrastructure
Component: Other » Unexplained Failures/Support
gisle’s picture

Component: Unexplained Failures/Support » Miscellaneous
Assigned: Unassigned » gisle
Issue summary: View changes
Status: Active » Fixed

Figured it out!

Updated summary to show solution, and moving to "Fixed".

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.