I've been able to get the Forum Access test down to this:

(screenshot)

The remaining 78 fails are all caused by the same assert, which is executed repeatedly under varying circumstances. It tries to find a comment link by href:

            $this->assertLinkByHref("/comment/$comment->cid#comment-$comment->cid");

This works fine on my local machine, and I'm out of ideas why it could go wrong in the testbot. Is there a way to get a glimpse at the detailed test log and the screenshots before they are deleted?

CommentFileSizeAuthor
#12 FA6_tests.png5.55 KBsalvis
#4 forum_access.tests_.png5.38 KBsalvis
forum_access_7x_test.png5.15 KBsalvis

Comments

rfay’s picture

rfay
he,him
Primary language English
Location Palisade, CO, USA
commented

Your test assumes that clean_urls are on. They're not on the testbot (and are not on in Drupal by default; that's why).

There are two key differences between most people's environments and the testbot:

1. No clean urls by default (although your test could turn this on)
2. It runs in a subdirectory

So a typical URL would be http://drupaltestbot654.osuosl.test/checkout/?q=comment/...

rfay’s picture

rfay
he,him
Primary language English
Location Palisade, CO, USA
commented

I should mention here that you can easily run your own testbot, in a virtual machine, on an VM on the internet, or etc. It's linked to at the bottom of http://drupal.org/project/testbot

Also, I added this question to the FAQ there.

salvis’s picture

salvis
he/his
commented
Status: Active » Fixed

Oh, wow — thank you!

It has been years since I got in contact with 'dirty' urls... :-)

And thanks for the additional information, too.

salvis’s picture

salvis
he/his
commented
Title: Is there a way to get at the detailed log and the screenshots? » Neglecting to use url() in asserts leads to fails on qa.d.o
StatusFileSize
new forum_access.tests_.png5.38 KB 
            $this->assertLinkByHref(url("comment/$comment->cid", array('fragment' => "comment-$comment->cid")));

was the key to getting this to pass.

No sloppy programming in the test code either!

I'm just so thrilled to have recurring tests in place now:

(screenshot, all green!)

Thank you for that!

rfay’s picture

rfay
he,him
Primary language English
Location Palisade, CO, USA
commented

Congratulations!

Just FYI though... they're not recurring at this point, although I hope they will be. Currently tests get executed when you do a commit or post a patch.

20K assertions in D6 and 10K in D7? You're really working that testbot hard!

salvis’s picture

salvis
he/his
commented

20K assertions in D6 and 10K in D7? You're really working that testbot hard!

Yes, I know, and I'm somewhat uneasy that this might get me into trouble...

The problem with Forum Access is that there are many different combinations of permissions and grants that result in all sorts of interactions. If you really want to nail this down, you have to test a large number of these combinations or you risk missing one that could open up unwanted access.

The testbot is amazingly fast though!

jthorson’s picture

jthorson commented

... they're not recurring at this point

Actually, Randy, they are ... I turned them on for Forum Access earlier this week. ;)

rfay’s picture

rfay
he,him
Primary language English
Location Palisade, CO, USA
commented

@jthorson I meant that they don't happen on a repeating basis without something triggering them. Unless you've implemented a feature I've been hoping for.

Examples D8 has already broken several times due to core shifts, but we don't find out until a commit...

salvis’s picture

salvis
he/his
commented

D7 core shouldn't change so much as to regularly break contribs anymore, but the modules that want to track the bleeding edge D8 would benefit greatly from periodic retests, maybe in idle times of the bots, if these exist.

salvis’s picture

salvis
he/his
commented

Re. #1, differences between most people's environments and the testbot:

Everything is green, but we're seeing different pass counts for Forum Access D6 on qa.d.o and when we run the tests locally. Initially, we had the verbose setting on locally, and this resulted in about 10% more passes, because every verbose message increases the pass count by 1.

After turning verbose off, we're seeing about 3% less passes locally (and on the local testbot that daffie set up) than on qa.d.o. Does qa.d.o run some additional tests?

I've tried turning clean URLs off locally, but this didn't make any difference. I've also looked at the D7 version, and got mixed results, see #762270-37: WANTED: SimpleTests for Forum Access.

Any idea what could be causing the different pass counts?

rfay’s picture

rfay
he,him
Primary language English
Location Palisade, CO, USA
commented

For a while on D6 some core tests were being run, but I think that's been fixed.

I think you can look at the summary and see what tests are being run, although that doesn't account for all the assertions counted.

salvis’s picture

salvis
he/his
commented
StatusFileSize
new FA6_tests.png5.55 KB

Yes, these are still running...

Block
 - Block functionality (BlockTestCase)
Forum access
 - Forum access functionality #1 (ForumAccessTestCase1)
 - Forum access functionality #2 (ForumAccessTestCase2)
 - Forum access functionality #3 (ForumAccessTestCase3)
 - Forum access functionality #4 (ForumAccessTestCase4)
 - Forum access functionality #5 (ForumAccessTestCase5)
 - Forum access functionality #6 (ForumAccessTestCase6)
 - Forum access functionality #7 (ForumAccessTestCase7)
SimpleTest
 - SimpleTest browser (SimpleTestBrowserTestCase)
 - Testing SimpleTest setUp (SimpleTestFolderTestCase)
 - SimpleTest functionality (SimpleTestFunctionalTest)
 - SimpleTest e-mail capturing (SimpleTestMailCaptureTestCase)].

... but the way this is presented I don't think their counts are added to the FA counts.

BTW, in the log the tests show up in a systematic way, but further up on http://qa.drupal.org/pifr/test/193549 they look less orderly:

Only local images are allowed.

Status: Fixed » Closed (fixed)

Automatically closed -- issue fixed for 2 weeks with no activity.