Investigate why DrupalCI fails on checkout based Functional and FunctionalJavascript tests. [#2855587]

See: https://www.drupal.org/pift-ci-job/606884

All tests are passing on DrupalCI except for Functional and FunctionalJavascript tests involving the checkout. The key difference is that DrupalCI uses the run-tests.sh runner and executes in parallel.

1) Drupal\Tests\commerce_checkout\Functional\CheckoutOrderTest::testOrderAccess
Behat\Mink\Exception\ExpectationException: Current response status code is 404, but 403 expected.

2) Drupal\Tests\commerce_checkout\Functional\CheckoutOrderTest::testGuestOrderCheckout
Behat\Mink\Exception\ElementNotFoundException: Button with id|name|label|value "Continue as Guest" not found.

3) Drupal\Tests\commerce_checkout\Functional\CheckoutOrderTest::testRegisterOrderCheckout
Behat\Mink\Exception\ResponseTextException: The text "New Customer" was not found anywhere in the text of the current page.

4) Drupal\Tests\commerce_checkout\Functional\CheckoutOrderTest::testOrderSummary
Behat\Mink\Exception\ElementNotFoundException: Element matching css ".view-id-commerce_checkout_order_summary" not found.

5) Drupal\Tests\commerce_checkout\Functional\CheckoutOrderTest::testCheckoutFlowOnCartUpdate
Behat\Mink\Exception\ElementHtmlException: The string "Order information" was not found in the HTML of the element matching css "h1.page-title".

1) Drupal\Tests\commerce_payment\Functional\PaymentCheckoutTest::testCheckoutWithPayment
Behat\Mink\Exception\ResponseTextException: The text "Order Summary" was not found anywhere in the text of the current page.

2) Drupal\Tests\commerce_payment\Functional\PaymentCheckoutTest::testDeclineStopsCheckout
Behat\Mink\Exception\ResponseTextException: The text "Order Summary" was not found anywhere in the text of the current page.

3) Drupal\Tests\commerce_payment\Functional\PaymentCheckoutTest::testTransactionModeAuthorizeOnly
Behat\Mink\Exception\ResponseTextException: The text "Order Summary" was not found anywhere in the text of the current page.

1) Drupal\Tests\commerce_payment\Functional\PaymentCheckoutOffsiteRedirectTest::testCheckoutWithOffsiteRedirectPost
Behat\Mink\Exception\ResponseTextException: The text "Order Summary" was not found anywhere in the text of the current page.

2) Drupal\Tests\commerce_payment\Functional\PaymentCheckoutOffsiteRedirectTest::testCheckoutWithOffsiteRedirectGet
Behat\Mink\Exception\ResponseTextException: The text "Order Summary" was not found anywhere in the text of the current page.

Comment	File	Size	Author
#10	Drupal_Commerce___Drupal_org.png	48.93 KB	mglaman
#8	checkout_order_test.png	82.08 KB	drugan
#8	drupalcommerce_executables.png	71.27 KB	drugan
#4	investigate_why-2855587-4.patch	10.97 KB	mglaman
#4

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

Comment #1

24 February 2017 at 05:47

mglaman created an issue. See original summary.

Comment #2

mglaman

English

WI, USA

CreditAttribution: mglaman at Centarro commented 24 February 2017 at 06:03

It is also reproduceable locally

------------

Drupal\Tests\commerce_checkout\Kernel\ChainCheckoutFlowResol   1 passes                                      
Drupal\Tests\commerce_checkout\Functional\CheckoutFlowTest     4 passes                                      
Drupal\Tests\commerce_checkout\Functional\CheckoutOrderTest    0 passes             1 exceptions             
FATAL Drupal\Tests\commerce_checkout\Functional\CheckoutOrderTest: test runner returned a non-zero error code (2).
Drupal\Tests\commerce_checkout\Functional\CheckoutOrderTest    0 passes   1 fails                            

Test run duration: 3 min 58 sec

That's no good.

Comment #3

drugan CreditAttribution: drugan as a volunteer commented 25 February 2017 at 07:58

Seems that on some environments the relative url path passed to BrowserTestBase->drupalGet() does not become resolved to a proper absolute url.

For example, on my local server the url to the product normally looks like this:

http://mysites/acommerce/web/product/1

But if it is passed as a relative path ($this->product->toUrl()->toString() === /product/1) then it is resolved to this:

http://mysites/acommerce/web/acommerce/product/1

Obviously, there is no such a page, therefore any test will fail.

The method can take as the first argument either relative path or absolute path or Url object. My suggestion is whenever possible to use the Url object:

BrowserTestBase->drupalGet($my_test_entity->toUrl());

If you don't have toUrl() method on your test entity then try this:

$url = BrowserTestBase->getAbsoluteUrl('/relative/url/path/to/my/test/entity');
BrowserTestBase->drupalGet($url);

The PR: https://github.com/drupalcommerce/commerce/pull/652

The patch: https://patch-diff.githubusercontent.com/raw/drupalcommerce/commerce/pul...

Note: the PR is red for now because of this:

The job exceeded the maximum time limit for jobs, and has been terminated.

No errors, no warnings or something indicating a trouble. It happens only on the PHP 7 version.

Does anybody know what is it?

Comment #4

mglaman

English

WI, USA

CreditAttribution: mglaman at Centarro commented 25 February 2017 at 16:51

Status:

Active

» Needs review

File	Size
investigate_why-2855587-4.patch	10.97 KB

Based on drugan's findings, the issue is how we're passing paths in tests. And, as a highlighted, getting DrupalCI to work is critical since we're hitting Travis's build limit of 50 minutes granted to open source projects. Here's a patch based on drugan's initial work with some tweaks.

Comment #5

drugan CreditAttribution: drugan as a volunteer commented 25 February 2017 at 18:09

Great work!

The only notice is that Url object parameter is not documented in BrowsertestBase->drupalGet() and if ever they'll decide to not use buildUrl() for parsing passed $path then we could have a problem.

Still, I hope they will not do this otherwise our own CommerceBrowsertestBase->drupalGet() should be defined.

Just to keep in mind before applying the patch.

Sorry for that I can't test a patch because local server on my notebook is so sloooooooow. It'll take a day for me to run all the tests in the patch :). Even through CLI. But I'm sure they all should pass.

Comment #6

mglaman

English

WI, USA

CreditAttribution: mglaman at Centarro commented 25 February 2017 at 19:06

Status:

Needs review

» Needs work

That patch still failed: https://dispatcher.drupalci.org/job/drupal_contrib/12490/console

So it isn't just the absolute URLs.

Comment #7

mglaman

English

WI, USA

CreditAttribution: mglaman at Centarro commented 25 February 2017 at 19:12

drugan, so the issue is definitely when running the tests within run-tests.sh and having concurrency. TravisCI directly invokes PHPUnit. There's something about how either the tests are written or our checkout access has a weird flaw.

Comment #8

drugan CreditAttribution: drugan as a volunteer commented 26 February 2017 at 13:32

File	Size
drupalcommerce_executables.png	71.27 KB
checkout_order_test.png	82.08 KB

mglaman, thanks for sharing Drupal CI output but it didn't help a lot.

I've tried to run CheckoutOrderTest with run-tests.sh once more on my local server and it's passed.

It is pretty difficult to find the reason for test fail if you can't reproduce it locally. Obviously that on Drupal CI you also can't do any print_r() or similar variable dump just to see what we have at the moment of the test run.

Not sure if that could be related with the issue but somehow I've found that on any of my drupalcommerce clones checkout and payment dot module files have executable bits for all user and group and others. Actually they should not. No any other of drupalcommerce files has it except travis-before-script.sh.

When committing git does not respect any user permissions by default beyond a simple executable bit. Useful info on git administering permissions:

http://stackoverflow.com/questions/10516201/updating-file-permissions-on...

On the latest commit I've already removed executable bits from the dot modules files but if you are not ready to commit this PR yet then try to do the same bits removing and running Drupal CI test again.

Comment #9

7 September 2017 at 14:40

mglaman credited Mixologic.

Comment #10

mglaman

English

WI, USA

CreditAttribution: mglaman at Centarro commented 7 September 2017 at 14:40

Status:

Needs work

» Fixed

File	Size
Drupal_Commerce___Drupal_org.png	48.93 KB

This was due to a core bug where having Drupal in a subdirectory which matches the beginning of a route wreaked havoc. This is why our checkout tests broke. DrupalCI would place Drupal inside the checkout directory. The core bug is reproducible by putting Drupal in coreand trying to install.

The fix can be found here: http://cgit.drupalcode.org/drupalci_testbot/commit/?id=35e96d7