Set up to run tests on DrupalCI

As part of this setup, we will need to apply a patch that gets around issue #2905295: Configuration language being overwritten during module install. Here is the necessary patch (see also the README.txt file in the auto_screenshots directory, which explains this patch).

It looks like what I actually need to do here is move the tests and auto screenshots code to a separate project. DrupalCI seems to have a problem dealing with the User Guide repo, because it is VERY large (with all the images for all the languages). So... I'll be setting that up shortly, and then once that is done, deleting this part of the code from the existing User Guide project and adding a note to the README and to the project page about the separate project just for testing.

Moving to the new project...

Well, I finally got the environment set up on DrupalCI. But the tests are failing the first time they come to a screenshot. It looks like the image from the chromedriver browser is empty, because when I try to auto-crop it down with
http://us1.php.net/manual/en/function.imagecropauto.php
the return value is a Boolean (false I would assume).

It looks like this is because Chrome is being set up to run in "headless" mode (no UI). And it looks like when it's in headless mode, screenshot images come up blank:
https://stackoverflow.com/questions/52416069/how-to-get-screenshots-from...
https://serverfault.com/questions/856748/selenium-chromedriver-makes-bla...
https://bugs.chromium.org/p/chromium/issues/detail?id=714058

It seems to be a bug in Chrome and/or Chromedriver, because it is supposed to work. And it looks like the bug has been fixed in the dev versions of Chromium, but the fix may not be in the version of Chrome/Chromium that DrupalCI is running.

Someone posted a work-around to unset the "TZ" environment variable before running Chrome, via unset TZ, so I am going to try adding that in.

That TZ setting didn't help. I'll have to talk to infra next week.

Updating summary with remaining To Dos if I get the tests running.

I created (temporarily) a Drupal Core test to verify that screenshots were not all blank on this related issue. The test passed. So, I am trying to recreate the problem in a new test file in this project...

Good point, I will check on that shortly! Adding a bit to a new test here too... It looks like it is specifically the cropping that is failing. When I test pixels, that works in this environment.

To clarify:

The test I patched in the drupal core issue took a screenshot and verified it had at least 2 colors in it.

The test I am trying here (in commits on this issue) goes one step further and uses the imagecropauto GD function to crop it down, which is what is currently failing in my (much larger) tests (the ones that work on my home system but not on DrupalCI).

The fail I am seeing here in my latest tests on drupal CI is in the crop step. The part that verifies the colors is working. So, I have added some to the test so that it will also save the uncropped image so I can look at it manually.

I should check on PHP 7.2 though, that's a good point. My home system where the tests were working is running PHP 7.2.15.

Sigh. I really don't know what is going on.

In my latest test, I saved the image to a file before it was cropped. The image looks fine:
https://dispatcher.drupalci.org/job/drupal_contrib/339932/artifact/jenki...

Then just after saving that image, I do in my test

    $cropped = imagecropauto($image, IMG_CROP_SIDES);
    $this->assertTrue($cropped !== FALSE, 'Image crop succeeded');

That fails. However, I tried these commands at home in php -- interactive:

$image = imagecreatefrompng('test-screenshot-before-crop.png');
$cropped = imagecropauto($image, IMG_CROP_SIDES);
if ($cropped !== FALSE) { print "full image"; } else { print "empty image"; }
print imagesx($cropped);

This prints out "full image" and a 650 pixels width. So, imagecropauto is working fine on my local system, at least after importing the saved image file, but it is failing on the bot.

I have now also tested this in both PHP 7.1 and 7.2, and it behaves the same.

I am trying to do IMAGE_CROP_SIDES, and that is failing in my pared-down test... although I could try the THRESHOLD function and just crop white, which is probably what I need in 99% of the images anyway.

In the cases in #33, the problem there is that not enough of the image is cropped, not that the whole thing is cropped.

But yeah I'll try the threshold crop and see if that works. Good idea!

I played around with the threshold cropping on my local PHP install, and it looks like this line works the closest to the (working for me) IMG_CROP_SIDES:

    $cropped = imagecropauto($image, IMG_CROP_THRESHOLD, 5, 0xFFFFFF);

So that last commit tries that in my DrupalScreenshotTest. Should have a result in a little while...

Thanks for the suggestions @tacituseu

Sigh. The IMG_CROP_THRESHOLD change did not fix the problem either. The cropping still fails on DrupalCI.

So, I think I just have to not do the cropping. It's not terrible -- I can crop the images at the command line afterwards with a simple shell script.

I just turned on cropping again in my short test, and it even failed on PHP 7.3.x-dev. So, I think it is just too problematic.

OK, now that I am not trying to crop images, I have a new problem with some of the tests. Again, this works locally but not on DrupalCI.

The test is attempting, at several points, to submit an image file to an image file field on a node/add page.

Earlier in the test, I have verified that the file exists and is readable -- at least, it is readable by the web server user:

    $this->assetsDirectory = drupal_get_path('module', 'user_guide_tests') . '/assets/';
    $this->assertTrue(is_readable(DRUPAL_ROOT . '/' . $this->assetsDirectory . 'farm.jpg'), 'Farm asset file exists and is readable');

But the test is failing on this line later on, when trying to upload that exact same file:

    $this->drupalPostForm(NULL, [
        'title[0][value]' => $this->demoInput['vendor_1_title'],
        'files[field_' . $main_image . '_0]' => DRUPAL_ROOT . '/' . $this->assetsDirectory . 'farm.jpg',
        'path[0][alias]' => $this->demoInput['vendor_1_path'],
        'field_' . $vendor_url . '[0][uri]' => $this->demoInput['vendor_1_url'],
      ], $this->callT('Save'));

with this message and call stack:

Exception: invalid argument: File not found : /var/www/html/modules/contrib/user_guide_tests/assets/farm.jpg

/var/www/html/vendor/instaclick/php-webdriver/lib/WebDriver/Exception.php:144
/var/www/html/vendor/instaclick/php-webdriver/lib/WebDriver/AbstractWebDriver.php:157
/var/www/html/vendor/instaclick/php-webdriver/lib/WebDriver/AbstractWebDriver.php:218
/var/www/html/vendor/instaclick/php-webdriver/lib/WebDriver/Container.php:224
/var/www/html/vendor/behat/mink-selenium2-driver/src/Selenium2Driver.php:670
/var/www/html/vendor/behat/mink/src/Element/NodeElement.php:105
/var/www/html/core/tests/Drupal/Tests/UiHelperTrait.php:95
/var/www/html/core/tests/Drupal/Tests/UiHelperTrait.php:209
/var/www/html/modules/contrib/user_guide_tests/tests/src/FunctionalJavascript/UserGuideDemoTestBase.php:3643
/var/www/html/modules/contrib/user_guide_tests/tests/src/FunctionalJavascript/UserGuideDemoTestBase.php:1127
/var/www/html/modules/contrib/user_guide_tests/tests/src/FunctionalJavascript/UserGuideDemoTestBase.php:316

So, apparently the file or directory is not readable by the chromedriver.

@lendude helped me out in Slack with this. Apparently in WebDriverTestBase, when running on DrupalCI, you cannot post forms with images. You need to use attachFileToField() instead. So... a new commit for that.

Thanks! I doubt that is the problem but I have just fixed it. That was left over from when this test had a different directory structure.

Making progress! The next problem I've encountered is that in one part of the test, we go to the Available Updates page to see what updates are available. When running the tests in person, the environment set up (pre-using-composer) had you download a not-quite-current version of a contrib module and a contrib theme, so that there would be something on this page.

The current setup is using Composer to download these by putting explicit versions in the composer.json file, but they are not showing up on Available Updates. Having run the same composer commands on my local system, I can see that the correct versions are being downloaded from Git anyway. Also, you can see it in the build artifacts for the test run:

Package operations: 3 installs, 0 updates, 0 removals
  - Installing drupal/admin_toolbar (1.25.0): Cloning 8.x-1.25
> Drupal\Core\Composer\Composer::vendorTestCodeCleanup
  - Installing drupal/mayo (1.2.0): Cloning 8.x-1.2

However, I can see on my local system that the info.yml files for Mayo and Admin Toolbar don't have the versions in them (they would be in there if you downloaded the tar file but not via Git), and I can see from the screenshots in the test output that the Mayo theme is showing up on the Appearance page as being version 8.6.10 (the same as the version of Core I am using). It looks like when Drupal parses .info files, it will put in whatever is in core/lib/Drupal.php in the VERSION constant as a default (which is the core version).

So I think I just need to add two lines to my build to put the versions into the info.yml files for Mayo and Admin Toolbar. Should be easy enough...

There has been a problem in some languages early in the test with the node/add/page testing. This turned out to be a Core bug:
#3037729: Duplicate filter format weights (turns off CKEditor by default in some languages)
which I patched, so adding this patch to the drupalci.yml file.

Also there is still a problem in the updates report that I cannot reproduce locally. Trying a work-around...

Woot! The tests finally all passed! I fixed the problem with the update report by adding to the info.yml files for the contrib theme/module, so they look like what you get when you download a tar.gz file of a release (i.e, added the project, version, and timestamp lines) -- this allowed the Update module to get correct update information.

So, I'm now going over to #3035960: Export new database and file backups for all languages to update the screenshots (as needed) and backups.

Thanks again to @tacituseu, @Mixologic, and @Lendude for all of your help, on this issue and in Slack!

Yeah, I've filed an issue in Mayo about a config problem as well -- could be a duplicate?
#2790873: Missing schema
Oh, both are by me. :)