Problem/Motivation

Just noticed this on https://www.drupal.org/pift-ci-job/518115

Migrate.Drupal\Tests\migrate\Kernel\process\CopyFileTest
✗	
rupal\Tests\migrate\Kernel\process\CopyFileTe
exception: [Other] Line 0 of sites/default/files/simpletest/phpunit-1153.xml:
PHPunit Test failed to complete; Error: PHPUnit 4.8.27 by Sebastian Bergmann and contributors.

....E

Time: 5.37 seconds, Memory: 3.00Mb

There was 1 error:

1) Drupal\Tests\migrate\Kernel\process\CopyFileTest::testRenameFile
mkdir(): File exists

/var/www/html/core/tests/Drupal/KernelTests/Core/File/FileTestBase.php:67
/var/www/html/core/tests/Drupal/KernelTests/KernelTestBase.php:274
/var/www/html/core/tests/Drupal/KernelTests/KernelTestBase.php:233
/var/www/html/core/tests/Drupal/KernelTests/Core/File/FileTestBase.php:39
/var/www/html/core/modules/migrate/tests/src/Kernel/process/CopyFileTest.php:34

FAILURES!
Tests: 5, Assertions: 31, Errors: 1.

Proposed resolution

One option here is to convert to use VFS since this type of clash is totally impossible with that. Doesn't fix the root cause but will fix this fail.

By not using the FileTestBase parent class we use the default VFS of KernelTestBase.

Remaining tasks

User interface changes

None

API changes

None

Data model changes

None

Post #2783075: Add a "download" process plugin, remove remote capability from FileCopy

CommentFileSizeAuthor
#22 random_fail_in-2823400-21.patch2.25 KBAda Hernandez
#22 interdiff-16-21.txt767 bytesAda Hernandez
#16 interdiff-12-16.txt883 bytesAnonymous (not verified)
#16 2823400-16.patch2.24 KBAnonymous (not verified)
#14 interdiff-12-13.txt3.04 KBAnonymous (not verified)
#14 2823400-13.patch5.28 KBAnonymous (not verified)
#13 interdiff-11-12.txt646 bytesAnonymous (not verified)
#13 2823400-12.patch2.24 KBAnonymous (not verified)
#11 2823400-11.patch2.24 KBhardikpandya
#11 interdiff-2823400-2-11.txt1.12 KBhardikpandya
#2 2823400-2.patch2.26 KBalexpott
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

alexpott created an issue. See original summary.

alexpott’s picture

Status: Active » Needs review
FileSize
2.26 KB

This is super amazing since it should be impossible for something to get the same test ID. We've done lots of work on that recently. The only thing I can think is somehow this is about container re-use. One option here is to convert to VFS since this type of clash is totally impossible with that. Doesn't fix the root cause but will fix this fail.

Berdir’s picture

Did you actually mean to write random *file* or should it be random fail? :)

alexpott’s picture

Title: Random file in CopyFileTest » Random fail in CopyFileTest

lol

catch’s picture

Is the temp directory definitely different between test runs?

alexpott’s picture

@catch I don't think it is but that's not the line that has failed. The line that has failed is:

    mkdir($this->siteDirectory, 0775);

And $this->siteDirectory is definitely unique to this test run since we did all the work with test IDs recently. However I think there might be a problem if a previous test has failed on the same test container and things are not being cleaned up properly.

phenaproxima’s picture

Assigned: Unassigned » phenaproxima

Assigning to myself for grokking and review.

star-szr’s picture

Issue tags: +Triaged D8 critical

Discussed with @xjm @effulgentsia @cilefen @catch, and we agreed that this is a critical issue because it's a random failure.

phenaproxima’s picture

  1. +++ b/core/modules/migrate/tests/src/Kernel/process/CopyFileTest.php
    @@ -180,4 +180,39 @@ protected function doImport($source_path, $destination_path, $configuration = []
    +    if (!isset($scheme)) {
    

    I'd rather this were empty() instead of isset(), to protect against anything falsy, but that's not a big deal.

  2. +++ b/core/modules/migrate/tests/src/Kernel/process/CopyFileTest.php
    @@ -180,4 +180,39 @@ protected function doImport($source_path, $destination_path, $configuration = []
    +    if (!isset($contents)) {
    

    Ditto.

  3. +++ b/core/modules/migrate/tests/src/Kernel/process/CopyFileTest.php
    @@ -180,4 +180,39 @@ protected function doImport($source_path, $destination_path, $configuration = []
    +    $this->assertTrue(is_file($filepath), t('The test file exists on the disk.'), 'Create test file');
    

    Why not use assertFileExists()?

phenaproxima’s picture

Status: Needs review » Needs work
Issue tags: +Novice

Tagging for novices -- all I need is my nitpicks in #9 fixed.

hardikpandya’s picture

Assigned: phenaproxima » Unassigned
Status: Needs work » Needs review
FileSize
1.12 KB
2.24 KB

Status: Needs review » Needs work

The last submitted patch, 11: 2823400-11.patch, failed testing.

Anonymous’s picture

FileSize
2.24 KB
646 bytes

#11: fix typo t()

Anonymous’s picture

Status: Needs work » Needs review
FileSize
5.28 KB
3.04 KB

fix similar places in other files

Status: Needs review » Needs work

The last submitted patch, 14: 2823400-13.patch, failed testing.

Anonymous’s picture

Status: Needs work » Needs review

The last submitted patch, 2: 2823400-2.patch, failed testing.

mikeryan’s picture

Status: Needs review » Needs work
+++ b/core/modules/migrate/tests/src/Kernel/process/CopyFileTest.php
@@ -180,4 +180,39 @@ protected function doImport($source_path, $destination_path, $configuration = []
+  function createUri($filepath = NULL, $contents = NULL, $scheme = NULL) {

Yes, FileTestBase is a bit lax in specifying the visibility of its methods - but there's no reason we can't set this protected, is there?

Otherwise, if I'm understanding this correctly (by not extending FileTestBase we get KernelTestBase's VFS root, right?) this looks good. Setting off a bunch of tests just to be sure.

mikeryan’s picture

8.2.x/PHP 5.5/SQLite got sibling random failure #2806697: Random fail for AlreadyInstalledException, retesting that one.

Ada Hernandez’s picture

Status: Needs work » Needs review
mikeryan’s picture

Status: Needs review » Reviewed & tested by the community

Thanks!

Anonymous’s picture

We have a couple of similar places like #9:

+++ b/core/modules/file/src/Tests/FileManagedTestBase.php
@@ -183,12 +183,12 @@ function createUri($filepath = NULL, $contents = NULL, $scheme = NULL) {

-    if (!isset($scheme)) {

-    if (!isset($contents)) {
 

+++ b/core/modules/file/tests/src/Kernel/FileManagedUnitTestBase.php

-    if (!isset($scheme)) {

-    if (!isset($contents)) {

-    $this->assertTrue(is_file($filepath), t('The test file exists on the disk.'), 'Create test file');


+++ b/core/tests/Drupal/KernelTests/Core/File/FileTestBase.php

-    if (!isset($scheme)) {

-    if (!isset($contents)) {

-    $this->assertTrue(is_file($filepath), t('The test file exists on the disk.'), 'Create test file');

We need or not need changes it in this issue too?

klausi’s picture

Status: Reviewed & tested by the community » Needs work
Issue tags: +Needs issue summary update

I don't understand this patch. The exact same method exists on FileTestBase and FileTestBase is a kernel test already? Why should the issue be fixed by just copying a method? Can you explain the solution in the issue summary?

alexpott’s picture

Issue summary: View changes
Status: Needs work » Needs review
Issue tags: -Needs issue summary update

Added explanation of how this fixes this to issue summary.

alexpott’s picture

+++ b/core/modules/migrate/tests/src/Kernel/process/CopyFileTest.php
@@ -13,7 +13,7 @@
-class CopyFileTest extends FileTestBase {
+class CopyFileTest extends KernelTestBase {

This is how we switch to vfs...

klausi’s picture

Issue summary: View changes
Status: Needs review » Reviewed & tested by the community

Ah, sorry.

xjm’s picture

Status: Reviewed & tested by the community » Needs review
Issue tags: -Novice

So this fixes the problem by working around a problem with FileTestBase, but what about all the other tests using FileTestBase? How do we ensure we don't introduce another fail like this in the future when someone else extends that class?

xjm’s picture

It's not a small number; there are like 15 tests that extend FileTestBase. So if it has the potential to cause random fails because of how it interacts with the file system, I really think we should discuss it at least.

catch’s picture

@xjm we should discuss that but probably in a new issue? We might have more test cases to convert, we might end up deprecating the base class, but this specific change seems fine and removes one of 19 critical issues.

mikeryan’s picture

@xjm - if catch's response makes sense to you, can you move back to RTBC?

Thanks.

alexpott’s picture

Well only \Drupal\KernelTests\Core\File\DirectoryTest and \Drupal\KernelTests\Core\File\StreamWrapperTest fail if we remove FileTestBase::setUpFilesystem(). DirectoryTest fails because drupal_realpath() struggles with VFS here for some reason. The other test fails because of:

    1) Drupal\KernelTests\Core\File\StreamWrapperTest::testFileFunctions
    stream_select(): cannot represent a stream of type user-space as a
    select()able descriptor
xjm’s picture

Issue tags: +Needs followup

I think we at least need a followup maybe?

heddn’s picture

Status: Needs review » Reviewed & tested by the community

Opened follow-up #2845122: mkdir(): File exists in BrowserHtmlDebugTrait.php:141 during BrowserTestBase. Back to RTBC now we have the follow-up opened.

  • catch committed d07733f on 8.3.x
    Issue #2823400 by vaplas, Adita, hardik.p, alexpott: Random fail in...

  • catch committed 62ea311 on 8.2.x
    Issue #2823400 by vaplas, Adita, hardik.p, alexpott: Random fail in...
catch’s picture

Status: Reviewed & tested by the community » Fixed

Thanks for opening the follow-up.

Committed/pushed to 8.3.x and cherry-picked to 8.2.x, thanks!

  • catch committed d07733f on 8.4.x
    Issue #2823400 by vaplas, Adita, hardik.p, alexpott: Random fail in...

  • catch committed d07733f on 8.4.x
    Issue #2823400 by vaplas, Adita, hardik.p, alexpott: Random fail in...
xjm’s picture

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

jonathan1055’s picture

Is this happening again? I have two recent examples, but decided to post the details on the open issue #2845122-5: mkdir(): File exists in BrowserHtmlDebugTrait.php:141 during BrowserTestBase