Problem/Motivation

This is a follow up to #2925899: MigrateUpgradeImportBatch does not use source_private_file_path & source_base_path correctly, making it impossible to have public & private files in separate locations.
benjifisher reported

When I run file migrations and there are missing files, the message often lists a path including //, and I think one of those characters comes from this line. Do we need to add a trailing / here?

$configuration['source']['constants']['source_base_path'] = rtrim($base_path, '/') . '/';

Proposed resolution

TBD

Remaining tasks

User interface changes

API changes

Data model changes

Release notes snippet

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

quietone created an issue. See original summary.

quietone’s picture

Do you recall the reason for the failure? I would be surprised if it was because of the double slash.

And d7_file, d6_file and d7_file_private migrations add a slash.

  source_full_path:
    -
      plugin: concat
      delimiter: /
      source:
        - constants/source_base_path
        - filepath
    -
      plugin: urlencode
benjifisher’s picture

Status: Active » Needs review
FileSize
923 bytes

The failure is typically caused by a missing file. It is not caused by the duplicated slash, but I see // in the logged message.

I am not sure that this is the right way to fix it, but the attached patch removes the . '/'. Let's see what the testbot has to say.

mikelutz’s picture

Status: Needs review » Reviewed & tested by the community
Related issues: +#2804611: Migrate sources and destinations need a way to get their requirements

I've gone back and forth on this 3 times now. Just do it. There is so much wrong with that code block that is all being worked on in #2925899: MigrateUpgradeImportBatch does not use source_private_file_path & source_base_path correctly, making it impossible to have public & private files in separate locations. That particular line seems minor, but it's obvoiusly wrong to have a trailing slash there when the migration definitely appends one, and it's far better to change the core batch UI form to match the migration than to change the migration itself.

Status: Reviewed & tested by the community » Needs work

The last submitted patch, 3: 3151363-3.patch, failed testing. View results

mikelutz’s picture

Status: Needs work » Reviewed & tested by the community

unrelated test failure

catch’s picture

Status: Reviewed & tested by the community » Needs review
Issue tags: +Needs tests

Do we not need test coverage for this one?

quietone’s picture

In my opinion specific tests are not needed. The fact that both the Upgrade6Test and Upgrade7Test pass is sufficient in my mind.

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

quietone’s picture

Well, I suppose it can be tested like this.

The last submitted patch, 10: 3151363-10-fail.patch, failed testing. View results

asishsajeev’s picture

Assigned: Unassigned » asishsajeev
asishsajeev’s picture

Status: Needs review » Reviewed & tested by the community

#10 Patch applied successfully.

asishsajeev’s picture

Assigned: asishsajeev » Unassigned
quietone’s picture

Status: Reviewed & tested by the community » Needs review

@asishsajeev, thank you for setting this patch to RTBC, but having a patch apply successfully is only one of the criteria a patch must met in order to be RTBC. In this case, the patch in #10 adds a new test which needs to be reviewed. If you did that please add a comment explaining what you did to review the test. It takes a while to know the whole process and there is general information about reviews are in the handbook, there is a page about reviewing patches and there is a contributor page about patch review. Thx!

benjifisher’s picture

Status: Needs review » Needs work

Normally I do not review an issue where I have contributed a patch. In this case, my contribution (the non-test part of the patch) was approved in #6. I think it is OK for me to review the test.

  1. I checked the test-only patch, and it fails in the expected way:

Failed asserting that ‘File’/var/www/html/core/modules/migrate_drupal_ui/tests/src/Functional/d7/files//sites/default/files/foo.txt’ does not exist’ contains "/migrate_drupal_ui/tests/src/Functional/d7/files/sites/default/files/foo.txt".

  1. Do we need all this?

     +++ b/core/modules/migrate_drupal_ui/tests/src/Functional/d7/DoubleSlashTest.php
     @@ -0,0 +1,130 @@
     ...
     +    $default_db = Database::getConnection()->getKey();
     +    Database::setActiveConnection($this->sourceDatabase->getKey());
     +    $db = Database::getConnection($this->sourceDatabase->getKey());
     ...
     +    Database::setActiveConnection($default_db);

I think we just need the line that sets $db. Also, Database::setActiveConnection() returns the old value, so we could combine the first two lines if we did need them.

  1.   +    $db = Database::getConnection($this->sourceDatabase->getKey());
      +
      +    $db->update('file_managed')
      +      ->condition('fid', 1)
      +      ->fields([
      +        'filename' => 'foo.txt',
      +        'uri' => 'public://foo.txt',
      +      ])
      +      ->execute();

Since $db is never used again, we could eliminate it. That would also remove the annoyance of the not-quite-aligned -> operators.

  1.   +    // Run the upgrade.
      +    $this->drupalGet('/upgrade');
      +    $session = $this->assertSession();
      +
      +    // Get valid credentials.
      +    $edits = $this->translatePostValues($this->getCredentials());
      +
      +    // Start the upgrade process.
      +    $this->drupalGet('/upgrade');
      +    $session->responseContains("Upgrade a site by importing its files and the data from its database into a clean and empty new install of Drupal $this->destinationSiteVersion.");
      +
      +    $this->submitForm([], t('Continue'));
      +    $session->pageTextContains('Provide credentials for the database of the Drupal site you want to upgrade.');
      +
      +    $this->submitForm($edits, t('Review upgrade'));
      +
      +    $this->submitForm([], t('I acknowledge I may lose data. Continue anyway.'));
      +    $session->statusCodeEquals(200);
      +
      +    $this->submitForm([], t('Perform upgrade'));

It looks as though this is mostly copied from IdConflictTest, and the assertions duplicate parts of that test. Can we get rid of the first drupalGet() and everything involving $session?

quietone’s picture

Status: Needs work » Needs review
FileSize
1.87 KB
3.89 KB

@benjifisher, thanks.

Yes this was a copy/paste from another test file and I meant to come back and clean it up. Clearly I forgot. Sorry.

benjifisher’s picture

Status: Needs review » Reviewed & tested by the community

@quietone, thanks for the quick response!

I think the patch looks good. If the testbot disagrees, then it can overrule me. ;)

alexpott’s picture

Status: Reviewed & tested by the community » Fixed
diff --git a/core/modules/migrate_drupal_ui/tests/src/Functional/d7/DoubleSlashTest.php b/core/modules/migrate_drupal_ui/tests/src/Functional/d7/DoubleSlashTest.php
index 1533ef3488..d1972b529a 100644
--- a/core/modules/migrate_drupal_ui/tests/src/Functional/d7/DoubleSlashTest.php
+++ b/core/modules/migrate_drupal_ui/tests/src/Functional/d7/DoubleSlashTest.php
@@ -2,7 +2,6 @@
 
 namespace Drupal\Tests\migrate_drupal_ui\Functional\d7;
 
-use Drupal\Core\Database\Database;
 use Drupal\Tests\migrate_drupal_ui\Functional\MigrateUpgradeExecuteTestBase;
 use Drupal\migrate\Plugin\MigrationInterface;
 

Remvoed unused use on commit.

  • alexpott committed e2333cf on 9.2.x
    Issue #3151363 by quietone, benjifisher, mikelutz, catch: Double // in...
benjifisher’s picture

Oops. Thanks for catching that!

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.