Problem/Motivation

They currently do not, and it causes a lot of weird acrobatics and unsavory hacking in Migrate's tests. The way around this problem, in my opinion, is the path through the jungle: add proper dependency injection to all Migrate source plugins in core and, ultimately, completely remove all mentions of \Drupal from SourcePluginBase and its many descendants.

Steps to reproduce

Proposed resolution

Remaining tasks

Make a new patch here for migrate and move relevant parts of the patch to #2853647: Migrate Drupal source plugins should use dependency injection

User interface changes

N/A

API changes

Data model changes

Release notes snippet

CommentFileSizeAuthor
#51 interdiff_50-51.txt686 bytesimmaculatexavier
#51 2791041-51.patch32.84 KBimmaculatexavier
#50 interdiff_49-50.patch8.64 KBimmaculatexavier
#50 2791041-50.patch32.8 KBimmaculatexavier
#49 reroll_diff_34-49.patch39.39 KBimmaculatexavier
#49 2791041-49.patch31.64 KBimmaculatexavier
#34 2791041-34.patch35.36 KBjofitz
#34 interdiff_31-34.txt980 bytesjofitz
#31 2791041-31.patch35.38 KBjofitz
#31 interdiff_29-31.txt1.68 KBjofitz
#29 2791041-29.patch33.99 KBjofitz
#29 interdiff_27-29.txt3.74 KBjofitz
#27 2791041-27.patch32.17 KBjofitz
#27 interdiff_26-27.txt711 bytesjofitz
#26 2791041-26.patch31.98 KBjofitz
#22 2791041-22.patch31.99 KBshashikant_chauhan
#18 2791041-18.patch31.94 KBshashikant_chauhan
#11 interdiff-2791041-10-11.txt1.56 KBphenaproxima
#11 2791041-11.patch31.92 KBphenaproxima
#10 interdiff-2791041-8-10.txt2.47 KBphenaproxima
#10 2791041-10.patch30.2 KBphenaproxima
#8 interdiff-2791041-6-8.txt6.03 KBphenaproxima
#8 2791041-8.patch27.76 KBphenaproxima
#6 interdiff-2791041-3-6.txt14.29 KBphenaproxima
#6 2791041-6.patch21.27 KBphenaproxima
#3 interdiff-2791041-2-3.txt4.42 KBphenaproxima
#3 2791041-3.patch6.59 KBphenaproxima
#2 2791041-2.patch6.12 KBphenaproxima
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

phenaproxima created an issue. See original summary.

phenaproxima’s picture

Status: Active » Needs review
FileSize
6.12 KB

Here's a start. This will break a huge number of tests.

phenaproxima’s picture

Forgot to inject the module handler into SourcePluginBase.

The last submitted patch, 2: 2791041-2.patch, failed testing.

Status: Needs review » Needs work

The last submitted patch, 3: 2791041-3.patch, failed testing.

phenaproxima’s picture

Status: Needs work » Needs review
FileSize
21.27 KB
14.29 KB

OK, this provides a few necessary mocked services in MigrateSqlSourceTestCase, and removes a couple of protected wrappers around \Drupal.

Status: Needs review » Needs work

The last submitted patch, 6: 2791041-6.patch, failed testing.

phenaproxima’s picture

Status: Needs work » Needs review
FileSize
27.76 KB
6.03 KB

This should fix most, if not all of the remaining failures.

Status: Needs review » Needs work

The last submitted patch, 8: 2791041-8.patch, failed testing.

phenaproxima’s picture

Status: Needs work » Needs review
FileSize
30.2 KB
2.47 KB

Nearly there. This one should pass everything.

phenaproxima’s picture

Dang, there was one more I missed.

The last submitted patch, 10: 2791041-10.patch, failed testing.

heddn’s picture

How or does this effect contrib and custom source plugins? Me thinks it wouldn't.

phenaproxima’s picture

Any SourcePluginBase subclasses that use ContainerFactoryPluginInterface, or are calling parent::__construct() -- which they should be doing -- will break because the arguments to the constructor have changed for all of SourcePluginBase's descendants. It's extremely likely that virtually all source plugins in contrib, and any custom ones as well, will be extending it since it does a lot of important and complicated stuff (particularly in its Iterator methods).

mikeryan’s picture

Version: 8.3.x-dev » 8.2.x-dev

Moving to 8.2.x - we are trying very hard to stop BC breaks (move migrate to experimental beta status and commit to a stable API) with 8.2.

Status: Needs review » Needs work

The last submitted patch, 11: 2791041-11.patch, failed testing.

phenaproxima’s picture

Issue tags: +Needs reroll
shashikant_chauhan’s picture

Status: Needs work » Needs review
Issue tags: -Needs reroll
FileSize
31.94 KB

Created patch from patch 11 for branch 8.2.x

heddn’s picture

If this is going to get into migrate, then this will need to get in before it bumps from experimental.

heddn’s picture

Discussed with @phenaproxima and removing this as a blocker from beta. While it is a BC break, it should not block beta.

iMiksu’s picture

Status: Needs review » Needs work
Issue tags: +Needs reroll

#18 needs reroll.

Your branch is up-to-date with 'origin/8.2.x'.
$ curl https://www.drupal.org/files/issues/2791041-18.patch | patch -p1
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 32703  100 32703    0     0  48926      0 --:--:-- --:--:-- --:--:-- 48956
patching file core/modules/migrate/src/Plugin/migrate/source/EmbeddedDataSource.php
patching file core/modules/migrate/src/Plugin/migrate/source/SourcePluginBase.php
Hunk #1 succeeded at 2 with fuzz 2.
Hunk #2 succeeded at 14 (offset 2 lines).
Hunk #3 FAILED at 26.
Hunk #4 succeeded at 150 with fuzz 1 (offset 10 lines).
Hunk #5 succeeded at 178 (offset 13 lines).
Hunk #6 succeeded at 200 (offset 13 lines).
Hunk #7 FAILED at 405.
Hunk #8 FAILED at 418.
3 out of 8 hunks FAILED -- saving rejects to file core/modules/migrate/src/Plugin/migrate/source/SourcePluginBase.php.rej
patching file core/modules/migrate/src/Plugin/migrate/source/SqlBase.php
patching file core/modules/migrate/tests/src/Unit/MigrateSourceTest.php
Hunk #1 FAILED at 89.
Hunk #2 FAILED at 126.
Hunk #3 FAILED at 258.
Hunk #4 succeeded at 305 (offset 26 lines).
Hunk #5 succeeded at 315 (offset 26 lines).
Hunk #6 FAILED at 326.
Hunk #7 succeeded at 339 (offset 26 lines).
Hunk #8 FAILED at 355.
Hunk #9 succeeded at 368 (offset 26 lines).
Hunk #10 FAILED at 384.
Hunk #11 succeeded at 397 (offset 26 lines).
Hunk #12 succeeded at 453 (offset 41 lines).
6 out of 12 hunks FAILED -- saving rejects to file core/modules/migrate/tests/src/Unit/MigrateSourceTest.php.rej
patching file core/modules/migrate/tests/src/Unit/MigrateSqlSourceTestCase.php
Hunk #1 FAILED at 2.
Hunk #2 succeeded at 85 with fuzz 2 (offset 7 lines).
Hunk #3 succeeded at 123 (offset 28 lines).
1 out of 3 hunks FAILED -- saving rejects to file core/modules/migrate/tests/src/Unit/MigrateSqlSourceTestCase.php.rej
patching file core/modules/migrate_drupal/src/Plugin/migrate/source/DrupalSqlBase.php
patching file core/modules/migrate_drupal/src/Plugin/migrate/source/EmptySource.php
patching file core/modules/migrate_drupal/src/Plugin/migrate/source/Variable.php
patching file core/modules/migrate_drupal/src/Plugin/migrate/source/d6/i18nVariable.php
patching file core/modules/migrate_drupal/tests/src/Unit/source/DrupalSqlBaseTest.php
Hunk #2 succeeded at 46 with fuzz 1 (offset -1 lines).
patching file core/modules/migrate_drupal/tests/src/Unit/source/d6/Drupal6SqlBaseTest.php
shashikant_chauhan’s picture

Status: Needs work » Needs review
FileSize
31.99 KB

adding updated patch.

Status: Needs review » Needs work

The last submitted patch, 22: 2791041-22.patch, failed testing.

Manuel Garcia’s picture

Issue tags: -Needs reroll
jofitz’s picture

Issue tags: +Needs reroll
jofitz’s picture

Issue tags: -Needs reroll
FileSize
31.98 KB

Re-roll (but will still fail tests - further patch pending...).

jofitz’s picture

Status: Needs work » Needs review
FileSize
711 bytes
32.17 KB

Set missing module handler property of SourcePluginBase.

Status: Needs review » Needs work

The last submitted patch, 27: 2791041-27.patch, failed testing.

jofitz’s picture

Status: Needs work » Needs review
FileSize
3.74 KB
33.99 KB
  1. Set missing cache property of SourcePluginBase.
  2. Correctly mock-up container in MigrateSqlSourceTestCase.
  3. Add missing dependencies to d6/Node.

Status: Needs review » Needs work

The last submitted patch, 29: 2791041-29.patch, failed testing.

jofitz’s picture

Status: Needs work » Needs review
FileSize
1.68 KB
35.38 KB

Correct final 2 test failures.

Status: Needs review » Needs work

The last submitted patch, 31: 2791041-31.patch, failed testing.

quietone’s picture

  1. +++ b/core/modules/migrate/src/Plugin/migrate/source/EmbeddedDataSource.php
    @@ -1,11 +1,19 @@
    + * Expects at least two configuration values:
    

    Let's make this similar to the process plugin documentation.
    s/Expects at least two configuration values:/Available configuration keys:/

  2. +++ b/core/modules/migrate/src/Plugin/migrate/source/EmbeddedDataSource.php
    @@ -1,11 +1,19 @@
    + *
    

    Remove blank line.

  3. +++ b/core/modules/migrate/src/Plugin/migrate/source/EmbeddedDataSource.php
    @@ -1,11 +1,19 @@
    + * - 'data_rows': Array of data rows, each one an array of values keyed by
    

    s/'data_rows'/data_rows/

  4. +++ b/core/modules/migrate/src/Plugin/migrate/source/EmbeddedDataSource.php
    @@ -1,11 +1,19 @@
    + *
    

    Remove blank line.

  5. +++ b/core/modules/migrate/src/Plugin/migrate/source/EmbeddedDataSource.php
    @@ -1,11 +1,19 @@
    + * - 'ids': Array describing the unique ID fields for the source data, keyed
    

    s/'ids'/id/

jofitz’s picture

Status: Needs work » Needs review
FileSize
980 bytes
35.36 KB

Comment corrections made.

heddn’s picture

Status: Needs review » Needs work
  1. +++ b/core/modules/migrate/src/Plugin/migrate/source/SourcePluginBase.php
    @@ -146,7 +150,7 @@
    +  public function __construct(array $configuration, $plugin_id, $plugin_definition, MigrationInterface $migration, ModuleHandlerInterface $module_handler, CacheBackendInterface $cache) {
    

    I don't see how this can be done without breaking BC.

  2. +++ b/core/modules/migrate/src/Plugin/migrate/source/SourcePluginBase.php
    @@ -169,6 +173,24 @@ public function __construct(array $configuration, $plugin_id, $plugin_definition
    +
    +    $this->moduleHandler = $module_handler;
    +
    +    $this->cache = $cache;
    

    Nit: not sure we need all this extra whitespace here.

  3. +++ b/core/modules/migrate/src/Plugin/migrate/source/SqlBase.php
    @@ -62,8 +63,8 @@
    +  public function __construct(array $configuration, $plugin_id, $plugin_definition, MigrationInterface $migration, ModuleHandlerInterface $module_handler, CacheBackendInterface $cache, StateInterface $state) {
    

    This feels like a BC break. Lots of contrib and custom migrations extend from SqlBase and might be calling parent. With this addition, it will break them.

phenaproxima’s picture

Version: 8.2.x-dev » 9.x-dev

This is a big fat BC break (I knew that when I wrote it), and Migrate is now in beta.

This cannot be done in Drupal 8. Regrettable, but we'll live.

joelpittet’s picture

Discussed with @phenaproxima in IRC and followed up with splitting out the migrate_drupal part since it's not in beta yet.

#2853647: Migrate Drupal source plugins should use dependency injection

phenaproxima’s picture

Issue tags: +Needs reroll

This will need a reroll once #2853647: Migrate Drupal source plugins should use dependency injection lands. And, y'know, once we start working on Drupal 9.

heddn’s picture

Status: Needs work » Postponed

Let's postpone this.

Version: 9.x-dev » 9.0.x-dev

The 9.0.x branch will open for development soon, and the placeholder 9.x branch should no longer be used. Only issues that require a new major version should be filed against 9.0.x (for example, removing deprecated code or updating dependency major versions). New developments and disruptive changes that are allowed in a minor version should be filed against 8.9.x, and significant new features will be moved to 9.1.x at committer discretion. For more information see the Allowed changes during the Drupal 8 and 9 release cycles and the Drupal 9.0.0 release plan.

xjm’s picture

Version: 9.0.x-dev » 9.1.x-dev

This would be a minor-only change (and it would need to be done with BC/deprecation). Since 8.9.x and 9.0.x are now in beta, I'm moving this to 9.1.x. Thanks!

Sahil Khambra’s picture

Assigned: Unassigned » Sahil Khambra
Status: Postponed » Active

I am taking this issue and try to resolve this.

Sahil Khambra’s picture

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

DamienMcKenna’s picture

Assigned: Sahil Khambra » Unassigned

There has not been progress in a year, so I'm unassigning it.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

immaculatexavier’s picture

Reroll patch against 9.5
Attached reroll diff against #34

immaculatexavier’s picture

Reroll patch against #49
Attached interdiff

immaculatexavier’s picture

Reroll patch against #50
Attached interdiff

Status: Needs review » Needs work

The last submitted patch, 51: 2791041-51.patch, failed testing. View results

quietone’s picture

Issue summary: View changes

@immaculatexavier, thanks for working on this issue! Have a look at the instructions for running the coding standard checks locally so you can be sure the tests will run before uploading a patch. It also has the advantage of saving resources, including money, for the Drupal association. There are real costs for running the tests.

It is good practice to test any tests that your patch changes before uploading your patch to an issue. Running PHPUnit tests.

According to #38. This will need a reroll after
#2853647: Migrate Drupal source plugins should use dependency injection. The other issue is not yet committed. The work in the latest patch includes work that should be in the other issue.

This is needs work to move the migrate_drupal parts to the other issue.

quietone’s picture

Status: Needs work » Postponed

Actually, because of #20 in the other issue, I am postponing until discussed at a migrate meeting. I have added a comment to #3281351: [meeting] Migrate Meeting 2022-05-26 2100Z.

Version: 9.5.x-dev » 10.1.x-dev

Drupal 9.5.0-beta2 and Drupal 10.0.0-beta2 were released on September 29, 2022, which means new developments and disruptive changes should now be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.