Problem/Motivation

Presently the node source in a migration can only support one node type.

It happens that one can need multiple node types in one migration (like a node alias migration).

This is already supported by taxonomy source plugins:
https://git.drupalcode.org/project/drupal/-/blob/9.2.x/core/modules/taxo...

Proposed resolution

Follow the taxonomy term source plugin as an example and change
$query->condition('n.type', $this->configuration['node_type']);
to
$query->condition('n.type', (array) $this->configuration['node_type'], 'IN');

This was discussed by the migrate maintainer and there were some concerns about adding this feature to core (rather than migrate plus) since this doesn't directly help core's built-in migrations. But it does help contrib and definitely custom migrations. It is a really trivial change and will make filtering things from legacy sites just that much easier. See #6

Issue fork drupal-2937989

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Support from Acquia helps fund testing for Drupal Acquia logo

Comments

jcisio created an issue. See original summary.

jcisio’s picture

Status: Active » Needs review
FileSize
1.25 KB

For me, $query->condition should default the operator to different value based on the $value argument, but it's another history. Now just fix this in the migration system.

heddn’s picture

Status: Needs review » Needs work
Issue tags: +Needs tests

Can we add a test?

Version: 8.6.x-dev » 8.7.x-dev

Drupal 8.6.0-alpha1 will be released the week of July 16, 2018, which means new developments and disruptive changes should now be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

quietone’s picture

Status: Needs work » Needs review
Issue tags: -Needs tests
FileSize
6.59 KB
5.34 KB

Added more tests to the source plugin. Tests getting 1 content type, 2 content types and all content types if the configuration key 'node_type' is not set.

heddn’s picture

Status: Needs review » Reviewed & tested by the community

Discussed this migrate maintainers calls. Code looks fine. We had some concerns about adding this feature to core (rather than migrate plus) since this doesn't directly help core's built-in migrations. But it does help contrib and definitely custom migrations. It is a really trivial change and will make filtering things from legacy sites just that much easier.

The last submitted patch, 2: core-2937989-node-source-multiple.patch, failed testing. View results

alexpott’s picture

Status: Reviewed & tested by the community » Needs work
  1. +++ b/core/modules/node/src/Plugin/migrate/source/d6/Node.php
    @@ -113,7 +113,7 @@ public function query() {
    -      $query->condition('n.type', $this->configuration['node_type']);
    +      $query->condition('n.type', $this->configuration['node_type'], is_array($this->configuration['node_type']) ? 'IN' : '=');
    
    +++ b/core/modules/node/src/Plugin/migrate/source/d7/Node.php
    @@ -94,7 +94,7 @@ public function query() {
    -      $query->condition('n.type', $this->configuration['node_type']);
    +      $query->condition('n.type', $this->configuration['node_type'], is_array($this->configuration['node_type']) ? 'IN' : '=');
    

    This can be written with less conditions - which is always a bit simpler.

    $query->condition('n.type', (array) $this->configuration['node_type'], 'IN');

    Going to commit this anyway this the utility is more important but someone can feel free to file a follow-up issue if they like. Ah.... just spotted something else this can be addressed here.

  2. +++ b/core/modules/node/src/Plugin/migrate/source/d7/Node.php
    index d5a8db09da..9c3a5a4102 100644
    --- a/core/modules/node/tests/src/Kernel/Plugin/migrate/source/d6/NodeByNodeTypeTest.php
    
    --- a/core/modules/node/tests/src/Kernel/Plugin/migrate/source/d6/NodeByNodeTypeTest.php
    +++ b/core/modules/node/tests/src/Kernel/Plugin/migrate/source/d6/NodeByNodeTypeTest.php
    

    Only adding test coverage to d6 is probably not enough - what happens if we move d6 out of core?

Version: 8.7.x-dev » 8.8.x-dev

Drupal 8.7.0-alpha1 will be released the week of March 11, 2019, which means new developments and disruptive changes should now be targeted against the 8.8.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.0-alpha1 will be released the week of October 14th, 2019, which means new developments and disruptive changes should now be targeted against the 8.9.x-dev branch. (Any changes to 8.9.x will also be committed to 9.0.x in preparation for Drupal 9’s release, but some changes like significant feature additions will be deferred to 9.1.x.). For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 8.9.x-dev » 9.1.x-dev

Drupal 8.9.0-beta1 was released on March 20, 2020. 8.9.x is the final, long-term support (LTS) minor release of Drupal 8, which means new developments and disruptive changes should now be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

Matroskeen made their first commit to this issue’s fork.

Matroskeen’s picture

Title: Support multiple node types in node source » Support multiple node types in node source plugins
Issue summary: View changes
Status: Needs work » Needs review
FileSize
7.88 KB
9.74 KB

I recently discovered this limitation and will be happy to move this issue forward.

Here is the merge request with 3 commits:
1) Contains test only from #5;
2) Contains a similar test for D7 plugin per suggestion in #8;
3) Contains the actual fix for all node source plugins including NodeEntityTranslation.php.

I didn't find a way to run tests for specific commit in GitLab, so I'm also attaching patches (tests only + tests with fix). The first one is expected to fail. The second one is equal to the merge request.

I also updated the issue summary according to my report in duplicated issue: #3184835: Allow multiple bundles in node source plugins configuration.

The last submitted patch, 15: 2937989-15-test_only.patch, failed testing. View results

quietone’s picture

Status: Needs review » Needs work
Issue tags: +Needs tests

@Matroskeen, thanks. I may have noticed that MRs are tested. Have you seen, https://www.drupal.org/docs/develop/git/using-git-to-contribute-to-drupa...

I didn't review the tests yet but I did notice that NodeEntityTranslationTest needs to be updated to includes different configurations. Setting to NW and adding tag for this.

Matroskeen’s picture

Status: Needs work » Needs review
Issue tags: -Needs tests

Added one more dataset to NodeEntityTranslationTest with multiple content type configuration.
Moving back to NR.

quietone’s picture

Status: Needs review » Needs work

Just some minor tidying up to do.

Matroskeen’s picture

Status: Needs work » Needs review
quietone’s picture

Status: Needs review » Needs work

@Matroskeen, thanks.

Matroskeen’s picture

Status: Needs work » Needs review

Thanks for the review. MR was updated.

quietone’s picture

Issue summary: View changes
Status: Needs review » Reviewed & tested by the community

@Matroskeen, thanks for the quick reply.

I applied the latest patch and read the NodeEntityTranslationTest. The extra addition provides proof that the configuration value, node_type, works as expected. Nice!

All the points from the review have been addressed and there are no coding standard issues.

So RTBC!

The last submitted patch, 15: 2937989-15-test_only.patch, failed testing. View results

The last submitted patch, 15: 2937989-15-test_only.patch, failed testing. View results

quietone’s picture

To stop messages about the patch in #15 failing I have made a patch from the changes in Merge request !77 and uploaded it here.

alexpott’s picture

Status: Reviewed & tested by the community » Needs work

I think we should document this. Not sure how we document migrate source configuration options.

quietone’s picture

Status: Needs work » Needs review
FileSize
2.86 KB
13.07 KB

Unfortunately, the source plugin configurations not documented. I have added it here using the style of the migrate process plugins in the migrate module as a guide.

Matroskeen’s picture

@quitone, thanks!

I applied your patch to open MR and added one more commit with more examples: https://git.drupalcode.org/project/drupal/-/merge_requests/77/diffs?comm...

alexpott’s picture

Status: Needs review » Reviewed & tested by the community

Thanks for adding the docs - this looks nice. Given it was rtbc in #23 and only docs have been added I'm re-instating that rtbc.

alexpott’s picture

Status: Reviewed & tested by the community » Fixed

Committed 4a30675 and pushed to 9.2.x. Thanks!

  • alexpott committed 4a30675 on 9.2.x
    Issue #2937989 by Matroskeen, quietone, jcisio, alexpott, heddn: Support...
Matroskeen’s picture

Pushed one more commit to fix the typo 🙄

Matroskeen’s picture

Title: Support multiple node types in node source plugins » [backport] Support multiple node types in node source plugins
Status: Fixed » Reviewed & tested by the community
FileSize
1.6 KB

@alexpott, thanks for committing the patch to 9.2.x. Can we also fix the typo mention in this note? (please apply the patch 2937989-9.2.x-typo.txt)

Can we also cherry-pick this commit to 9.1.x? Here is a link to the patch if it makes it easier: https://git.drupalcode.org/project/drupal/-/merge_requests/77.patch

alexpott’s picture

Title: [backport] Support multiple node types in node source plugins » Support multiple node types in node source plugins
Status: Reviewed & tested by the community » Fixed

@Matroskeen new features do not get backported to 9.1.x. And please open a new issue to fix the typo. Re-opening fixed issues for typos makes issue / commit history more difficult than necessary.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

Ghost of Drupal Past’s picture