Provide a node type plugin that permits the developer to specify which node types should be imported. The UserRoleFilter plugin at https://www.drupal.org/project/migrate_plus/issues/3057685 is a great example of the kind of thing desired.

Amended description to better match the work done:
Create a standardized method for filtering imported content. Use this method to create filtered source plugins.

CommentFileSizeAuthor
#9 migrate_plus-filtered_node_plugin-3067902-9.patch3.7 KBgbirch
#2 migrate_plus-filter-nodetypes-3067902-2.patch1.8 KBgbirch

Comments

gbirch created an issue. See original summary.

gbirch’s picture

gbirch commented
StatusFileSize
new migrate_plus-filter-nodetypes-3067902-2.patch1.8 KB

Patch.

gbirch’s picture

gbirch commented
Issue summary: View changes
gbirch’s picture

gbirch commented
Status: Active » Needs review
benjifisher’s picture

benjifisher
he/him or they/them
Primary language English
Location Boston area
commented

I think this option is a very good idea. My only suggestion is that we think more generically.

For example, my current project requires me to migrate nodes with a minimum creation date. (Do not migrate really old content.) Maybe this is an edge case that should be implemented with custom code. For the sake of discussion, assume not: then this is an option we might want to add in the future. If not this option, then I am sure there will be other restrictions that are common enough that we will want to add options for them.

I do not want to add a separate source plugin for every such feature request. Two reasons:

  1. That would not be very DRY.
  2. I might want to restrict node types AND creation date, but I can only use one source plugin per migration.

Adding an option for creation date is out of scope for this issue, but thinking ahead is in scope. If we simply change the name, then we can add further options in future issues. Instead of class NodeTypeFilter and id = "d7_node_type_filter", let's make it class NodePlus and id = "d7_node_plus", the "Plus" indicating that this is the version from the Migrate Plus module.

heddn’s picture

heddn
Primary language English
Location Nicaragua
commented

#3069776: SQL source plugins: allow defining conditions and join in migration yml seems like a good general solution to all of this.

gbirch’s picture

gbirch commented

Agree with both comments. Has anyone taken a shot at a general purpose filter?

Presumably a simple configuration yaml could look something like this:

conditions:
  - column: [any property of a node]
    value: [value]
    operator: [=, >=, IN, NOT IN, etc.]
  - column: 
    value:
    operator:
...

This could only handle a simple list of ANDed conditions, but it would be a decent start.

gbirch’s picture

gbirch commented

Or, say something more flexible like this:

default_conjunction: AND
conditions:
  - conjunction: OR
    - 
      - column: uid
        value: [2, 3]
        operator: IN
      - column: changed
        value: timestamp
        operator: >
  - conjunction: AND
    -
      - column: created
        value: timestamp1
        operator: >
      - column: created
        value: timestamp2
        operator: <=

Which would translate to: (uid IN (2,3) OR changed > timestamp) AND (created > timestamp1 AND created <= timestamp2)

gbirch’s picture

gbirch commented
StatusFileSize
new migrate_plus-filtered_node_plugin-3067902-9.patch3.7 KB

I have not provided an interdiff, as this is a new approach. The trait could also be used to create FilteredX plugins.

gbirch’s picture

gbirch commented

Sorry, I realize that I provided a FilteredNode plugin, rather than a FilteredNodeType plugin. If this approach seems viable, I'll open a new issue and upload all of the FilteredX plugins I've created, which are working well for me at present.

chi’s picture

chi commented

Source Plugin to only import specified node types

d7_node plugin can process specific node type.

source:
  plugin: d7_node
  node_type: article

Though it is not possible to process multiple node types.

source:
  plugin: d7_filtered_node
  filter:
   conjunction: AND
   conditions:
     - column: n.created
       value: 1546300800
       operator: >=
     - conjunction: OR
       conditions:
         - column: n.type
           value: blog
         - column: node_uid
           value: [3,4]

I think writing complex SQL queries like this in yaml files will not improve developer experience. In this case I would rather consider creating a custom source plugin that is suited to the needs of the project.

gbirch’s picture

gbirch commented

@Chi, thanks for looking at it. I admit that I have difficulty imagining that anyone would rather write a custom plugin with a custom query if they could just have a quick YAML file, but you may be right. That being said, it's not like this would prevent them from doing so.

gbirch’s picture

gbirch commented
Issue summary: View changes
chi’s picture

chi commented 
conditions:
  - column: n.type

Another issue here is that the annotation relies on table alias n which is kind of internal thing. So that the alias now must be considered as part of public API and should be never changed.

heddn’s picture

heddn
Primary language English
Location Nicaragua
commented
Status: Needs review » Closed (duplicate)

The great gitlab migration is upon us! This is now a duplicate of https://gitlab.com/drupalspoons/migrate_plus/-/issues/150. Please provide any comments from here, over there.