Comments

DamienMcKenna created an issue. See original summary.

damienmckenna’s picture

Version: 8.x-5.x-dev » 8.x-4.x-dev
Status: Active » Needs review
StatusFileSize
new2.06 KB

This adds a new source plugin called d7_user_role_filter which looks for a "roles" value in the yml file. Here's the docblock:

/**
 * Drupal 7 user source from database that filters based upon role(s).
 *
 * Requires the "roles" annotation value; see examples below.
 *
 * @MigrateSource(
 *   id = "d7_user_role_filter",
 *   source_module = "user"
 * )
 *
 * Examples:
 *
 * Example usage that requires users to have role rid values 3, 5.
 * @code
 * source:
 *   plugin: d7_user_role_filter
 *   roles:
 *     - 3
 *     - 5
 * @endcode
 */
damienmckenna’s picture

Thanks to the folks in the #migration channel on Slack for their help, including ttamniwdoog, benjifisher, timwood, heddn and mikelutz.

marvil07’s picture

Status: Needs review » Needs work
  • The plugin looks good and sounds really useful.
  • This seems to be possible to do at run time, with skip_on_value based on the added rid source field. Naturally an implementation like this will have better performance because it will not need to go over all users, so it is a better option when the amount of d7 users is not small.
  • I see that core d7_user is also handling d7 profile contrib module, so an option to filter on roles sounds also as a good canditate for core itself. We may want to move it over there.
  • Filtering on role ids is great, but for self-documentation purposes on the migration yml's, I would like to suggest to also add an option to use a role name option, i.e. d7 role.name table field.
  • The assignation on the constructor may be a cast to array, but the conditional is also OK, so mentioning it just as suggestion.
  • Also a negative option can be helpful, to be able to migrate all users except some roles.
  • For the query() method calling the parent seems a bit more natural.
  • required_roles data member probably should be requiredRoles

Marking as NW for at least the last two items.

damienmckenna’s picture

StatusFileSize
new3.42 KB

While this doesn't work on the changes requested, ran out of time today, this adds *another* plugin that also adds all users who have created nodes, though I need to test & tweak it.

damienmckenna’s picture

Status: Needs work » Needs review
StatusFileSize
new905 bytes
new2.23 KB

Fixed the UserAuthorRoleFilter and changed UserRoleFilter::query() so it extends the base query instead of duplicating the code.

damienmckenna’s picture

StatusFileSize
new3.72 KB
new2.41 KB

Gah, mucked up the last patches, sorry.

damienmckenna’s picture

Going over the feedback from #4:

  • From what I could see, "skip_on_value" would only work on the source data, and the d7_user migration plugin doesn't pull in the roles data (or the node_revisions data) so it wasn't going to work.
  • I agree that it would be better to filer based upon the role name, I just didn't do that extra step yet.
  • An option to negate the filter would also be good.
damienmckenna’s picture

StatusFileSize
new3.69 KB
new1.87 KB

This renames the class variable, changes the variable check to a cast statement, removes a commented out print_r() statement, and leaves two @todo items for later.

damienmckenna’s picture

Status: Needs review » Needs work

The UNION statement isn't working as intended because SqlBase::initializeIterator() is joining a extra table to the first query, but not the second one, so it throws this error:

Cardinality violation: 1222 The used SELECT statements have a different number of columns
SELECT DISTINCT u.*
FROM {users} u
INNER JOIN {node_revision} nr ON u.uid = nr.uid
WHERE (u.uid > :db_condition_placeholder_0)

and turning it into this:

SELECT DISTINCT u.*, map.sourceid1 AS migrate_map_sourceid1,
map.source_row_status AS migrate_map_source_row_status
FROM {users} u
INNER JOIN {node_revision} nr ON u.uid = nr.uid
LEFT OUTER JOIN SOURCEDB.migrate_map_PLUGINNAME map ON u.uid = map.sourceid1
WHERE (u.uid > :db_condition_placeholder_0) AND ((map.sourceid1 IS NULL) OR (map.source_row_status = :db_condition_placeholder_1))
damienmckenna’s picture

Status: Needs work » Needs review
StatusFileSize
new2.08 KB

I split this up into two separate requests, so this one is back to just the user role filter.

damienmckenna’s picture

This needed a $query->distinct(TRUE) to make sure the query count was correct.

heddn’s picture

Version: 8.x-4.x-dev » 8.x-5.x-dev
patpluspun’s picture

Status: Needs review » Reviewed & tested by the community

Just tested it with a single role and multiple roles, works like a charm. Thanks Damien!

damienmckenna’s picture

Status: Reviewed & tested by the community » Needs work

Glad it worked for you, @patpluspun.

Let's go back to the request to allow limiting it by role name.

patpluspun’s picture

StatusFileSize
new2.21 KB

Patch #16 addresses changes in drupal_migrate that we encountered. Very minor change. I'll try and tackle limiting by role name in the near future.

lisotton’s picture

StatusFileSize
new2.82 KB
new2.21 KB

I have included the possibility to filter roles by name.

lisotton’s picture

Status: Needs work » Needs review
matroskeen’s picture

This is definitely a common requirement for a custom migration. Thanks for the all hard work done here 👍

However, you might be interested in #3069776: SQL source plugins: allow defining conditions and join in migration yml, which allows supplying additional joins and conditions to the existing source plugins. For this particular case, the source plugin can be configured in the following way:

source:
  plugin: d7_user
  joins:
    - table: users_roles
       alias: ur
       condition: u.uid = ur.uid
  conditions:
    - field: ur.rid
       value: [1, 2, 3]
       operator: IN
  distinct: TRUE

It'll make the source plugin retrieve users that have one of these roles: 1, 2, 3.

I think we can close this one in favor of #3069776: SQL source plugins: allow defining conditions and join in migration yml.