Problem/Motivation

Currently, the 'get' process plugin is the only place that contains the logic needed to process row keys by unescaping '@' and determining if the key represents a source or destination property. As such, any process plugin that might take in a row key as a configuration value must instantiate a copy of the get plugin to determine the value.

Proposed resolution

Currently Row has a getSourceProperty method and a getDestinationProperty method. Lets add a generic get method which performs the same logic as the plugin, and implement the magic __get method as well to access a property and have the Row object determine if it is a source or destination property and return the correct one.

Remaining tasks

Add a get, getMultiple, and __get method to the Row Class
Write Tests
Update MigrationLookup to use this instead of Get

User interface changes

None

API changes

Add \Drupal\migrate\Row::get($property)
Add \Drupal\migrate\Row::getMultiple($properties)
Add \Drupal\migrate\Row::__get($property)
Remove plugin.manager.migrate.process dependency from MigrationLookup

Data model changes

None

CommentFileSizeAuthor
#26 interdiff.2999906.20-26.txt5.35 KBmikelutz
#26 2999906-26.drupal.Move-logic-from-the-get-plugin-into-the-Row-class.patch24.67 KBmikelutz
#20 Interdiff.2999906.18-20.txt3.44 KBmikelutz
#20 2999906-20.drupal.Move-logic-from-the-get-plugin-into-the-Row-class.patch23.58 KBmikelutz
#18 2999906-18.drupal.Move-logic-from-the-get-plugin-into-the-Row-class.patch24.53 KBmikelutz
#18 interdiff.2999906.16-18.txt666 bytesmikelutz
#16 interdiff.2999906.12-16.txt5.69 KBmikelutz
#16 2999906-16.drupal.Move-logic-from-the-get-plugin-into-the-Row-class.patch24.53 KBmikelutz
#12 interdiff.2999906.9-12.txt11.41 KBmikelutz
#12 2999906-12.drupal.Move-logic-from-the-get-plugin-into-the-Row-class.patch26.06 KBmikelutz
#9 interdiff.2999906.7-9.txt961 bytesmikelutz
#9 2999906-9.drupal.Move-logic-from-the-get-plugin-into-the-Row-class.patch16.1 KBmikelutz
#7 interdiff.2999906.4-7.txt13.98 KBmikelutz
#7 2999906-7.drupal.Move-logic-from-the-get-plugin-into-the-Row-class.patch15.83 KBmikelutz
#4 interdiff.2999906.3-4.txt1.72 KBmikelutz
#4 2999906-4.drupal.Move-logic-from-the-get-plugin-into-the-Row-class.patch27.49 KBmikelutz
#3 interdiff.2999906.2-3.txt12.56 KBmikelutz
#3 2999906-3.drupal.Move-logic-from-the-get-plugin-into-the-Row-class.patch27.41 KBmikelutz
#2 2999906-2.drupal.Move-logic-from-the-get-plugin-into-the-Row-class.patch20.38 KBmikelutz
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

mikelutz created an issue. See original summary.

mikelutz’s picture

mikelutz’s picture

This adds support for retrieving multiple values at once from the row.

mikelutz’s picture

mikelutz’s picture

Issue summary: View changes
heddn’s picture

Discussed briefly in the migrate maintainers meeting w/ @mikelutz @maxocub @heddn and discussed removing the row interface. Also discussed the possibility of later, down the road removing entirely the Get plugin. For now leaving it.

mikelutz’s picture

phenaproxima’s picture

Big +1 for this patch. I find nothing actually wrong with it; just minor bits and bobs.

  1. +++ b/core/modules/migrate/src/Row.php
    @@ -109,6 +109,20 @@ public function __construct(array $values = [], array $source_ids = [], $is_stub
    +   * @return bool|mixed|null
    +   *   The source or destination property requested.
    

    This can just be mixed|null, and the description should say that NULL is returned if the property doesn't exist.

  2. +++ b/core/modules/migrate/src/Row.php
    @@ -109,6 +109,20 @@ public function __construct(array $values = [], array $source_ids = [], $is_stub
    +    return reset($properties);
    

    reset() returns FALSE if the array is empty. So we should probably translate that to NULL, like so: return $properties ? reset($properties) : NULL.

  3. +++ b/core/modules/migrate/src/Row.php
    @@ -246,6 +260,61 @@ public function setEmptyDestinationProperty($property) {
    +    if (is_array($properties)) {
    +      return $this->getMultiple($properties);
    +    }
    +    $values = $this->getMultiple([$properties]);
    

    If you cast a string to array, you get [$string]. So this can just be $values = $this->getMultiple((array) $properties).

mikelutz’s picture

#8.1 - I updated the documentation.

#8.2 - getMultiple wouldn't ever return an empty array unless it were called with one, (which it can't be from the magic method). If it or ::get() were to be called with an empty array, a return value of an empty array would be appropriate. Either way, the lookup might result in an actual boolean FALSE value, so we couldn't blanket convert boolean false to NULL anyway. Edit - Seems I didn't read your code fully, your solution would not have converted a boolean false value to null. I still think it's unnecessary though, since getMultiple will always return an array of the same length as the one it received.

#8.3 - We use the if(!is_array($foo)) $foo=[$foo] pattern several places (see Get, StaticMap, and MigrationLookup). I'm not sure typecasting is simpler, (array) $foo vs array($foo) vs [$foo], but I left it as [$foo] for consistency with the other plugins for now. I will change it if there is an overwhelming desire to.

mikelutz’s picture

Status: Needs review » Needs work

Bah, I was perfectly happy with this patch until phenaproxima made me look at it again. I just realized that moving the get plugin instantiation out of migration lookup means the migrate plugin manager is injected and never used, so I need to address that when I get to work.

mikelutz’s picture

Issue tags: +Needs change record

Going to need crs too, I suppose.

mikelutz’s picture

I expect this will need additional test updates...

EDIT - I was wrong, I got them all, woohoo!

mikelutz’s picture

Issue summary: View changes
Status: Needs work » Needs review
Issue tags: -Needs change record
phenaproxima’s picture

Status: Needs review » Needs work
Issue tags: +DX (Developer Experience)

Only a few small things, and then this is ready to go. I really love the test coverage -- seems quite thorough!

  1. +++ b/core/modules/migrate/src/Plugin/migrate/process/MigrationLookup.php
    @@ -123,13 +116,31 @@ class MigrationLookup extends ProcessPluginBase implements ContainerFactoryPlugi
    +   * @param \Drupal\migrate\Plugin\MigratePluginManagerInterface|null $process_plugin_manager
    +   *   (deprecated) The $process_plugin_manager parameter is deprecated since
    +   *   version 8.7.x and will be removed in 9.0.0.
    

    Constructors (and plugin implementations in general) are not considered part of the API, so I think we can actually remove this parameter entirely.

  2. +++ b/core/modules/migrate/src/Row.php
    @@ -109,6 +109,25 @@ public function __construct(array $values = [], array $source_ids = [], $is_stub
    +   * Retrieve a source or destination property.
    +   * If the property key begins with '@' return a destination property,
    +   * otherwise return a source property. the '@' symbol itself can be escaped
    +   * as '@@'. Returns NULL if property is not found.
    

    We can remove the final sentence here, as it is repeated in the @return documentation. Also -- super nitpick time! -- can these lines be nicely formatted and wrapped to 80 characters?

  3. +++ b/core/modules/migrate/src/Row.php
    @@ -109,6 +109,25 @@ public function __construct(array $values = [], array $source_ids = [], $is_stub
    +    $properties = $this->getMultiple([$property]);
    +    return reset($properties);
    

    Nitpick, but is there any reason not to have __get() essentially be an alias of get()? So the whole body of __get() could be return $this->get($property).

  4. +++ b/core/modules/migrate/src/Row.php
    @@ -246,6 +265,61 @@ public function setEmptyDestinationProperty($property) {
    +   * @param array|string $properties
    

    Should be @param string|string[], since AFAIK this cannot be an array of anything but strings.

  5. +++ b/core/modules/migrate/src/Row.php
    @@ -246,6 +265,61 @@ public function setEmptyDestinationProperty($property) {
    +   * @param array $properties
    

    Can this be string[]?

  6. +++ b/core/modules/migrate/src/Row.php
    @@ -246,6 +265,61 @@ public function setEmptyDestinationProperty($property) {
    +      if ($property[0] == '@') {
    

    I'd say we should use ===, but I'm not sure if that, unlikely as it is, will break existing code somewhere. I guess we'll leave it for now.

  7. +++ b/core/modules/migrate/tests/src/Unit/RowTest.php
    @@ -271,4 +308,148 @@ public function testMultipleDestination() {
    +   * @param bool $is_stub
    +   *   Whether this row is a stub row.
    

    This should mention that $is_stub defaults to FALSE.

mikelutz’s picture

The only reason for 14.3 is that ::get() will take an array or a string, while ::__get() will only ever be called with a string, so it didn't need the extra is_array check.

for 14.6, the logic was directly copied over from the plugin, I didn't want to risk changing the behavior as part of this exercise.

mikelutz’s picture

There you go, addressed everything, but left 14.6 alone. We can add it as a novice issue if we want.

phenaproxima’s picture

Status: Needs review » Reviewed & tested by the community
+++ b/core/modules/migrate/src/Row.php
@@ -112,10 +112,9 @@ public function __construct(array $values = [], array $source_ids = [], $is_stub
+   * '@' return a destination property, otherwise return a source property. the

The tiniest thing -- "the" needs to be capitalized after the period.

No need to go back to "needs review"; this is RTBC from me once green on all backends. Awesome!

alexpott’s picture

Status: Reviewed & tested by the community » Needs review
  1. +++ b/core/modules/migrate/src/Row.php
    @@ -109,6 +109,23 @@ public function __construct(array $values = [], array $source_ids = [], $is_stub
    +  public function __get($property) {
    

    Why do do we need this magic?

  2. +++ b/core/modules/migrate/src/Row.php
    @@ -246,6 +263,61 @@ public function setEmptyDestinationProperty($property) {
    +   * @param string|string[] $properties
    +   *   The property to get, or an array of properties to get.
    ...
    +  public function get($properties) {
    ...
    +   * @param string[] $properties
    ...
    +  public function getMultiple(array $properties) {
    

    Why do we need a get that supports an array when we have getMultiple()?

mikelutz’s picture

mikelutz’s picture

Issue summary: View changes
heddn’s picture

Status: Needs review » Reviewed & tested by the community

Feedback addressed. Back to RTBC.

Status: Reviewed & tested by the community » Needs work

The last submitted patch, 20: 2999906-20.drupal.Move-logic-from-the-get-plugin-into-the-Row-class.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

mikelutz’s picture

Status: Needs work » Reviewed & tested by the community

Testbot hiccup..

alexpott’s picture

Status: Reviewed & tested by the community » Needs work

One thought I have is after this patch what is the point in getSourceProperty() / getDestinationProperty()? It would be handy if the document described when you should use the new methods over the existing ones.

Also the place where the methods have been added to the Row class seems inconsistent. I would have thought the two new methods would come before or after any getSource* or getDestination*

+++ b/core/modules/migrate/tests/src/Unit/process/MigrationLookupTest.php
@@ -13,6 +14,7 @@
+use Drupal\Tests\Traits\ExpectDeprecationTrait;

Not used.

mikelutz’s picture

I've added some additional documentation as to when to use the methods, and I can add it to the CR as well. The new method should typically be used in process plugins to pull a property defined in a migration. These are usually source properties, but sometimes you want to process a source property and save it in the destination and use it. The use case in this patch is in MigrationLookup, where you specify the property to use as a lookup when looking up multiple migraitons. This is generally a source id, but you might want to run it through a process first like a map or default value and use it as a destination property. This was previously done by creating a copy of the get plugin and using that to retrieve the value. The patch makes that unnecessary, however destination and source plugins that need to access properties generally know whether they want a source or destination property, and they don't want to have to worry about adding a '@' to a destination property name, or escaping '@' symbols in general, so they would still use the respective retrieval methods. I think they still have use and should stay part of the API for that reason.

process:
  mapped_id:
    plugin: static_map
    source: source_id
    map:
      id1: id2
  id:
    plugin: migration_lookup
    migrations: 
      -  migration_1
      -  migration_2
    source_ids:
      migration_1:
        -  source_id
      migration_2:
        - @mapped_id
alexpott’s picture

So I guess that leads me to wondering how useful will this really be for contrib and if the flip case is true - in that it might cause some contrib / custom plugins to use the ->get() method when they should be using one of the others?

At least the new document makes it clearer about when to use the different things.

mikelutz’s picture

This all started with working on some refactoring around the migration lookup plugin, which has lots of issues, and high churn. There are a few places in core, and more in contrib where copies of the migration_lookup plugin is created inside of other process plugins.

I started to think that if the lookup/stub creation logic is so useful in other processes, it should probably be available as a service, or somewhere else in the api so that you aren't creating a plugin inside a plugin. That made me realize that migration_lookup was doing that with get and that I've done that with get as well in custom processes, and it just seemed logical to move the logic of that directly to the row class.

This will also support some future refactoring around migration lookup and migration executable to avoid creating get plugins when they aren't really needed.

heddn’s picture

Status: Needs review » Reviewed & tested by the community

re #27: after reviewing the patch, the CR and the docs on the new methods, it does make sense to add this to Row. Because we use the 'get' logic all over the place in custom and contrib process plugins. And to make it generic, we need a way to get those values from either source or destination. We already had that code duplication, just it was stuffed down in Get process plugin. And in retrospect, we probably don't need the getSource or getDestination methods as public. We might even want to eventually deprecate them or see if we can move them to protected. Where the data is stored is an implementation details irrelevant to callers.

Plenty of test coverage. CR is good. And since this is to help us improve migration_lookup, let's get this in.

quietone’s picture

+1 This is really nice work and the two CRs are spot on.

I have one personal reservation and that is the potential loss of getSourceProperty and getDestinationProperty which, I think, are easier to understand than get('name') or get('@name'). Just have to wait and see on that one.

alexpott’s picture

Status: Reviewed & tested by the community » Fixed

Committed a9f1973 and pushed to 8.7.x. Thanks!

diff --git a/core/modules/migrate/src/Plugin/migrate/process/MigrationLookup.php b/core/modules/migrate/src/Plugin/migrate/process/MigrationLookup.php
index 0643b93f65..aa7483a95a 100644
--- a/core/modules/migrate/src/Plugin/migrate/process/MigrationLookup.php
+++ b/core/modules/migrate/src/Plugin/migrate/process/MigrationLookup.php
@@ -4,7 +4,6 @@
 
 use Drupal\Core\Plugin\ContainerFactoryPluginInterface;
 use Drupal\migrate\MigrateSkipProcessException;
-use Drupal\migrate\Plugin\MigratePluginManagerInterface;
 use Drupal\migrate\Plugin\MigrationPluginManagerInterface;
 use Drupal\migrate\Plugin\MigrateIdMapInterface;
 use Drupal\migrate\ProcessPluginBase;

Fixed unused use on commit.

  • alexpott committed a9f1973 on 8.7.x
    Issue #2999906 by mikelutz, heddn, phenaproxima, alexpott: Move logic...

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.