Problem/Motivation

While sourcepluginbase loads the source plugin configuration into the row to facilitate constants, SubProcess doesn't do this. We need access to all the statics that are setup in the source while inside of the SubProcess plugin.

Proposed resolution

$defaults = $this->configuration;
unset($defaults['plugin'], $defaults['key'], $defaults['process']), 
  $new_row = new Row($new_value + $defaults, array());

is my best guess.

Remaining tasks

Determine what breaks. Code, test, debate, review, commit, dance.

User interface changes

API changes

Data model changes

Comments

chx created an issue. See original summary.

chx’s picture

Issue summary: View changes
chx’s picture

Status: Active » Needs review
StatusFileSize
new973 bytes
benjy’s picture

Issue tags: +Needs tests

We might even have unit tests for this already

chx’s picture

StatusFileSize
new2.41 KB

So. much. boilerplate.

Status: Needs review » Needs work

The last submitted patch, 5: 2684689_5.patch, failed testing.

chx’s picture

We picked a custom plugin instead at the end so I will not be finishing this core patch. I am very strictly only working on client required patches.

Version: 8.1.x-dev » 8.2.x-dev

Drupal 8.1.9 was released on September 7 and is the final bugfix release for the Drupal 8.1.x series. Drupal 8.1.x will not receive any further development aside from security fixes. Drupal 8.2.0-rc1 is now available and sites should prepare to upgrade to 8.2.0.

Bug reports should be targeted against the 8.2.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.3.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

imiksu’s picture

StatusFileSize
new2.41 KB

Testing if needs reroll and had some offset. Updating patch.

Your branch is up-to-date with 'origin/8.2.x'.
nothing to commit, working directory clean
$ curl https://www.drupal.org/files/issues/2684689_5.patch | patch -p1
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2466  100  2466    0     0   2109      0  0:00:01  0:00:01 --:--:--  2111
patching file core/modules/migrate/src/Plugin/migrate/process/Iterator.php
Hunk #1 succeeded at 2 (offset -5 lines).
Hunk #2 succeeded at 19 (offset -5 lines).

Version: 8.2.x-dev » 8.3.x-dev

Drupal 8.2.6 was released on February 1, 2017 and is the final full bugfix release for the Drupal 8.2.x series. Drupal 8.2.x will not receive any further development aside from critical and security fixes. Sites should prepare to update to 8.3.0 on April 5, 2017. (Drupal 8.3.0-alpha1 is available for testing.)

Bug reports should be targeted against the 8.3.x-dev branch from now on, and new development or disruptive changes should be targeted against the 8.4.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

heddn’s picture

Status: Needs work » Postponed
nevergone’s picture

Version: 8.3.x-dev » 8.4.x-dev
Status: Postponed » Needs review
StatusFileSize
new2.44 KB
new2.45 KB

Patch re-rolled, please review. 8.4.x and 8.3.x attached.

gábor hojtsy’s picture

Status: Needs review » Needs work

Definitely needs tests still. Also for someone who is not well versed in migrations:

+++ b/core/modules/migrate/src/Plugin/migrate/process/SubProcess.php
@@ -116,16 +119,48 @@
+    $defaults = $this->configuration + $this->sourceConfiguration;
+    unset($defaults['plugin'], $defaults['key'], $defaults['process']);
     if (is_array($value) || $value instanceof \Traversable) {
       foreach ($value as $key => $new_value) {
-        $new_row = new Row($new_value, []);
+        $new_row = new Row($new_value + $defaults, []);

This passes in both $this->configuration and $this->sourceConfiguration. Neither were passed earlier. What is the role of each and why do we need both?

heddn’s picture

Issue summary: View changes

The reason for passing both config and source is mentioned in the IS. But it could be clearer.

  1. +++ b/core/modules/migrate/src/Plugin/migrate/process/SubProcess.php
    @@ -116,16 +119,48 @@
    +  protected $sourceConfiguration;
    

    Could use better docs on this variable to address the question in #13.

  2. +++ b/core/modules/migrate/src/Plugin/migrate/process/SubProcess.php
    @@ -116,16 +119,48 @@
    +   * Iterator constructor.
    

    This isn't called Iterator any more.

heddn’s picture

Title: Iterator process plugin doesn't support constants » SubProcess process plugin doesn't support constants
jofitz’s picture

Status: Needs work » Needs review
StatusFileSize
new747 bytes
new2.57 KB

Addressing @heddn's comments in #14.

Still needs tests.

heddn’s picture

Status: Needs review » Needs work

NW because needs tests. But #16 is looking good.

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.0-alpha1 will be released the week of July 31, 2017, which means new developments and disruptive changes should now be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

jofitz’s picture

Status: Needs work » Needs review
Issue tags: -Needs tests
StatusFileSize
new2.86 KB
new5.43 KB

Added tests.
No interdiff because the only change is adding the tests.

The last submitted patch, 19: 2684689-19-test_only.patch, failed testing. View results

Version: 8.5.x-dev » 8.6.x-dev

Drupal 8.5.0-alpha1 will be released the week of January 17, 2018, which means new developments and disruptive changes should now be targeted against the 8.6.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

maxocub’s picture

Status: Needs review » Needs work
  1. +++ b/core/modules/migrate/src/Plugin/migrate/process/SubProcess.php
    @@ -116,16 +119,51 @@
    +   * The source plugin configuration, facilitating access to all the statics
    +   * that are set up in the source.
    

    Coding standard asks that the docblock summary be on only one line.

  2. +++ b/core/modules/migrate/src/Plugin/migrate/process/SubProcess.php
    @@ -116,16 +119,51 @@
    +   * @param array $configuration
    +   * @param string $plugin_id
    +   * @param mixed $plugin_definition
    +   * @param array $source_configuration
    

    Params are missing a description.

  3. +++ b/core/modules/migrate/src/Plugin/migrate/process/SubProcess.php
    @@ -116,16 +119,51 @@
    +    $defaults = $this->configuration + $this->sourceConfiguration;
    +    unset($defaults['plugin'], $defaults['key'], $defaults['process']);
         if (is_array($value) || $value instanceof \Traversable) {
           foreach ($value as $key => $new_value) {
    -        $new_row = new Row($new_value, []);
    +        $new_row = new Row($new_value + $defaults, []);
    

    A comment explaining why we are doing this would help here.

mohit1604’s picture

Issue tags: +Needs reroll

Patch #19 no longer apply.

mohit1604’s picture

StatusFileSize
new5.44 KB
mohit1604’s picture

Status: Needs work » Needs review
Issue tags: -Needs reroll
maxocub’s picture

Status: Needs review » Needs work

Thanks for the re-roll @mohit1604, back to NW for #22.

jgrunert’s picture

Status: Needs work » Needs review
StatusFileSize
new5.73 KB

Added comments as per #22

jgrunert’s picture

StatusFileSize
new1.88 KB
heddn’s picture

Status: Needs review » Needs work

Pretty close. Just small stuff.

As a side note typically interdiffs are uploaded with a .txt extension. That wouldn't be readily apparent though, since our docs are out of sync with the norm. That's now fixed now: https://www.drupal.org/documentation/git/interdiff

+++ b/core/modules/migrate/src/Plugin/migrate/process/SubProcess.php
@@ -161,16 +164,56 @@
+   * Source plugin configuration, to allow access to all source static variables

Nit: sentence need a closing stop, also known as a period.

Back to NW for the small nit.

jgrunert’s picture

Status: Needs work » Needs review
StatusFileSize
new5.67 KB
new1.84 KB

Reworked one comment for proper punctuation and line length.

heddn’s picture

Status: Needs review » Needs work
  1. +++ b/core/modules/migrate/src/Plugin/migrate/process/SubProcess.php
    @@ -161,16 +164,56 @@
    +  protected $sourceConfiguration;
    ...
    +   * @param array $source_configuration
    +   *   The configuration values from the source plugin.
    ...
    +    $this->sourceConfiguration = $source_configuration;
    ...
    +    // Merge in the source plugin configuration to provide access to all the
    +    // source plugin statics.
    +    $defaults = $this->configuration + $this->sourceConfiguration;
    
    +++ b/core/modules/migrate/tests/src/Unit/process/SubProcessTest.php
    @@ -31,18 +31,20 @@ class SubProcessTest extends MigrateTestCase {
    +  public function testSubProcess($value, $configuration = [], $source_configuration = []) {
    ...
    +    $plugin = new SubProcess($configuration, 'sub_process', [], $source_configuration);
    
    @@ -59,24 +61,29 @@ public function testSubProcess() {
    +      'with source config' => [[['source_foo' => 'test']], [], ['source_id' => 42]],
    +      'with both configs' => [[[]], ['source_foo' => 'test'], ['source_id' => 42]],
    

    I was about to RTBC, then I noticed something important. We aren't actually passing in the source config. We're passing in source values. This is a subtle but important distinction. This means we have access to statics and all other values in the source. Which is really what we want. But that also means we need to rename variables, tests, etc. to align with reality And the docs should b focused on the source values, not on a specific subset of those values. No need to mention statics at all.

  2. +++ b/core/modules/migrate/src/Plugin/migrate/process/SubProcess.php
    @@ -161,16 +164,56 @@
    +    unset($defaults['plugin'], $defaults['key'], $defaults['process']);
    

    This deserves a comment. Uncommenting plugin and process make some sense to me. But why key? Let's explain why for all, just to make it super clear.

SilviuChingaru’s picture

You can always use something like this to obtain constants behaviour in sub_process plugin:

id: migration_id
label: Migration Name
(...)
process:
  field_dest:
    plugin: sub_process
    source: field_source
    process:
      constants:
        plugin: default_value
        default_value:
          first_constant: first_constant_value
          second_constant: second_constant_value
      concated_field:
        plugin: concat
        delimiter: /
        source:
          - '@constants/first_constant'
          - source_field
(...)
phenaproxima’s picture

Status: Needs work » Needs review
StatusFileSize
new6.52 KB

Slightly alternative approach here. I ran into a situation where I needed the source ID values from the parent row, and @heddn directed me to this issue. I decided to introduce two new configuration options to the SubProcess plugin -- include_source_id_values and include_source_constants. When include_source_constants is set, only the source plugin constants will be added to the sub-processed row, rather than the entire source configuration.

I figured it was better to introduce this behavior behind config flags, in order to maintain BC.

phenaproxima’s picture

This blocks the migration from Panelizer to Layout Builder (see #2969376: [META] Migration path to Layout Builder).

heddn’s picture

Status: Needs review » Needs work
+++ b/core/modules/migrate/src/Plugin/migrate/process/SubProcess.php
@@ -161,16 +170,64 @@
+    if ($this->configuration['include_source_constants']) {
+      $source_configuration = $this->migration->getSourceConfiguration();
+
+      if (isset($source_configuration['constants'])) {
+        $merge['constants'] = $source_configuration['constants'];
+      }
+    }

This won't work. Constants are /typically/ prefixed with the word "constants". But that is only by convention and convenience. Let's not add the complexity of this and just pass everything.

phenaproxima’s picture

Status: Needs work » Needs review
Issue tags: +Needs tests
StatusFileSize
new6.89 KB
new2.31 KB

Discussed with @heddn on Slack and we decided that it would be better to copy all source configuration into the sub-processed row, since not everybody prefixes their source constants with 'constants'. I insisted on prefixing the copied source configuration, though, to avoid unintentional wonkiness. The prefix defaults to 'source_configuration', but can be overridden.

Still needs test coverage.

heddn’s picture

+++ b/core/modules/migrate/src/Plugin/migrate/process/SubProcess.php
@@ -17,6 +20,17 @@
+ *   - include_source_configuration: If TRUE, all source plugin configuration

Nit: let's say include_source_values. And pass all values, instead of just the config. That was the original need for this issue.

OR, have a config and a source values option. But I insist on a source values option too.

phenaproxima’s picture

Status: Needs review » Needs work

That sounds reasonable. Back to the ol' drawing board.

heddn’s picture

Title: SubProcess process plugin doesn't support constants » SubProcess process plugin doesn't support passing context values from source plugin
dylan donkersgoed’s picture

StatusFileSize
new7.53 KB
new1.26 KB

I'm attaching an updated version of the patch that brings in source values as well. I did add them to a separate key.

I've done some manual testing of it but haven't expanded the automated tests, so I'm leaving it at "Needs Work" for now. Honestly not sure quite how to bring in source values to test. Drupal 8's automated testing stuff is still somewhat alien to me, especially when you bring the database into it. I'll try to come back to this and add them later when I have some time to look at it properly.

Version: 8.6.x-dev » 8.7.x-dev

Drupal 8.6.0-alpha1 will be released the week of July 16, 2018, which means new developments and disruptive changes should now be targeted against the 8.7.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

heddn’s picture

Status: Needs work » Needs review
Issue tags: -Needs tests
StatusFileSize
new9.52 KB
new8.84 KB

I've responded to my own points. This fixes the tests and merged things down to a single set of context values called, 'source'. And this context values can be renamed if the 'source' doesn't make sense.

This results in yaml of this form, where the value of baf in the source becomes the values of baaa inside of the sub_process plugin.

        baf: source_baz,
        field_my_field:
          plugin: sub_process
          include_source: true,
          source_key: my_source, // < Optional, defaults to 'source'
          process:
            foo: source_foo,
            baaa: 'my_source/baf',

Status: Needs review » Needs work

The last submitted patch, 42: 2684689-42.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

heddn’s picture

Status: Needs work » Needs review
StatusFileSize
new6.98 KB
new8.89 KB

This is a more sane approach. Plus fixes some tests.

maxocub’s picture

Status: Needs review » Reviewed & tested by the community

I really like this new simpler approach. The patch looks good to me. Let's move this forward.

alexpott’s picture

Status: Reviewed & tested by the community » Needs review
+++ b/core/modules/migrate/src/Plugin/migrate/process/SubProcess.php
@@ -161,16 +171,57 @@
+class SubProcess extends ProcessPluginBase implements ContainerFactoryPluginInterface {
...
+  /**
+   * {@inheritdoc}
+   */
+  public static function create(ContainerInterface $container, array $configuration, $plugin_id, $plugin_definition, MigrationInterface $migration = NULL) {
+    return new static($configuration, $plugin_id, $plugin_definition, $migration);
+  }

How come we need to implement ContainerFactoryPluginInterface? This gives us the $migration property but we never appear to use it?

heddn’s picture

Issue tags: +Novice

Good question. We can probably remove that. Removing the interface and the constructor is definitely a novice programming task. And fixing the tests so they don't inject that variable is probably too. So tagging novice.

heddn’s picture

Status: Needs review » Needs work
bendeguz.csirmaz’s picture

Status: Needs work » Needs review
StatusFileSize
new7.05 KB
new3.72 KB

Removed.

phenaproxima’s picture

Status: Needs review » Reviewed & tested by the community
Issue tags: -Novice

Thanks! Back to RTBC we go.

catch’s picture

Status: Reviewed & tested by the community » Fixed

Committed 971c946 and pushed to 8.7.x. Thanks!

  • catch committed 971c946 on 8.7.x
    Issue #2684689 by jgrunert, Jo Fitzgerald, heddn, phenaproxima, chx,...

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.