The only started using the _extract_ plugin recently and it works great. However, say, you run an array_filter on an array and remove empty values. The value then becomes:

[
  1 => "Foo",
  2 => "Bar",
  5 => "Baz",
]

It gets difficult to get the first value (shift) or the last value (pop). Here are some proposed solutions:

  1. Have a parameter named "method" which can be set to "index", "pop", "shift"
  2. Have separate parameters named "index", "shift", "pop" - the first one found is evaluated
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

jigarius created an issue. See original summary.

Version: 8.4.x-dev » 8.5.x-dev

Drupal 8.4.0-alpha1 will be released the week of July 31, 2017, which means new developments and disruptive changes should now be targeted against the 8.5.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

jigarius’s picture

Version: 8.5.x-dev » 8.4.x-dev
jigarius’s picture

Status: Active » Needs review
FileSize
3.22 KB
jigarius’s picture

Version: 8.4.x-dev » 8.5.x-dev

Oops. Changing back to 8.5.x.

jigarius’s picture

jigarius’s picture

maxocub’s picture

I think we already can do that with the callback plugin:

plugin: callback
source: some_assoc_array
callable: array_pop

or

plugin: callback
source: some_assoc_array
callable: array_shift
heddn’s picture

Status: Needs review » Closed (works as designed)

Good work there, but I think I agree with #8.

jigarius’s picture

Thanks for the feedback! Two things to note here:

  • Unlike the extract plugin, the callback plugin doesn't "handle_multiple" values. So the solution in #8 is going to work on individual values separately and not treat them together as a whole (which is my use case)
  • If the multiple values plugin (https://www.drupal.org/node/2901617) gets into migrate_plus, then #8 would make sense.

Please feel free to re-open if that makes sense.

jigarius’s picture

Status: Closed (works as designed) » Needs review

I just tried to do the thing suggested in #8 and I remembered why I had to create a separate plugin called ArrayShift in my project in the first place. When you try to call array_pop or array_shift using the "callback" plugin, you get this message:

Parameter 1 to array_shift() expected to be a reference, value given Callback.php:54
Parameter 1 to array_pop() expected to be a reference, value given Callback.php:54

Hence, I'm reopening this issue. I think this is quite a common use case because I've stumbled upon such a problem 2 times in less than 3 months. I will see if I can write some tests for this this weekend.

heddn’s picture

Project: Drupal core » Migrate Plus
Version: 8.5.x-dev » 8.x-4.x-dev
Component: migration system » Plugins

Let's make a few new process plugins for push and pop. And these should go into migrate_plus for now, until we find a use for them in core. Or at least incubate.

jigarius’s picture

FileSize
2.47 KB

Created 2 plugins - array_pop and array_shift. Do I need to include tests as well?

heddn’s picture

Status: Needs review » Needs work
Issue tags: +Needs tests

Yup, process plugins should test the edge cases and any exceptions. What happens if the data isn't an array, etc

jigarius’s picture

Status: Needs work » Needs review
FileSize
5.93 KB

Done. Added tests for both the shift and pop plugins.

jigarius’s picture

BTW, thinking aloud, I just thought of something different. Is there a reason why the callable plugin doesn't call the callable with arguments by reference? What if we had something like this in the callable plugin?

public function transform($value, MigrateExecutableInterface $migrate_executable, Row $row, $destination_property) {
    if (is_callable($this->configuration['callable'])) {
      $callable = is_array($this->configuration['callable']) ? implode('::', $this->configuration['callable']) : $this->configuration['callable'];
      // Call by reference.
      if (isset($this->configuration['reference'])) {
        $callable($value);
      }
      // Call by value.
      else {
        $value = $callable($value);
      }
    }
    return $value;
  }

That way, people will be able to call a plethora of callbacks without having to write new plugins. Example: array_pop, array_shift, sort and all other functions which expect arguments by reference instead of value. Any thoughts? Also, I think the callback process plugin should throw an exception when the callback doesn't exist or is not callable instead of silently doing nothing. Let me know if this sounds good and I'll create a patch, with tests, etc to get the wheel rolling. I'm feeling highly motivated this week (:

heddn’s picture

Status: Needs review » Needs work
Issue tags: -Needs tests

re #16: We don't want callable to become essentially the php module. If it isn't super simple, then write a process plugin and create some tests. It isn't hard.

Open a core issue for the exception throwing. That seems like a reasonable suggestion.

Now for a review of #15

  1. +++ b/src/Plugin/migrate/process/ArrayPop.php
    @@ -0,0 +1,42 @@
    + *     source: array
    

    I think an example where we provide some sample values of what is in 'array' would help. Plus, let's use a more better name than array. Maybe fruit or favorite movies can inspire you with a better name? Also applies to array_shift.

  2. +++ b/src/Plugin/migrate/process/ArrayShift.php
    @@ -0,0 +1,42 @@
    + * performing a "pop" operation.
    ...
    + * To perform an array pop on the input array:
    ...
    + *     plugin: array_shift
    

    The docs are wrong here.

  3. +++ b/tests/src/Unit/process/ArrayPopTest.php
    @@ -0,0 +1,62 @@
    +  public function testArrayPopFromIndexedArray() {
    ...
    +  public function testArrayPopFromAssocArray() {
    
    +++ b/tests/src/Unit/process/ArrayShiftTest.php
    @@ -0,0 +1,62 @@
    +  public function testArrayShiftFromIndexedArray() {
    ...
    +  public function testArrayShiftFromAssocArray() {
    

    Can we use a dataprovider for this? And I'd like to see us to test the edge case where the array is empty. This can all be done with a data provider pretty easily.

jigarius’s picture

Status: Needs work » Needs review
FileSize
6.64 KB

I switched to a data provider except for the error test case where an exception in expected. I also fixed the documentation (which was caused by copy and paste). Examples are also improved now.

heddn’s picture

Can you post an interdiff? That will help make the review process easier.

jigarius’s picture

I am attaching a diff of the changes I made after your previous feedback to ensure you have something. I will figure out what an interdiff is and post it ASAP.

Update: I think what I posted with this comment is actually an interdiff. I had never heard the word before.

heddn’s picture

Status: Needs review » Needs work

Here's a link to the docs on creating an interdiff: https://www.drupal.org/documentation/git/interdiff. I didn't see the docs changes in the ArrayPop class in the interdiff, but I found that they were done.

Just a small nit. I almost don't want to mention it, but it would be nice to add. Otherwise this is looking really good.

+++ b/tests/src/Unit/process/ArrayPopTest.php
@@ -0,0 +1,65 @@
+    // Indexed array.
+    $output[] = [['v1', 'v2', 'v3'], 'v3'];
+    // Associative array.
+    $output[] = [['i1' => 'v1', 'i2' => 'v2', 'i3' => 'v3'], 'v3'];
+    // Empty array.
+    $output[] = [[], NULL];

If you make the array keys 'index_array', 'associative_array', 'empty_array', it will be self documenting.

i.e.

return [
  ['indexed_array' => [['v1', 'v2', 'v3'], 'v3'],
  ['associative_array' => [['i1' => 'v1', 'i2' => 'v2', 'i3' => 'v3'], 'v3'],
...
];
jigarius’s picture

Status: Needs work » Needs review
FileSize
1.55 KB

I am submitting a diff again for now. I saw the interdiff docs - thanks for that. I think I'll need to find some time to try that out sometime soon.

I have removed the comments and used array keys to define test context. However, I have used spaces instead of underscores in keys cause I think it's mostly for the UI only (which looks better without underscores). Let me know if it's better to use underscores and I'll change. Thanks again.

Update: I think the simpletest UI might have some kind of a problem. It shows this text (note the wrongly positioned parentheses):
Drupal\Tests\migrate_plus\Unit\process\ArrayShiftTest->testArrayShift with data set "indexed array"()

heddn’s picture

Status: Needs review » Needs work
+++ b/tests/src/Unit/process/ArrayPopTest.php
@@ -30,12 +30,9 @@ class ArrayPopTest extends MigrateProcessTestCase {
+    $output['indexed array'] = [['v1', 'v2', 'v3'], 'v3'];

Maybe add a key of 'actual' and 'expected' to these too? That would make it even more super clear. I just now noticed that, sorry I didn't mention it earlier.

And the patch file needs to end in .patch for the testbot to pick it up.

jofitz’s picture

Status: Needs work » Needs review
FileSize
1.51 KB
6.59 KB
1.97 KB
6.88 KB

Addressed @heddn's comment in #23.

Also included the interdiff and patch for the work in #22.

jigarius’s picture

Ok. So Joe managed to submit his patch before I could upload mine. Great! Progress = Good.

jigarius’s picture

FileSize
1.66 KB
6.89 KB

Some coding standard fixes.

jigarius’s picture

Assigned: Unassigned » jigarius
jigarius’s picture

heddn’s picture

Status: Needs review » Reviewed & tested by the community

Looks good. Thanks everyone your contributions.

  • heddn committed 8148250 on 8.x-4.x authored by jigarius
    Issue #2881779 by jigarius, Jo Fitzgerald, heddn: Support "shift" and "...
heddn’s picture

Status: Reviewed & tested by the community » Fixed

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

benjifisher’s picture

I ran into this problem, too. It is convenient to have the new array_shift plugin, but you can get the same effect like this:

first_author:
  -
    plugin: callback
    callable: array_values
    source: authors
  -
    plugin: extract
    index:
      - 0

Note that array_values() calls its argument by value, not reference. Thus it does not have the problem mentioned in #11 for array_pop() and array_shift().