My source data contains the following data:

"field_person_topic": []

When parsing with the entity_lookup module, I get the following error:

exception 'Drupal\Core\Database\InvalidQueryException' with message 'Query condition 'taxonomy_term_field_data.name IN ()' cannot be empty.' in /var/www/core/lib/Drupal/Core/Database/Query/Condition.php:103

I think the problem is that the plugin is not handling empty arrays in the "query" method; it merely checks to see if $value is an array without checking that it contains values.

Here's the basics of my migration:

langcode: en
status: true
id: migrate_profile

source:
  plugin: url
  data_fetcher_plugin: http
  data_parser_plugin: json
  urls: 'https://domain.com/profile.json?id=abc123'
  fields:
    -
      name: topic
      selector: field_person_topic
  
process:
  field_bio_topic:
    plugin: entity_lookup
    source: topic
    value_key: name
    bundle_key: vid
    bundle: profile_topics
    entity_type: taxonomy_term
    ignore_case: true
Support from Acquia helps fund testing for Drupal Acquia logo

Comments

tmountjr created an issue. See original summary.

tmountjr’s picture

heddn’s picture

Status: Active » Needs work
Issue tags: +Needs tests

We're really trying to see increased test coverage. So bumping back to NW for a test.

tmountjr’s picture

Issue summary: View changes
tmountjr’s picture

I'm happy to write a few tests for this but I have to confess I've never written unit tests for Drupal and it's a bit of a learning curve. Is there documentation for how to get everything set up relating to this module specifically? There's no phpunit.xml file in this repo, so I assume testing has to be done in the context of a full drupal installation, but the way the current tests are organized just for migrate_plus is a bit confusing to a newcomer. A few questions off the top of my head:

* I assume this should be in the Unit\process namespace, not the Kernel\Plugin\migrate\process namespace? The Kernel stuff looks more end-to-end, while this test just needs to make sure the transform method works properly. (Though, after writing this and playing with it a bit more, I'm not sure I can get all the dependencies required by the entity_lookup plugin from within a simple unit test; maybe it has to go into the Kernel tests?)
* The only unit test is currently for skip_on_value, but I can't quite use that as a pattern because the constructor for entity_lookup uses some values that I'm not sure where they'd come from in a testing scenario (specifically the MigrationInterface, EntityManagerInterface, and SelectionPluginManagerInterface values if I'm using "new EntityLookup(...)" or the ContainerInterface if I'm using "EntityLookup::create()" in my setup).
* I ended up getting stuck while writing the Kernel test because I think if I'm going to write the kernel test I have to basically write an entire migration in the test?

Like I said, I'm new to Drupal unit testing and I'm not 100% sure how to proceed in this case. I'm attaching what I have so far as a test:

<?php

namespace Drupal\Tests\migrate_plus\Kernel\Plugin\migrate\process;

use Drupal\taxonomy\Entity\Term;
use Drupal\KernelTests\KernelTestBase;
use Drupal\taxonomy\Entity\Vocabulary;
use Drupal\migrate\Plugin\MigrationInterface;
use Drupal\migrate_plus\Plugin\migrate\process\EntityLookup;

/**
 * Tests the Entity Lookup process plugin.
 *
 * @group migrate_plus
 * @coversDefaultClass \Drupal\migrate_plus\Plugin\migrate\process\EntityLookup
 */
class EntityLookupTest extends KernelTestBase {

  /**
   * {@inheritdoc}
   */
  public static $modules = [
    'taxonomy',
    'text',
  ];

  /**
   * The vocabulary id.
   *
   * @var string
   */
  protected $vocabulary = 'test_vocab';

  /**
   * The vocabulary terms.
   *
   * @var array
   */
  protected $terms = [
    'test_term1',
    'test_term2',
  ];

  /**
   * {@inheritdoc}
   */
  protected function setUp() {
    parent::setUp();

    $this->installEntitySchema('taxonomy_term');
    $this->installEntitySchema('taxonomy_vocabulary');

    $vocab = Vocabulary::create([
      'name' => $this->vocabulary,
      'description' => $this->vocabulary,
      'vid' => $this->vocabulary,
      'langcode' => 'en',
    ]);
    $vocab->save();

    foreach ($this->terms as $term) {
      $t = Term::create([
        'name' => $term,
        'vid' => $this->vocabulary,
      ]);
      $t->save();
    }
  }

  /**
   * @covers ::transform
   */
  public function testTransformWithSingleValue() {
    $configuration['bundle'] = $this->vocabulary;
    $entityLookup = EntityLookup::create(
      $this->container,
      $configuration,
      'entity_lookup',
      [],
      $this->getMock(MigrationInterface::class)
    );
    $value = $entityLookup->transform('test_term1', $this->migrateExecutable, $this->row, 'destinationproperty');
    $this->assertNotNull($value);
  }

}
tmountjr’s picture

Component: API » Plugins
heddn’s picture

There's already a test for entity generate: EntityGenerateTest. Use it as an example for lookup. Glad to see you jumping in there. And the reason that Kernel is used is because its just easier to mock things in a Kernel test rather than a Unit test.

tmountjr’s picture

I've written a test that should cover the basic functionality of entity_lookup as well as the observed defect:

1. Test that looking up a single value that has a matching term saves a term reference.
2. Test that looking up a single value that does not have a matching term does not save a term reference.
3. Test that looking up multiple values that all have matching terms saves multiple term references.
4. Test that looking up multiple values, one of which has a matching term and the other which does not, saves only the matched term reference.
5. *DEFECT* Test that looking up an empty array does not save any term references and also does not throw an exception.

tmountjr’s picture

Status: Needs work » Needs review
heddn’s picture

Status: Needs review » Needs work

Can you take a look at EntityGenerateTest more closely, there's a transformDataProvider there that should cover this scenario already. No need to add a whole new class, just add a new scenario.

tmountjr’s picture

No need to add a whole new class, just add a new scenario.

I created a new class because we're testing two different plugins - unfortunately entity_generate doesn't share any code with entity_lookup, even when entity_generate looks up taxonomy terms. And the specific test case I'm covering - that passing an empty array to entity_lookup causes the lookup to fail - isn't something that would be testable in entity_generate. (Though now that I'm thinking about it, I wonder how entity_generate would handle an empty array.) So from a code organization standpoint and a test coverage standpoint, the specific test case and indeed any of the tests for entity_lookup I would think should live in a class called EntityGenerateTest.

Having said that, there is definitely overlap between the code for both and I'm sure some of that overlap could be broken out into a common class and some of the adjustments I had to make for entity_lookup could be ported over to entity_generate; but the core of the test - that looking up an empty array doesn't cause an error - I think can't/shouldn't be done in entity_generate because entity_generate wasn't the problem.

ultimike’s picture

+1 for this patch, it saved my bacon today :)

-mike

heddn’s picture

#11: entity lookup is extended by entity generate.

tmountjr’s picture

I'll take another crack at this now that I've written my own plugins and tests for those plugins.

tmountjr’s picture

The plugin itself didn't need any modifications but the test needed to be reworked since the original EntityGenerateTest changed since I played with this last. So here's a new patch incorporating both the plugin and the test.

tmountjr’s picture

Status: Needs work » Needs review
JvE’s picture

+++ b/src/Plugin/migrate/process/EntityLookup.php
@@ -152,6 +152,16 @@ class EntityLookup extends ProcessPluginBase implements ContainerFactoryPluginIn
+      if (gettype($value) === 'string') {

I don't think the query should be skipped for strings.
You might want to find an entity which has an empty string as a field value.

tmountjr’s picture

As a lookup, though? That line is only seeing what kind of value should be returned in the case of an empty value; if the value passed in was an empty string, it returns NULL. I'm struggling to find a use case where an empty string would be intentionally used as a lookup value. I mean, if there is a use case, then sure, we can remove that logic; but part of me thinks if there's a legitimate need to look up something via an empty string, there's probably a better way to find what you're trying to look up.

JvE’s picture

Yeah I know. But the same case could be made for "0", FALSE, NULL and perhaps even 0.
And since the bug/error only occurs when $value is an array and not with any of the other "falsy" values, I would not change the behaviour for those.

In fact, I'd fix this bug in the query() method. You never know if a plugin extending this one has a use case for empty arrays.
It is after all only the sql backend that has a problem with empty arrays.

tmountjr’s picture

And since the bug/error only occurs when $value is an array and not with any of the other "falsy" values, I would not change the behaviour for those.

That's fair. I'll look into restricting the logic to just arrays for the moment (rather than PHP's nebulous definition of what constitutes an empty() value). Tracking and stopping this in the query() method is a separate ball of wax for the moment.

tmountjr’s picture

Made my modification a bit less greedy. Still kept it in the transform method though; no use wasting extra cycles going to the query method if we can bail before that point.

heddn’s picture

Status: Needs review » Needs work

This is looking real nice. Some minor feedback.

  1. +++ b/src/Plugin/migrate/process/EntityLookup.php
    @@ -152,6 +152,11 @@ class EntityLookup extends ProcessPluginBase implements ContainerFactoryPluginIn
    +    if (gettype($value) === 'array' && count($value) === 0) {
    

    Could we just do an empty check? And if empty return $value?

  2. +++ b/tests/src/Kernel/Plugin/migrate/process/EntityGenerateTest.php
    @@ -139,20 +143,52 @@ class EntityGenerateTest extends KernelTestBase implements MigrateMessageInterfa
    +                      $this->assertTrue($entity->{$property}->isEmpty(), "FOOBAR Expected value is empty but field $property is not empty.");
    ...
    +                    $this->assertTrue($entity->{$property}->isEmpty(), "BINBAZ Expected value is empty but field $property is not empty.");
    

    This seems like debug code?

  3. +++ b/tests/src/Kernel/Plugin/migrate/process/EntityGenerateTest.php
    @@ -405,6 +441,273 @@ class EntityGenerateTest extends KernelTestBase implements MigrateMessageInterfa
    +      'lookup multiple mixed terms returns correct terms' => [
    

    This array key seems unclear. Can we have one value that exists already and one that gets created? Or something to show a mixture?

JvE’s picture

+++ b/src/Plugin/migrate/process/EntityLookup.php
@@ -152,6 +152,11 @@ class EntityLookup extends ProcessPluginBase implements ContainerFactoryPluginIn
+    if (gettype($value) === 'array' && count($value) === 0) {

Could we just do an empty check? And if empty return $value?

Nope. The $value is a key or set of keys used to look up the actual value(s).
The problem is in the query() method. That treats arrays as special but does not deal with empty arrays.
See #19

heddn’s picture

OK, that makes sense. Someone might want to actually query for a value of empty or 0.

Would the follow then work?

     // If the source data is an empty array, return the same.
-    if (gettype($value) === 'array' && count($value) === 0) {
-      return [];
+    if (is_array($value) && empty($value)) {
+      return $value;
     }
eelkeblok’s picture

The last patch nog longer applies, it would appear some or all WIP for this issue was committed as part of the last commit for #2902867: Documentation for skip_on_value.

jpoesen’s picture

FWIW: for me 8.x-4.x-rc1 fixed this problem.

eelkeblok’s picture

Status: Needs work » Closed (outdated)