Support for Drupal 7 is ending on 5 January 2025—it’s time to migrate to Drupal 10! Learn about the many benefits of Drupal 10 and find migration tools in our resource center.
My source data contains the following data:
"field_person_topic": []
When parsing with the entity_lookup module, I get the following error:
exception 'Drupal\Core\Database\InvalidQueryException' with message 'Query condition 'taxonomy_term_field_data.name IN ()' cannot be empty.' in /var/www/core/lib/Drupal/Core/Database/Query/Condition.php:103
I think the problem is that the plugin is not handling empty arrays in the "query" method; it merely checks to see if $value is an array without checking that it contains values.
Here's the basics of my migration:
langcode: en
status: true
id: migrate_profile
source:
plugin: url
data_fetcher_plugin: http
data_parser_plugin: json
urls: 'https://domain.com/profile.json?id=abc123'
fields:
-
name: topic
selector: field_person_topic
process:
field_bio_topic:
plugin: entity_lookup
source: topic
value_key: name
bundle_key: vid
bundle: profile_topics
entity_type: taxonomy_term
ignore_case: true
Comment | File | Size | Author |
---|---|---|---|
#21 | empty_array_check-2902335-21.patch | 13.64 KB | tmountjr |
#15 | empty_array_check-2902335-15.patch | 13.73 KB | tmountjr |
#8 | add_entity_lookup_test-2902335-8.patch | 14.53 KB | tmountjr |
#2 | empty_array_check-2902335.patch | 873 bytes | tmountjr |
Comments
Comment #2
tmountjr CreditAttribution: tmountjr at Eastern Standard commentedComment #3
heddnWe're really trying to see increased test coverage. So bumping back to NW for a test.
Comment #4
tmountjr CreditAttribution: tmountjr at Eastern Standard commentedComment #5
tmountjr CreditAttribution: tmountjr at Eastern Standard commentedI'm happy to write a few tests for this but I have to confess I've never written unit tests for Drupal and it's a bit of a learning curve. Is there documentation for how to get everything set up relating to this module specifically? There's no phpunit.xml file in this repo, so I assume testing has to be done in the context of a full drupal installation, but the way the current tests are organized just for migrate_plus is a bit confusing to a newcomer. A few questions off the top of my head:
* I assume this should be in the Unit\process namespace, not the Kernel\Plugin\migrate\process namespace? The Kernel stuff looks more end-to-end, while this test just needs to make sure the transform method works properly. (Though, after writing this and playing with it a bit more, I'm not sure I can get all the dependencies required by the entity_lookup plugin from within a simple unit test; maybe it has to go into the Kernel tests?)
* The only unit test is currently for skip_on_value, but I can't quite use that as a pattern because the constructor for entity_lookup uses some values that I'm not sure where they'd come from in a testing scenario (specifically the MigrationInterface, EntityManagerInterface, and SelectionPluginManagerInterface values if I'm using "new EntityLookup(...)" or the ContainerInterface if I'm using "EntityLookup::create()" in my setup).
* I ended up getting stuck while writing the Kernel test because I think if I'm going to write the kernel test I have to basically write an entire migration in the test?
Like I said, I'm new to Drupal unit testing and I'm not 100% sure how to proceed in this case. I'm attaching what I have so far as a test:
Comment #6
tmountjr CreditAttribution: tmountjr at Eastern Standard commentedComment #7
heddnThere's already a test for entity generate: EntityGenerateTest. Use it as an example for lookup. Glad to see you jumping in there. And the reason that Kernel is used is because its just easier to mock things in a Kernel test rather than a Unit test.
Comment #8
tmountjr CreditAttribution: tmountjr at Eastern Standard commentedI've written a test that should cover the basic functionality of entity_lookup as well as the observed defect:
1. Test that looking up a single value that has a matching term saves a term reference.
2. Test that looking up a single value that does not have a matching term does not save a term reference.
3. Test that looking up multiple values that all have matching terms saves multiple term references.
4. Test that looking up multiple values, one of which has a matching term and the other which does not, saves only the matched term reference.
5. *DEFECT* Test that looking up an empty array does not save any term references and also does not throw an exception.
Comment #9
tmountjr CreditAttribution: tmountjr at Eastern Standard commentedComment #10
heddnCan you take a look at EntityGenerateTest more closely, there's a transformDataProvider there that should cover this scenario already. No need to add a whole new class, just add a new scenario.
Comment #11
tmountjr CreditAttribution: tmountjr at Eastern Standard commentedI created a new class because we're testing two different plugins - unfortunately entity_generate doesn't share any code with entity_lookup, even when entity_generate looks up taxonomy terms. And the specific test case I'm covering - that passing an empty array to entity_lookup causes the lookup to fail - isn't something that would be testable in entity_generate. (Though now that I'm thinking about it, I wonder how entity_generate would handle an empty array.) So from a code organization standpoint and a test coverage standpoint, the specific test case and indeed any of the tests for entity_lookup I would think should live in a class called EntityGenerateTest.
Having said that, there is definitely overlap between the code for both and I'm sure some of that overlap could be broken out into a common class and some of the adjustments I had to make for entity_lookup could be ported over to entity_generate; but the core of the test - that looking up an empty array doesn't cause an error - I think can't/shouldn't be done in entity_generate because entity_generate wasn't the problem.
Comment #12
ultimike+1 for this patch, it saved my bacon today :)
-mike
Comment #13
heddn#11: entity lookup is extended by entity generate.
Comment #14
tmountjr CreditAttribution: tmountjr at Eastern Standard commentedI'll take another crack at this now that I've written my own plugins and tests for those plugins.
Comment #15
tmountjr CreditAttribution: tmountjr at Eastern Standard commentedThe plugin itself didn't need any modifications but the test needed to be reworked since the original
EntityGenerateTest
changed since I played with this last. So here's a new patch incorporating both the plugin and the test.Comment #16
tmountjr CreditAttribution: tmountjr at Eastern Standard commentedComment #17
JvE CreditAttribution: JvE at One Shoe commentedI don't think the query should be skipped for strings.
You might want to find an entity which has an empty string as a field value.
Comment #18
tmountjr CreditAttribution: tmountjr at Eastern Standard commentedAs a lookup, though? That line is only seeing what kind of value should be returned in the case of an empty value; if the value passed in was an empty string, it returns
NULL
. I'm struggling to find a use case where an empty string would be intentionally used as a lookup value. I mean, if there is a use case, then sure, we can remove that logic; but part of me thinks if there's a legitimate need to look up something via an empty string, there's probably a better way to find what you're trying to look up.Comment #19
JvE CreditAttribution: JvE at One Shoe commentedYeah I know. But the same case could be made for "0", FALSE, NULL and perhaps even 0.
And since the bug/error only occurs when $value is an array and not with any of the other "falsy" values, I would not change the behaviour for those.
In fact, I'd fix this bug in the query() method. You never know if a plugin extending this one has a use case for empty arrays.
It is after all only the sql backend that has a problem with empty arrays.
Comment #20
tmountjr CreditAttribution: tmountjr at Eastern Standard commentedThat's fair. I'll look into restricting the logic to just arrays for the moment (rather than PHP's nebulous definition of what constitutes an
empty()
value). Tracking and stopping this in thequery()
method is a separate ball of wax for the moment.Comment #21
tmountjr CreditAttribution: tmountjr at Eastern Standard commentedMade my modification a bit less greedy. Still kept it in the
transform
method though; no use wasting extra cycles going to thequery
method if we can bail before that point.Comment #22
heddnThis is looking real nice. Some minor feedback.
Could we just do an empty check? And if empty return $value?
This seems like debug code?
This array key seems unclear. Can we have one value that exists already and one that gets created? Or something to show a mixture?
Comment #23
JvE CreditAttribution: JvE at One Shoe commentedNope. The
$value
is a key or set of keys used to look up the actual value(s).The problem is in the
query()
method. That treats arrays as special but does not deal with empty arrays.See #19
Comment #24
heddnOK, that makes sense. Someone might want to actually query for a value of empty or 0.
Would the follow then work?
Comment #25
eelkeblokThe last patch nog longer applies, it would appear some or all WIP for this issue was committed as part of the last commit for #2902867: Documentation for skip_on_value.
Comment #26
jpoesen CreditAttribution: jpoesen at Code Enigma commentedFWIW: for me 8.x-4.x-rc1 fixed this problem.
Comment #27
eelkeblok