Problem/Motivation

In Drupal 7, the entityreference module allowed to create multiple field instances of the same field. The migration here "works" but then any time an entity without any data attempts to set an entity reference value an InvalidArgumentException is thrown in ContentEntityBase.

InvalidArgumentException: Field <field name> is unknown.

A comparison between getting field definitions directly from entity_field.manager reveals that the field does in fact exist on the bundle, but getting the field definition on the entity itself fails. I found that if I reset the field definitions before the exception, then it tries to go further, but runs into a similar exception in typed_data_manager.

Furthermore this is only an issue for one of the bundles. The other bundle works fine.

Steps to reproduce (as detailed in a test-only patch):

  1. Add field_reference_3 field_config
  2. Add field_reference_3 field_config_instance to both article and forum node bundles.
  3. Run the appropriate migrations.
  4. Load node 2 (an article node) e.g. $node = Node::load(2);
  5. Attempt to set an initial reference e.g. $node->set('field_reference_3', [5]); or $node->get('field_reference_3')->appendItem()->set('field_reference_3', [5]); or $node->field_reference_3 = ['target_id' => 5];
  6. An exception is thrown and we can see that the field definition doesn't exist on the node, but does exist on field manager.

This might be why #2814963: Support Drupal 7 entity reference fields is still open/postponed even though plugins exist for it?

Proposed resolution

Reset the entity cache of the entity type that a field instance is attached to after the field instance is migrated.

Remaining tasks

1. Review proposed resolution
2. Reword test comments to accurately describe the issue and problem

User interface changes

Unknown

API changes

Unknown

Data model changes

Unknown

CommentFileSizeAuthor
#53 interdiff-3008202-49-53.txt794 bytesvoleger
#53 3008202-53-test-only.patch11.41 KBvoleger
#49 3008202-49-test-only.patch10.96 KBbenjifisher
#49 diff-3008202-18-49.txt4.68 KBbenjifisher
#44 3008202-44.patch14.7 KBdurgeshs
#44 interdiff_42-44.txt2.85 KBdurgeshs
#42 interdiff_39-42.txt1.03 KBdurgeshs
#42 3008202-42.patch16.01 KBdurgeshs
#39 interdiff_35-39.txt1.98 KBdurgeshs
#39 3008202-39.patch15.78 KBdurgeshs
#35 3008202-35.patch16.21 KBdurgeshs
#35 interdiff_28-35.txt2.82 KBdurgeshs
#31 3008202-not-applied.png73.24 KBdurgeshs
#31 3008202-31.patch107.57 KBdurgeshs
#28 3008202-28.patch15.82 KBquietone
#28 interdiff-19-28.txt21.8 KBquietone
#24 3008202-d88-fix-only-no-test.patch3.63 KBmradcliffe
#19 interdiff-3008202-18-19.txt3.46 KBmradcliffe
#19 3008202-19.patch14.71 KBmradcliffe
#18 3008202-18-test-only.patch11.08 KBmradcliffe
#17 drupal-3008202-multiple-entityreference-17-test-only.patch11.05 KBmradcliffe
#17 interdiff-3008202-15-17.txt1.5 KBmradcliffe
#15 interdiff-3008202-testonly-13-15.txt1.44 KBmradcliffe
#15 drupal-3008202-multiple-entityreference-15-test-only.patch11.04 KBmradcliffe
#14 interdiff-3008202-testonly-13-14.txt2.45 KBmradcliffe
#14 drupal-3008202-multiple-entityreference-14-test-only.patch12.17 KBmradcliffe
#13 drupal-3008202-multiple-entityreference-13-test-only.patch10.89 KBmradcliffe
#9 interdiff-3008202-8-9.txt1.66 KBmradcliffe
#9 drupal-3008202-multiple-entityreference-9.patch12.21 KBmradcliffe
#8 interdiff-3008202-5-8.txt555 bytesmradcliffe
#8 drupal-3008202-multiple-entityreference-8-test-only.patch11.06 KBmradcliffe
#5 interdiff-3008202-2-5.txt17.03 KBmradcliffe
#5 drupal-3008202-multiple-entityreference-5-test-only.patch11.01 KBmradcliffe
#2 drupal-3008202-multiple-entityreference-2-test-only.patch10.19 KBmradcliffe
#2 interdiff-3008202-1-2.txt1.23 KBmradcliffe
drupal-entityreference-migrate-test-only.patch10.19 KBmradcliffe

Issue fork drupal-3008202

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

mradcliffe created an issue. See original summary.

mradcliffe’s picture

Uploaded a slightly better patch.

mradcliffe’s picture

I'm more confident this is an issue with migration rather than Entity API.

I tried to reproduce directly with the Entity API by creating 3 content types, 2 entity reference fields of the same attached to 2 of those node types, creating 1 node for each of the content types, saving, then setting the 2 nodes with reference fields. I didn't run into an issue with setting values.

Not really any value in uploading patch with that test, but here's the gist based on a FieldKernelTest:

  protected function setUp() {
    parent::setUp();

    $this->installEntitySchema('node');
    $this->installSchema('node', ['node_access']);

    $referencedNodeType = NodeType::create(['type' => $this->randomMachineName()]);
    $referencedNodeType->save();

    $this->referencedNode = Node::create(['title' => $this->getRandomGenerator()->sentences(5, TRUE), 'type' => $referencedNodeType->id(), 'status' => Node::PUBLISHED]);
    $this->referencedNode->save();

    $nodeType1 = NodeType::create(['type' => $this->randomMachineName()]);
    $nodeType1->save();

    $this->node1 = Node::create(['title' => $this->getRandomGenerator()->sentences(5, TRUE), 'type' => $nodeType1->id(), 'status' => NODE::PUBLISHED]);
    $this->node1->save();

    $nodeType2 = NodeType::create(['type' => $this->randomMachineName()]);
    $nodeType2->save();
    $this->node2 = Node::create(['title' => $this->getRandomGenerator()->sentences(5, TRUE), 'type' => $nodeType2->id(), 'status' => NODE::PUBLISHED]);
    $this->node2->save();

    $this->createEntityReferenceField('node', $nodeType1->id(), 'field_test_node', 'Node reference', 'node');
    $this->createEntityReferenceField('node', $nodeType2->id(), 'field_test_node', 'Node reference', 'node');
  }

  /**
   * Asserts can create entity reference to a node.
   */
  public function testBlah() {
    $this->node1->field_test_node[] = ['target_id' => $this->referencedNode->id()];
    $this->assertCount(1, $this->node1->field_test_node);

    $this->node2->field_test_node[] = ['target_id' => $this->referencedNode->id()];
    $this->assertCount(1, $this->node2->field_test_node);
  }

Version: 8.7.x-dev » 8.8.x-dev

Drupal 8.7.0-alpha1 will be released the week of March 11, 2019, which means new developments and disruptive changes should now be targeted against the 8.8.x-dev branch. For more information see the Drupal 8 minor version schedule and the Allowed changes during the Drupal 8 release cycle.

mradcliffe’s picture

Coming back to this issue.

- I fixed the failing test that happened when I added a new field config/instance.
- Rebased off of 8.8.x and fixed the config / test changes that landed recently in 8.7.x. This makes the interdiff kind of useless as it had problems dealing with the changes to core.

This should only have the one isolated test failure.

I think that narrowing things down, maybe add field data for forum (or article), and add an assert about that field definition to see if it only happens with reference fields migrated without data?

mpp’s picture

Status: Active » Needs review
mpp’s picture

Status: Needs review » Needs work

Test only so setting back to needs work.

mradcliffe’s picture

Another number to increment.

Also, I'm manually adding the test after I post a comment rather than flipping status.

mradcliffe’s picture

Status: Needs work » Needs review
Issue tags: +Performance
StatusFileSize
new12.21 KB
new1.66 KB

I'm fairly certain that this is the wrong solution, but it does fix the issue. I write this because of the research I did earlier about trying to reproduce in the entity system, in which I could not. However this could serve as a core patch for someone that needs migrations that don't lead to a broken site.

This probably has some performance impact.

Status: Needs review » Needs work

The last submitted patch, 9: drupal-3008202-multiple-entityreference-9.patch, failed testing. View results

mradcliffe’s picture

Resetting the storage cache seems to help, but I'm not sure how that could be done system-wide for all entity reference fields on migration.

Also, the other issue with the approach I had, is that there are non-properties on nodes and users such as "user_cancel_notify" or "original", which mean this probably gets called often.

Version: 8.8.x-dev » 8.9.x-dev

Drupal 8.8.0-alpha1 will be released the week of October 14th, 2019, which means new developments and disruptive changes should now be targeted against the 8.9.x-dev branch. (Any changes to 8.9.x will also be committed to 9.0.x in preparation for Drupal 9’s release, but some changes like significant feature additions will be deferred to 9.1.x.). For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

mradcliffe’s picture

Re-rolling #8 and adjusting field_config, field_storage_config counts in Upgrade7Test. It should be correct, but the merge conflict was a bit tricky. I removed the "fix" that I added in #9.

I think that saving the entity should work during migration to address this, but want a clean test-only patch to work with.

mradcliffe’s picture

Nope, I screwed up that merge. Sigh. Ended up in vim because phpstorm ground to a halt trying to syntax check.

mradcliffe’s picture

And of course I screwed THAT patch up by including the bad fix. Sigh. I'm tired and it's late.

mradcliffe’s picture

Issue tags: +Needs reroll

I still must have messed the merge up because all the migrate tests are now failing. :(

mradcliffe’s picture

I think this should get the patch back to the state it was in #8.

mradcliffe’s picture

StatusFileSize
new11.08 KB

Test only patch needs another re-roll.

2 new fields and field instances got added to drupal7.php. The interdiff failed.

mradcliffe’s picture

Status: Needs work » Needs review
StatusFileSize
new14.71 KB
new3.46 KB

Resetting the entity storage cache(s) of the entity type of a field instance after importing the field instance should do the trick here.

This could potentially cause a site to be slower post-migration if the migration was run in multiple parts as each entity will need to refetch the field definitions as needed. But that data was stale and bad anyway.

mradcliffe’s picture

Issue summary: View changes

Updated out-of-date issue summary.

mradcliffe’s picture

Issue summary: View changes
+++ b/core/modules/field/tests/src/Kernel/Migrate/d7/MigrateEmptyReferenceFieldTest.php
@@ -0,0 +1,93 @@
+   * Asserts that something is wrong with empty field instances on nodes.

I think that's why I had "refactor test" in the issue summary ;-)

Adding that back as a remaining task in the issue summary, but reworded to better clarify the intent of why I originally had it as a remaining task.

The last submitted patch, 18: 3008202-18-test-only.patch, failed testing. View results

Status: Needs review » Needs work

The last submitted patch, 19: 3008202-19.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

mradcliffe’s picture

StatusFileSize
new3.63 KB

Uploading a 8.8.x fix only patch so that I can apply it in composer.json.

dhirendra.mishra’s picture

Status: Needs work » Needs review

Pls make sure to move status to needs review after patch

mradcliffe’s picture

Status: Needs review » Needs work

This still needs to be Needs work because the relevant patch is #19. I probably should have re-uploaded it in #24 as well though. The patch in #24 is not for testing or review, but mainly for my own use with a stable version of Drupal.

Current status is to refactor the test assertions so that it's not a proof-of-concept test and to re-roll based on changes to core.

Version: 8.9.x-dev » 9.1.x-dev

Drupal 8.9.0-beta1 was released on March 20, 2020. 8.9.x is the final, long-term support (LTS) minor release of Drupal 8, which means new developments and disruptive changes should now be targeted against the 9.1.x-dev branch. For more information see the Drupal 8 and 9 minor version schedule and the Allowed changes during the Drupal 8 and 9 release cycles.

quietone’s picture

Status: Needs work » Needs review
StatusFileSize
new21.8 KB
new15.82 KB

Reroll and update of the patch in #19. The patch in #19 added a new test, which has been removed and the assertions moved to the relevant existing tests, MigrateField, MigrateFieldInstance and MigrateNode.

Status: Needs review » Needs work

The last submitted patch, 28: 3008202-28.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

durgeshs’s picture

Assigned: Unassigned » durgeshs
durgeshs’s picture

Issue summary: View changes
StatusFileSize
new107.57 KB
new73.24 KB

Previous patch no longer to applied

So I have rerolled previous patch. Kindly check it.

durgeshs’s picture

Status: Needs work » Needs review

Waiting for test cases. I am working for test cases.

quietone’s picture

@durgeshs, thanks for the reroll. There is no need to output the results of a git pull and git apply, we can see from the test run that the patch does not apply. To aid the reviewer please provide at least an interdiff or if that fails a diff, See Creating an interdiff in the handbook.

Status: Needs review » Needs work

The last submitted patch, 31: 3008202-31.patch, failed testing. View results
- codesniffer_fixes.patch Interdiff of automated coding standards fixes only.

durgeshs’s picture

Assigned: durgeshs » Unassigned
Status: Needs work » Needs review
StatusFileSize
new2.82 KB
new16.21 KB

@quietone,

I have added interdiff and new patch with resolve test case under core/modules/field/tests/src/Kernel/Migrate/d7.

Status: Needs review » Needs work

The last submitted patch, 35: 3008202-35.patch, failed testing. View results

quietone’s picture

Found two out of scope changes that were introduced by me earlier on.

  1. +++ b/core/modules/field/tests/src/Kernel/Migrate/d7/MigrateFieldInstanceTest.php
    @@ -51,7 +51,7 @@ protected function setUp(): void {
    +    [$expected_entity_type, $expected_bundle, $expected_name] = explode('.', $id);
    

    out of scope

  2. +++ b/core/modules/field/tests/src/Kernel/Migrate/d7/MigrateFieldTest.php
    @@ -53,7 +53,7 @@ protected function setUp(): void {
    +    [$expected_entity_type, $expected_name] = explode('.', $id);
    

    Out of scope

  3. +++ b/core/modules/field/tests/src/Kernel/Migrate/d7/MigrateFieldTest.php
    @@ -102,7 +102,7 @@
    +    $this->assertEntity('node.field_reference_3', 'entity_reference', TRUE, 1);
    

    Good. I checked the field definition and it does have a cardinality of 1.

  4. +++ b/core/modules/field/tests/src/Kernel/Migrate/d7/MigrateFieldInstanceTest.php
    @@ -211,7 +212,7 @@ public function testTextFieldInstances() {
    +    $this->assertCount(9, $errors);
    

    Why has the error count increased? Is is supposed to? What is the error message?

durgeshs’s picture

Assigned: Unassigned » durgeshs
durgeshs’s picture

Assigned: durgeshs » Unassigned
Status: Needs work » Needs review
StatusFileSize
new15.78 KB
new1.98 KB

@quietone,

I have done changes as per #37, kindly review it.

quietone’s picture

Status: Needs review » Needs work

@durgeshs, thanks for fixing the out of scope changes I introduced.

+++ b/core/modules/field/tests/src/Kernel/Migrate/d7/MigrateFieldInstanceTest.php
@@ -226,6 +226,8 @@
+    $message = 'New object was not saved, no error provided';

This will allow the test to pass but I don't think the test itself is correct. This issue is to migrate more that 1 entity reference field so getting a new errors seems wrong. The changes here that allow the test to pass need to be explained.

Settings to NW for #37.4 and the above change to the test.

durgeshs’s picture

Assigned: Unassigned » durgeshs

@quietone, Thanks for feedback.

Sure, I will check and resolve it.

durgeshs’s picture

Assigned: durgeshs » Unassigned
StatusFileSize
new16.01 KB
new1.03 KB

@quietone,

I have checked code and found that there are getting one extra message when migrate more than 1 entity reference field. This message occurring because of the following code
import(Row $row, array $destination_id_values),
$destination_id_values is getting blank array for first entity reference filed.
Please feel free to let me know if there are any best solution.

Meanwhile in the current patch, I have added a proper comment for this.

quietone’s picture

@durgeshs, Yes, that is not unexpected, it will be failing somewhere in the import process. Just have to find out where. Maybe I can do some debugging on this in the next couple of days.

+++ b/core/modules/field/tests/src/Kernel/Migrate/d7/MigrateFieldInstanceTest.php
@@ -211,7 +212,7 @@ public function testTextFieldInstances() {
+    $this->assertCount(9, $errors);

@@ -225,6 +226,11 @@ public function testTextFieldInstances() {
+    // Due to migrate more than 1 entity reference field so getting a new errors
+    // message. Because of in the import(Row $row, array $destination_id_values) ¶
+    // $destination_id_values is getting blank array for first entity.
+    $message = 'New object was not saved, no error provided';
+    $this->assertEquals($errors[8], $message);

The comment before the tests on the error messages states that it is for text field instances not entity reference field. So, these changes need to be removed. Sorry about that.

+++ b/core/modules/field/tests/src/Kernel/Migrate/d7/MigrateFieldInstanceTest.php
@@ -106,6 +106,7 @@ public function testFieldInstances() {
+    $this->assertEntity('node.article.field_reference_3', 'Reference 3', 'entity_reference', FALSE, TRUE);

The fixture add two instances of the field_reference_2 and we are only checking that one on the article is migrated. A similar test needs to be added for the forum bundle. Please add it a few lines down where the other 'forum' bundle field instances are tested.

durgeshs’s picture

StatusFileSize
new2.85 KB
new14.7 KB

@quietone, Thanks for the feedback, I have addressed those two comments of #43 and will be waiting for your debugging input.

quietone’s picture

Status: Needs work » Postponed (maintainer needs more info)

I manually tested today and wasn't able to produce an error. I followed the steps 1, 2 and 3 in the ISS then I used the UI to edit node 2 and add a reference. There were no errors in the log.

@mradcliffe, is this still a problem?

Version: 9.1.x-dev » 9.2.x-dev

Drupal 9.1.0-alpha1 will be released the week of October 19, 2020, which means new developments and disruptive changes should now be targeted for the 9.2.x-dev branch. For more information see the Drupal 9 minor version schedule and the Allowed changes during the Drupal 9 release cycle.

quietone’s picture

Is this issue still a problem?

There has been no activity here for 7 months.

Since we need more information to move forward with this issue, I am keeping the status at Postponed (maintainer needs more info). If we don't receive additional information to help with the issue, it may be closed after three months.

Version: 9.2.x-dev » 9.3.x-dev

Drupal 9.2.0-alpha1 will be released the week of May 3, 2021, which means new developments and disruptive changes should now be targeted for the 9.3.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

benjifisher’s picture

Issue summary: View changes
Status: Postponed (maintainer needs more info) » Needs work
StatusFileSize
new4.68 KB
new10.96 KB

Based on #45, it looks as though this issue was solved by some other changes in Drupal.

I tried to confirm this. I applied the test-only patch from #18 to commit 45b25f08b7 (2019-12-13). I installed PHP 7.2 and did composer install (with Composer 1.10.13) to get PHPUnit 6.5.14. After this experiment in time travel, I got the same test failure reported by the testbot in #18: "Failed asserting that an array has the key 'field_reference_3'."

Then, with some difficulty, I rerolled the test-only patch so that it applies to 9.3.x. With PHP 7.3 and PHPUnit 8.5.15, I get the same failure.

Part of the reroll is removing the reference to NodeCommentCombinationTrait, which was removed in #3111344: Remove deprecated code from migrate module. The test-only patch did not use the method defined in that trait, so no related changes were needed.

There are a lot of things I have not done: I have not looked at the test to decide what it does. Maybe it is bogus. Maybe it is not testing what the issue summary says. I have not compared the failing test to the tests added by @quietone in #28 and later patches. I have not done any manual testing.

I did check that the patch in #18 is a test-only version of the patch in #19.

I am removing an unhelpful screenshot from the issue summary.

I think we have a failing test, so I am setting the status back to NW.

P.S. It is also quite possible that I made a mistake while rerolling the patch. But then it would be quite a coincidence that I get the same test failure.

Version: 9.3.x-dev » 9.4.x-dev

Drupal 9.3.0-rc1 was released on November 26, 2021, which means new developments and disruptive changes should now be targeted for the 9.4.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.4.x-dev » 9.5.x-dev

Drupal 9.4.0-alpha1 was released on May 6, 2022, which means new developments and disruptive changes should now be targeted for the 9.5.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

Version: 9.5.x-dev » 10.1.x-dev

Drupal 9.5.0-beta2 and Drupal 10.0.0-beta2 were released on September 29, 2022, which means new developments and disruptive changes should now be targeted for the 10.1.x-dev branch. For more information see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

voleger’s picture

StatusFileSize
new11.41 KB
new794 bytes

Rerolled #43 test-only patch

voleger’s picture

Version: 10.1.x-dev » 11.x-dev

Drupal core is moving towards using a “main” branch. As an interim step, a new 11.x branch has been opened, as Drupal.org infrastructure cannot currently fully support a branch named main. New developments and disruptive changes should now be targeted for the 11.x branch, which currently accepts only minor-version allowed changes. For more information, see the Drupal core minor version schedule and the Allowed changes during the Drupal core release cycle.

joelpittet made their first commit to this issue’s fork.

joelpittet’s picture

Version: 11.x-dev » main

Drupal core is now using the main branch as the primary development branch. New developments and disruptive changes should now be targeted to the main branch.

Read more in the announcement.