Problem/Motivation

We created a feed to import an xml file. This works fine with a single import.
However, as soon we import more files or try to import with scheduled cron, we get in 75% of cases the error "entity already exists ", even it does not exist

We tested many workarounds or "fixes" all day:

  • removing our title field (which is the id) to be unique
  • adding GUID to mapping
  • allow updates
  • updating parsing
  • deleting everything

What are we doing wrong? With thousands of installations I cant imagine that this is not working, so we are obvisiouly something doing wrong.
Any help is appreciated!

Steps to reproduce

We tested on 2 websites, we deleted the module and installed new, no change

Proposed resolution

Remaining tasks

User interface changes

API changes

Data model changes

Issue fork feeds-3569983

Command icon Show commands

Start within a Git clone of the project using the version control instructions.

Or, if you do not have SSH keys set up on git.drupalcode.org:

Comments

beauregard created an issue. See original summary.

megachriz’s picture

Status: Active » Postponed (maintainer needs more info)

I've not seen an issue like this before. Is "entity already exists" the exact error message?

If you enable the Feeds Log module, does that reveal more information? With that module, after an import, you can see the exact data that Feeds receives and how the data looked like after parsing. Or did you already try that and is that where you got the "entity already exists" message from?

beauregard’s picture

StatusFileSize
new124.89 KB

yes, we have log enabled, here what I get
I cant paste images here, dont know why drupal.org has such an old editor instead ckeditor5.
Please find the PDF enclosed with the error message and the existing records
As you can see 3 records are not imported with "skipped", but it is not true, that the record exists. You can see the value in "Item ID" in the Log does not match any of the 2 values in all existing records

megachriz’s picture

Are the ID's of the two nodes 1904 and 1905? In the column "Entität" I see that are the node ID's that Feeds found when looking for existing entities (I see it only reports the ID instead of a link to the entity, probably because of avoiding loading the whole node).

The most likely cause is a misconfiguration on the mapping settings. Which target(s) do you have marked as unique? Have you perhaps set more than one target as unique? Feeds loops through all targets set as unique and tries to find an existing entity based on each individually. It does not look for existing entities based on a combination of targets. That behavior is perhaps a bit confusing. I'm not sure if that behavior was intentionally or not (it already worked that way before I took over the development of Feeds).

beauregard’s picture

correct, the two nodes are 1904 and 1905
No, i have marked exactly one field as "Unique". This is the title field of the node type we are filling
And we have these options
"Insert new content items"
"Do not update existing content items"

megachriz’s picture

Alright, can you share your feed type configuration + a sample source file with which the issue can be reproduced?

beauregard’s picture

StatusFileSize
new11.48 KB

I investigated now the problem with claude code (I provided on my local hd the feeds module, our custom module, feeds importer configuration and node type configurations). claude code analyzed first the files, then created a debug version and we debugged.
In the end the following problem was identified:

Bug: When a mapping has unique: value: '0' (unchecked in UI), the existingEntityId() method still processes it as a unique constraint because the code iterates over the array without checking if the value is truthy.

File: feeds/src/Feeds/Processor/EntityProcessorBase.php

Fix: Add if (empty($true)) { continue; } inside the foreach ($mapping['unique'] as $key => $true) loop at ~line 1259.

-----------------
Claude first assumed that I have several fields checked as "unique", but I dont have it, we checked it before. However in mapping configuration, for exactly three fields there is *no* checkbox "Unique", and this seems to be the problem.

See enclosed the screenshot of Admin UI in drupal to map fields, and the modified file as done by claude.
Could you please check it?

Note: I am not developer (this was 25 years ago), I am project manager and working with drupal 15+ years. Therefore I cant say if this is the right solution. I can only say that after this fix import is working

beauregard’s picture

StatusFileSize
new68.32 KB

here the screenshot of the Admin UI

megachriz’s picture

Okay, that is strange. When I add mapping to a field that cannot be marked as unique, I don't get the "unique" key set. And for mappings where "unique" is not checked, I get "unique" => []. So I wonder how something like "unique" => ["value"=> 0] could happen.

That being said, your fix seems reasonable. Just want to know if there is perhaps another bug in the UI that got you to this unusual state.

megachriz’s picture

Can you share your feed type configuration as well? (yml file) Maybe that helps to discover a bug in the UI as well.

beauregard’s picture

What you mean with "When I add mapping to a field that cannot be marked as unique"?
In my node type definition only title field is unique and a reference to a customer record. nothing else
But in feeds mapping page all fields have the checkbox "unique" offered, except the 3 mentioned fields
See enclosed the nodetype definition, maybe this helps

beauregard’s picture

StatusFileSize
new3.56 KB

and here the feeds type definition

megachriz’s picture

StatusFileSize
new287.78 KB
new288.34 KB

What you mean with "When I add mapping to a field that cannot be marked as unique"?

When I have the following mapping configuration:

I get:

mappings:
  -
    target: title
    map:
      value: ''
    settings:
      language: null
    unique: {  }
  -
    target: field_date
    map:
      value: ''
    settings:
      language: null
      timezone: UTC

And with the following:

I get:

mappings:
  -
    target: title
    map:
      value: ''
    settings:
      language: null
    unique:
      value: 1
  -
    target: field_date
    map:
      value: ''
    settings:
      language: null
      timezone: UTC

I never get:

    unique:
      value: '0'

Like you got.

It is still however worth it to add a fix for when "unique" has a value of zero. I just don't know how that could happen. There is a chance that there is another bug, probably in the UI, that is causing it.

megachriz’s picture

Status: Postponed (maintainer needs more info) » Needs review

I've added a test and a possible fix for the issue.

beauregard’s picture

thanks for your support!

megachriz’s picture

Status: Needs review » Fixed

I've slightly changed the fix, because I think it is unnecessary to check the unique flags on mappings every time getMappings() gets called - also because I think it's very uncommon to occur (only happens when manually editing the config file). But resaving the feed type would be enough for the fix to have effect.

I've scheduled merging the code!

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.

  • megachriz committed 4e25ba0d on 8.x-3.x
    fix: #3569983 On mappings, treat 'unique' flags set to zero as disabled...

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.