Problem/Motivation
We created a feed to import an xml file. This works fine with a single import.
However, as soon we import more files or try to import with scheduled cron, we get in 75% of cases the error "entity already exists ", even it does not exist
We tested many workarounds or "fixes" all day:
- removing our title field (which is the id) to be unique
- adding GUID to mapping
- allow updates
- updating parsing
- deleting everything
What are we doing wrong? With thousands of installations I cant imagine that this is not working, so we are obvisiouly something doing wrong.
Any help is appreciated!
Steps to reproduce
We tested on 2 websites, we deleted the module and installed new, no change
Proposed resolution
Remaining tasks
User interface changes
API changes
Data model changes
| Comment | File | Size | Author |
|---|---|---|---|
| #13 | mappings-title-unchecked.png | 288.34 KB | megachriz |
| #13 | mappings-title-checked.png | 287.78 KB | megachriz |
| #12 | feed camt53_observe.txt | 3.56 KB | beauregard |
| #11 | core.entity_form_display.node_.camt53.default.yml | 4.41 KB | beauregard |
| #8 | feeds mapping screenshot.pdf | 68.32 KB | beauregard |
Issue fork feeds-3569983
Show commands
Start within a Git clone of the project using the version control instructions.
Or, if you do not have SSH keys set up on git.drupalcode.org:
Comments
Comment #2
megachrizI've not seen an issue like this before. Is "entity already exists" the exact error message?
If you enable the Feeds Log module, does that reveal more information? With that module, after an import, you can see the exact data that Feeds receives and how the data looked like after parsing. Or did you already try that and is that where you got the "entity already exists" message from?
Comment #3
beauregard commentedyes, we have log enabled, here what I get
I cant paste images here, dont know why drupal.org has such an old editor instead ckeditor5.
Please find the PDF enclosed with the error message and the existing records
As you can see 3 records are not imported with "skipped", but it is not true, that the record exists. You can see the value in "Item ID" in the Log does not match any of the 2 values in all existing records
Comment #4
megachrizAre the ID's of the two nodes 1904 and 1905? In the column "Entität" I see that are the node ID's that Feeds found when looking for existing entities (I see it only reports the ID instead of a link to the entity, probably because of avoiding loading the whole node).
The most likely cause is a misconfiguration on the mapping settings. Which target(s) do you have marked as unique? Have you perhaps set more than one target as unique? Feeds loops through all targets set as unique and tries to find an existing entity based on each individually. It does not look for existing entities based on a combination of targets. That behavior is perhaps a bit confusing. I'm not sure if that behavior was intentionally or not (it already worked that way before I took over the development of Feeds).
Comment #5
beauregard commentedcorrect, the two nodes are 1904 and 1905
No, i have marked exactly one field as "Unique". This is the title field of the node type we are filling
And we have these options
"Insert new content items"
"Do not update existing content items"
Comment #6
megachrizAlright, can you share your feed type configuration + a sample source file with which the issue can be reproduced?
Comment #7
beauregard commentedI investigated now the problem with claude code (I provided on my local hd the feeds module, our custom module, feeds importer configuration and node type configurations). claude code analyzed first the files, then created a debug version and we debugged.
In the end the following problem was identified:
Bug: When a mapping has unique: value: '0' (unchecked in UI), the existingEntityId() method still processes it as a unique constraint because the code iterates over the array without checking if the value is truthy.
File: feeds/src/Feeds/Processor/EntityProcessorBase.php
Fix: Add if (empty($true)) { continue; } inside the foreach ($mapping['unique'] as $key => $true) loop at ~line 1259.
-----------------
Claude first assumed that I have several fields checked as "unique", but I dont have it, we checked it before. However in mapping configuration, for exactly three fields there is *no* checkbox "Unique", and this seems to be the problem.
See enclosed the screenshot of Admin UI in drupal to map fields, and the modified file as done by claude.
Could you please check it?
Note: I am not developer (this was 25 years ago), I am project manager and working with drupal 15+ years. Therefore I cant say if this is the right solution. I can only say that after this fix import is working
Comment #8
beauregard commentedhere the screenshot of the Admin UI
Comment #9
megachrizOkay, that is strange. When I add mapping to a field that cannot be marked as unique, I don't get the "unique" key set. And for mappings where "unique" is not checked, I get
"unique" => []. So I wonder how something like"unique" => ["value"=> 0]could happen.That being said, your fix seems reasonable. Just want to know if there is perhaps another bug in the UI that got you to this unusual state.
Comment #10
megachrizCan you share your feed type configuration as well? (yml file) Maybe that helps to discover a bug in the UI as well.
Comment #11
beauregard commentedWhat you mean with "When I add mapping to a field that cannot be marked as unique"?
In my node type definition only title field is unique and a reference to a customer record. nothing else
But in feeds mapping page all fields have the checkbox "unique" offered, except the 3 mentioned fields
See enclosed the nodetype definition, maybe this helps
Comment #12
beauregard commentedand here the feeds type definition
Comment #13
megachrizWhen I have the following mapping configuration:

I get:
And with the following:

I get:
I never get:
Like you got.
It is still however worth it to add a fix for when "unique" has a value of zero. I just don't know how that could happen. There is a chance that there is another bug, probably in the UI, that is causing it.
Comment #15
megachrizI've added a test and a possible fix for the issue.
Comment #16
beauregard commentedthanks for your support!
Comment #17
megachrizI've slightly changed the fix, because I think it is unnecessary to check the unique flags on mappings every time
getMappings()gets called - also because I think it's very uncommon to occur (only happens when manually editing the config file). But resaving the feed type would be enough for the fix to have effect.I've scheduled merging the code!