It seems there is a lot of work going on to improve Feeds with Entity integration, and to get Data completely supported by Feeds.

After these patches are committed in both Feeds and Data, the mini-module I wrote (attached) should be added to Data (you can just pull out the two functions and rename them). With that, Data will support unique field mapping with Feeds and allow for records to be updated/replaced when a unique or primary key is present in the imported data. As it stands now, you will get SQL errors for duplicate keys if the item being imported is an existing primary key; if it's not, you'll just get a new record instead of an update.

Here are the patches pending a commit. I've tested them with Feeds and Data and they seem to be working for me:

- Feeds: #661606: Support unique targets in mappers
- Feeds: #1033202: [Meta] Generic entity processor
- Data: #1671380: Data Entity allows entities to be created with entity ids that are not integers
- Data: #1538588: Add Entity Label field (Was: allow entity ref fields to point to data items)

CommentFileSizeAuthor
#8 2174373.8.data_.feeds-unique-field.patch2.51 KBjoachim
#6 2174373-6--feeds_unique_fields.patch2.14 KBdrunken monkey
#2 feeds_data_unique.tar_.gz973 bytesmstef
feeds_data_unique.tar_.gz951 bytesmstef

Comments

joachim’s picture

joachim commented

> With that, Data will support unique field mapping with Feeds and allow for records to be updated/replaced when a unique or primary key is present in the imported data.

I'm surprised Feeds needs that much work for this functionality. What happens when you use Feeds to import to a more 'regular' entity type such as nodes?

It looks like the 'unique_callbacks' property you're using is added by #661606: Support unique targets in mappers.

So this would be fine in Data module itself -- once that patch has been committed to Feeds.

mstef’s picture

mstef commented
StatusFileSize
new feeds_data_unique.tar_.gz973 bytes

Here's a slight update. This will also allow any Data field mapped in Feeds to be set as "unique".

mstef’s picture

mstef commented

I'm not entirely sure about regular entities (ie, nodes). I believe there's a module (Field validation) that people are using along with Feeds to enforce things like unique field values per import. The issue (#661606: Support unique targets in mappers) contains all of that information.

mstef’s picture

mstef commented

Just to document, as it stands now, this requires that the Data table used with feeds has a numeric primary key (limitation of Feeds).

anybody’s picture

anybody
Primary language German
Location Porta Westfalica
commented

Thank you for your quick help! The module works great!

drunken monkey’s picture

drunken monkey
he/him
Primary language German
Location Vienna, Austria
commented
Status: Active » Needs review
StatusFileSize
new 2174373-6--feeds_unique_fields.patch2.14 KB

Since all the issues blocking this have been resolved, I guess this could now be committed?
Patch attached, containing more or less a 1:1 transfer of mstef's module.

joachim’s picture

joachim commented
Assigned: Unassigned » joachim
Status: Needs review » Needs work 
+++ b/data.module
@@ -494,6 +494,67 @@ function data_date_views_fields($field) {
+/**
+ * Feeds unique callback for Data table field targets.
+ *
+ * @param string $target
+ *   The unique field name, containing the value to be checked.
+ * @param int|string $value
+ *   The unique field value to be checked.
+ * @param string $entity_type
+ *   Entity type for the entity to be processed.
+ * @param string $bundle_name
+ *   Bundle name for the entity to be processed.
+ *
+ * @return int
+ *   Feeds processor existing entity ID.
+ *
+ * @see data_feeds_processor_targets_alter().
+ * @see FeedsProcessor::existingEntityId()
+ */
+function data_feed_unique_callback($target, $value, $entity_type, $bundle_name) {

This does not match the function signature of the unique_callbacks callback as defined in feeds.api.php.

Assigning this to myself to work on it.

joachim’s picture

joachim commented
Status: Needs work » Needs review
StatusFileSize
new 2174373.8.data_.feeds-unique-field.patch2.51 KB

Getting some paid time to work on this issue :D

Here's an updated patch. The hook implementation and its associated callback really belong in data_entity module, as without that there's no data table-based entity types.

  • joachim committed 8bd2670 on 7.x-1.x authored by mstef 
    Issue #2174373 by mstef, drunken monkey, joachim: Added support for...
joachim’s picture

joachim commented
Status: Needs review » Fixed

Committed a slightly tweaked version of the above -- just docs changes now that the Feeds documentation issue #2397219: docs for my_module_mapper_unique() don't match implementation in tests has been fixed.

Thanks to everyone for working on this, and thank you to my sponsor for the paid time!

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.