For an example on how to use this feature, see the documentation.

Problem/Motivation

Currently you can choose any field as a unique target using the GUID mapping element. This method has the following limitations:
- this GUID is specific for each importer, so you cannot use multiple importers to update the same node.
- this GUID is stored in its own table, so nodes that are created from another source than Feeds, cannot be updated via Feeds.
Currently there is a hook to allow specifying any field as a unique target (GUID) when mapping is to implement, using hook_feeds_processor_targets_alter(). However, there is no hook allowing implementing modules to provide an existing entity ID for the specified unique field during Feeds import.

Proposed resolution

Add a new hook: hook_feeds_existing_entity_id()
[2013-07-25] Comments #111 and #127 contains the latest patches on Feeds and Field Validation modules.

User interface changes

None. The mapping UI will still respect unique target selection based on implementations of hook_feeds_processor_targets_alter(). However, this new hook will allow modules who specify that unique target to now ensure they import properly (do not duplicate).

API changes

Since Feeds should not be responsible for enforcing unique field values, it doesn't care how that gets accomplished. Any module (whether Field validation (general entity support), Unique Fields (supports nodes only), or other validation modules (contrib or custom) can accomplish this by implementing two hooks:

  1. hook_feeds_processor_targets_alter() for specifying which field it is validating as unique, and add the unique target option for selection on the mapper form (the Field validation patch from #87 is a working example for validating using that module)
  2. hook_feeds_existing_entity_id() for providing the existing entity ID for the specified unique field during import (that patch also contains an example implementation)

Original report by lunk_rat

Is it possible to use any CCK field as a unique target? I have a CCK filed that is unique to each node and I would like to update existing nodes based on a unique source field in a CSV file, using my CCK field as a unique target.

I have been trying to figure this out but no luck yet.

Thanks!

CommentFileSizeAuthor
#221 feeds-unique-target-661606-221.patch2.29 KByatendrasingh121
#220 feeds-unique-target-661606-220.patch1.79 KBmarcolz
#217 feeds-unique-target-661606-216.patch2.3 KByatendrasingh121
#197 feeds_unique_target.patch830 bytesmanojbisht_drupal
PASSED: [[SimpleTest]]: [MySQL] 5,930 pass(es). View
#196 feeds_unique_target.patch938 bytesmanojbisht_drupal
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds_unique_target.patch. Unable to apply patch. See the log in the details link for more information. View
#195 field_validation-feeds-unique-target-661606-95-do-not-test.patch3.65 KBMegaChriz
#195 interdiff-661606-186-195.txt3.16 KBMegaChriz
#195 feeds-unique-target-661606-195.patch9.67 KBMegaChriz
PASSED: [[SimpleTest]]: [MySQL] 6,041 pass(es). View
#186 feeds-unique-target-661606-186.patch9.69 KBtwistor
PASSED: [[SimpleTest]]: [MySQL] 5,632 pass(es). View
#186 interdiff.txt3.34 KBtwistor
#184 feeds-unique-target-661606-184.patch8.73 KBtwistor
FAILED: [[SimpleTest]]: [MySQL] 5,631 pass(es), 3 fail(s), and 0 exception(s). View
#184 interdiff.txt2.83 KBtwistor
#183 feeds-unique-target-661606-183.patch8.58 KBtwistor
PASSED: [[SimpleTest]]: [MySQL] 5,632 pass(es). View
#182 feeds-unique-target-661606-182.patch8.49 KBtwistor
PASSED: [[SimpleTest]]: [MySQL] 5,632 pass(es). View
#178 feeds-unique-target-661606-178.patch7.94 KBhairqles
FAILED: [[SimpleTest]]: [MySQL] 5,181 pass(es), 0 fail(s), and 367 exception(s). View
#175 interdiff-661606-171-175.txt3.4 KBMegaChriz
#175 feeds.unique-target-661606-175.patch7.3 KBMegaChriz
PASSED: [[SimpleTest]]: [MySQL] 5,183 pass(es). View
#171 interdiff-661606-170-171.txt442 bytesMegaChriz
#171 feeds.unique-target-661606-171.patch7.28 KBMegaChriz
FAILED: [[SimpleTest]]: [MySQL] 5,178 pass(es), 3 fail(s), and 0 exception(s). View
#170 interdiff-661606-167-170.txt6.68 KBMegaChriz
#170 feeds.unique-target-661606-170.patch6.85 KBMegaChriz
PASSED: [[SimpleTest]]: [MySQL] 5,080 pass(es). View
#167 feeds-unique-target-661606-167.patch965 bytesbradjones1
PASSED: [[SimpleTest]]: [MySQL] 5,080 pass(es). View
#155 entity_skip_new.inc_.txt1.61 KBemilyf
#154 feeds_plugins.zip9.09 KBmarktheshark
#131 feeds-661606-131.png12.47 KBpresleyd
#127 feeds-unique-target-661606-127.patch1.04 KBagileadam
PASSED: [[SimpleTest]]: [MySQL] 3,915 pass(es). View
#115 feeds-unique-target-661606-115.patch1018 bytesg089h515r806
FAILED: [[SimpleTest]]: [MySQL] 3,919 pass(es), 0 fail(s), and 84 exception(s). View
#111 feeds-unique-target-661606-111.patch943 bytesg089h515r806
FAILED: [[SimpleTest]]: [MySQL] 4,503 pass(es), 0 fail(s), and 84 exception(s). View
#111 field_validation-feeds-unique-target-661606-111.patch2.56 KBg089h515r806
PASSED: [[SimpleTest]]: [MySQL] 3,917 pass(es). View
#109 field_validation-feeds-unique-target-661606-107.patch4.34 KBscottrigby
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch field_validation-feeds-unique-target-661606-107.patch. Unable to apply patch. See the log in the details link for more information. View
#107 feeds-unique-target-661606-107.patch1.05 KBscottrigby
FAILED: [[SimpleTest]]: [MySQL] 3,915 pass(es), 0 fail(s), and 42 exception(s). View
#103 field_validation-feeds-unique-target-661606-103.patch3.86 KBtmsimont
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch field_validation-feeds-unique-target-661606-103.patch. Unable to apply patch. See the log in the details link for more information. View
#87 feeds-unique-target-661606-87.patch787 bytesscottrigby
PASSED: [[SimpleTest]]: [MySQL] 3,917 pass(es). View
#87 field_validation-feeds-unique-target-661606-87.patch3.85 KBscottrigby
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch field_validation-feeds-unique-target-661606-87.patch. Unable to apply patch. See the log in the details link for more information. View
#77 unique_field_targets-661606-77.patch1.35 KBCottser
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch unique_field_targets-661606-77.patch. Unable to apply patch. See the log in the details link for more information. View
#77 unique_field-feeds_integration.patch589 bytesCottser
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch unique_field-feeds_integration.patch. Unable to apply patch. See the log in the details link for more information. View
#75 unique_field_targets-661606-75.patch2.23 KBCottser
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch unique_field_targets-661606-75.patch. Unable to apply patch. See the log in the details link for more information. View
#75 interdiff.txt646 bytesCottser
#74 unique_field_targets-661606-74.patch2.27 KBCottser
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch unique_field_targets-661606-74.patch. Unable to apply patch. See the log in the details link for more information. View
#64 unique_field_targets-661606-64.patch2.26 KBOLD ACCOUNT USE ID 169175 INSTEAD
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch unique_field_targets-661606-64.patch. Unable to apply patch. See the log in the details link for more information. View
#58 unique_field_targets-661606-58.patch2.45 KBselim13
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch unique_field_targets-661606-58.patch. Unable to apply patch. See the log in the details link for more information. View
#57 unique_field_targets-661606-57.patch2.7 KBa.ross
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch unique_field_targets-661606-57.patch. Unable to apply patch. See the log in the details link for more information. View
#29 unique_field_targets-661606-29.patch2.27 KBMohammed J. Razem
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch unique_field_targets-661606-29.patch. Unable to apply patch. See the log in the details link for more information. View
#28 unique_field_targets-661606-28.patch2.26 KBMohammed J. Razem
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch unique_field_targets-661606-28.patch. Unable to apply patch. See the log in the details link for more information. View
#22 unique_field_targets-661606-22.patch2.23 KBchromix
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch unique_field_targets-661606-22.patch. Unable to apply patch. See the log in the details link for more information. View
Members fund testing for the Drupal project. Drupal Association Learn more

Comments

alex_b’s picture

Title: Custom unique target? » Support unique targets in mappers

This is not possible in the current implementation. Could be added though.

For simple text/number_integer/number_decimal fields (mapped with mappers/content.inc) this would mean:

1) Declare mapping targets unique in content_feeds_node_processor_targets_alter()


  foreach ($fields as $k => $name) {
    $targets[$k] = array(
      'name' => $name,
      'callback' => 'content_feeds_set_target',
      'description' => t('The CCK !name field of the node.', array('!name' => $name)),
      'optional_unique' => TRUE,
    );
  }

2) Add a new hook that is invoked in FeedsNodeProcessor::existingItemId() in the switch statement's 'default' state. The hook would ask all implementers for an existing $nid for a given $source_item / $source combination (in analogy to FeedsNodeProcessor::existingItemId()).

hook_feeds_node_processor_unique($source_item, FeedsSource $source);

3) Implement this hook in content.inc for all fields.

tomcatuk’s picture

Actually, I'm already doing pretty much exactly what linkovitch is asking. One (possibly) important note - all the nodes I'm updating were initially created from the .csv using the importer. I'd be amazed if this worked on nodes NOT created by feeds.

I've got a "unique" CCK field, and in my .csv it's got a column identifier of course so I can use mappings. First I mapped this unique field to the CCK field I want it to go into (so far so obvious). I then mapped the SAME field from the .csv to "GUID" and ticked the "unique target" checkbox. The field values aren't actually GUIDs of course.

I've got a bad feeling Alex will say this is the wrong thing to do, but I can say it's working for me - I've got 9000 nodes in that table and uploading a modified .csv updates the existing nodes, doesn't create duplicates.

TamboWeb’s picture

I am at a complete loss here. From what I understand ....

From #1 above,

part 1.-

<?php
  foreach ($fields as $k => $name) {
    $targets[$k] = array(
      'name' => $name,
      'callback' => 'content_feeds_set_target',
      'description' => t('The CCK !name field of the node.', array('!name' => $name)),
      'optional_unique' => TRUE,
    );
  }
?>

It adds the checkbox for unique target to the mapped fields. So far so good.

Now Part 2. Add a new hook to check if node already exists; This code checks for nid and guid.

<?php
    // Iterate through all unique targets and test whether they do already
    // exist in the database.
    foreach ($this->uniqueTargets($source_item) as $target => $value) {
      switch ($target) {
        case 'url':
          $nid = db_result(db_query("SELECT nid FROM {feeds_node_item} WHERE feed_nid = %d AND id = '%s' AND url = '%s'", $source->feed_nid, $source->id, $value));
          break;
        case 'guid':
          $nid = db_result(db_query("SELECT nid FROM {feeds_node_item} WHERE feed_nid = %d AND id = '%s' AND guid = '%s'", $source->feed_nid, $source->id, $value));
          break;
      }
      if ($nid) {
        // Return with the first nid found.
        return $nid;
      }
    }
    return 0;
?>

So to check mytargetid, I would need to add to the code above another case branch to check for mytargetid.

<?php
       case 'mytargetid':
          $nid = db_result(db_query("SELECT nid FROM {feeds_node_item} WHERE feed_nid = %d AND id = '%s' AND mytargetid = '%s'", $source->feed_nid, $source->id, $value));
          break;
?>

Where do I implement this?
Is is in the content.inc file, hard coded to the FeedsNodeProcessor.inc or somewhere else?
Or do I create the hook?, how?

Thanks.

twistor’s picture

subscribe

TamboWeb’s picture

I found a solution to my problem without the need to re-code as in #5 above.

In my application, I am using Feeds plus the XML parser for feeds. I am importing a non-standard XML file with lots of CCK fields. ( Real Estate Listings )

For some reason I overlooked the comments of XML parser readme file, my mistake.

While you are mapping the XML fields to the CCK content type, you can do the following:

  1. Assign (map) one of the xml fields to GUID.
  2. Mark this field as "unique target"

Then next time the xml feed comes in, any records that have a GUID match either get skip or get updated if there are any changes. This works for me since all properties have a unique listing number.

Furthermore, since I need the listing number in the cck content type, I was able to assign it to both the cck field and the GUID.

Hope this helps.

lunk_rat’s picture

I did exactly as you explain to solve my problem.

alex_b’s picture

#7 is a good solution and may be applicable in most cases where people are looking for a CCK field or similar as unique target.

#5 hook_feeds_node_processor_unique() needs to be invoked from the existingItemId() method FeedsNodeProcessor.inc

ccoppen’s picture

Subscribing, but going to try out the custom additions above. If it works, I'll make a patch.

I have nodes I imported from a MySQL db, but now I need to update them with info from a SQL Server db which does not have the same fields to use as unique targets.

I need this ASAP, so if someone else has figured it out, let me know. Otherwise, I'm working on it.

alex_b’s picture

Title: Support unique targets in mappers » Support unique targets in mappers / hook_feeds_node_processor_unique()

Another reason why #7 is generally a good solution is because "guid" and "url" are indexed.

If we offer e. g. CCK fields as potentially unique targets it won't perform out of the box, adding an index to those fields would be practically required - problem is, this is

a) paramount to hacking a module
b) given a) offering a UI that assumes that you will know and fix the implicit performance issue is a bad practice.

That said, there are a lot more potential 3rd party mapping targets that would be equipped with the right indexes to offer a unique flag. This means I would still accept a patch that introduces a hook_feeds_node_processor_unique(). However, I won't accept a patch using this new functionality for CCK mappers or similar 3rd party mappers in Feeds that don't have the proper indexes to function as a unique mapping target, in these cases #7 is the proper workaround.

alex_b’s picture

Component: User interface » Code
smscotten’s picture

Here's what I don't get: what is the purpose of having a unique target that isn't in a nodetype? Unless I am totally misunderstanding this, it only forces uniqueness in the set of new nodes being created, not uniqueness among the set of all nodes of that type.

alex_b’s picture

it only forces uniqueness in the set of new nodes being created, not uniqueness among the set of all nodes of that type.

It forces uniqueness among all nodes imported by the given importer from a given source. This ticket is about allowing node properties exposed by mappers to be unique targets - e. g. with this feature you could ensure that specific values on a CCK field can only occur once in a set of imported nodes.

smscotten’s picture

I'm sorry Alex, but I don't understand your comment. I don't believe I was hijacking the thread but explaining why the feature is important.

But perhaps I'm misunderstanding the scope of the feature entirely.

Does a unique target only constrain the data coming in during a specific import? That is to say, is there any facility in Feeds to check to see that the unique target is not only unique in the import but that it will also be unique in the list of nodes of that type?

I'm wondering if this feature means what I thought it did: that it can be used to differentiate between "create new" and "update existing" when bringing new node data in.

From "ensure that specific values on a CCK field can only occur once in a set of imported nodes" (emphasis obviously mine) I'm suspecting that there is no checking of existing nodes.

So if I have a CSV file with two rows, and GUID for each one is "XYZ123" and the import runs every day (even though the CSV file doesn't change), then every day I will get one new node, and at the end of the week I'll have seven nodes with XYZ123 (If I set GUID both to GUID and to another field).

What I am looking for is a way to make there be only one node with whatever field set to XYZ123 no matter how many times the feed importer runs.

If the module's design indicates I would end up with one row, then the ability to use a CCK field for a unique target is critical. If I would end up with seven I don't see how making CCK fields into unique targets is even remotely useful.

(If after a week I'd have fourteen XYZ123 nodes then I've completely misunderstood everything about this module.)

If you or someone would indicate which of these descriptions fits the unique target behavior, I'd appreciate it and be better able to participate in this discussion in a more constructive manner. Thank you!

EvanDonovan’s picture

[A unique target] forces uniqueness among all nodes imported by the given importer from a given source.

Does this mean that a unique target isn't actually checked for uniqueness between multiple source feeds, but only ensures that the items from a single feed are unique? I need something that is unique across all nodes of a given type, not just those from a particular source.

My apologies if I'm misunderstanding or further leading this thread off topic.

It looks like from the code in the existingItemId method of FeedsNodeProcessor that the query looks at feed_nid as well as id and either guid and url. That suggests to me then that there is the possibility of duplicates being created across separate feeds.

EvanDonovan’s picture

Sorry for derailing this issue - I decided that #998426: Unique targets ignored unless nodes have same parent feed node was a better place to take the discussion from #13 on. This issue can return back to its original purpose - creating a hook for defining new unique targets. Theoretically, the ideal solution for the issue I describe in #998426: Unique targets ignored unless nodes have same parent feed node might require the hook described in this issue though.

EvanDonovan’s picture

To answer the question in #5, I think what Alex is suggesting is something like:

2)

switch ($target) {
        case 'url':
          $nid = db_result(db_query("SELECT nid FROM {feeds_node_item} WHERE feed_nid = %d AND id = '%s' AND url = '%s'", $source->feed_nid, $source->id, $value));
          break;
        case 'guid':
          $nid = db_result(db_query("SELECT nid FROM {feeds_node_item} WHERE feed_nid = %d AND id = '%s' AND guid = '%s'", $source->feed_nid, $source->id, $value));
          break;
        default:
          // changed the parameters of the hook from what Alex suggested - first FeedsSource, then the name of the target, 
          // then the value of the target
          $nid = module_invoke('feeds_processor_unique_value', $source, $target, $value);
          break;
      }

3) And then in your module, implement the hook with something like:


CUSTOM_feeds_processor_unique_value($source, $target, $value) {
  if($target = 'test_field') {
    $nid = db_result(db_query("SELECT fni.nid FROM {feeds_node_item} fni JOIN {test_table} t ON t.nid = fni.nid 
                           WHERE fni.feed_nid = %d AND fni.id = '%s' AND t.test_field = '%s'", 
                           $source->feed_nid, $source->id, $value));
  }
  return $nid;
}

Obviously for CCK, the queries would be more complex, and you would want to use a switch statement.

Does that make sense? Does the idea of a hook_feeds_processor_unique_value() sound like something that could get in to the actual module? If so, that would mean that I wouldn't have to do what I do in #998426-9: Unique targets ignored unless nodes have same parent feed node as a hack.

EvanDonovan’s picture

Status: Active » Needs review

Setting this to "needs review" so that the proposed hook could be reviewed, prior to me writing an actual patch.

EvanDonovan’s picture

I think that I am going to need to follow the approach in my comment #18 for the project I am currently working on, since I have a case in which the feed doesn't have unique URLs. Rather the uniqueness is determined by a combination of title, description, and a custom tag called fp:provider.

I'm thinking that I will have to do the following:

1) Define a new source field that is an md5 hash of the title + description + fp:provider tag.
2) Define a new target field that is unique, and which is stored in its own database table, on which the foreign key is the nid (uniqueness between feeds is not an issue; I want this to be global).
3) Change the code of the FeedsNodeProcessor to have a hook_feeds_processor_unique_value().
4) Implement that hook to query the table into which my hashes will get stored.

Does this sound like a good strategy? Is this the only way to accomplish what I would like to do, or is there an easier way?

EvanDonovan’s picture

Actually, I think I really only need to do #1, and then set that as the GUID field. I'll see if that works.

chromix’s picture

FileSize
2.23 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch unique_field_targets-661606-22.patch. Unable to apply patch. See the log in the details link for more information. View

I decided to take this on using the Unique Field module, since it takes care of the uniqueness validation out of the box. My patch works directly with the module's settings variables to figure out which CCK fields should be used as unique target fields. For the sake of simplicity, this patch will only work in cases where the node has a single unique field and not multiple unique fields working in combination. Either way, thanks to everyone else who contributed code to this ticket. Enjoy!

msimanga’s picture

Thank you #22 chromix, worked like a charm. I vote for inclusion in the next release of feeds.

itai_biren’s picture

Version: 6.x-1.x-dev » 7.x-1.x-dev

Hi,
I'm using drupal 7 and try to patch this file (#22) but i cant
i get an error:
can't find file to patch at line 5

EvanDonovan’s picture

Version: 7.x-1.x-dev » 6.x-1.x-dev

I believe all the code in this issue was against 6.x. I'm not sure if there are substantial differences in the architecture of 7.x. I think we should keep this against 6.x, for now, since patches are still getting committed to that branch. If you wanted to re-roll for 7.x, though, that would be cool, though, I'm sure.

lhugg’s picture

Reply to #22:

In my testing, the Unique Field module fails if any of the original nodes created by feeds is deleted. In our test case, nine node were imported, and used a unique path to the original source as the unique field. Additional runs of feeds did not re-add the nodes. BUT, if just a single one of the nine nodes was deleted, then the entire nine were re-created in the next run.

BTW, we are using Drupal 6 with Feeds 6.x-1.0-beta11, and the included Node Processor.

I don't understand why Feeds doesn't directly address duplicates itself. It's seems like such an essential function.

Niklas Fiekas’s picture

Subscribe.

Mohammed J. Razem’s picture

Version: 6.x-1.x-dev » 7.x-2.x-dev
FileSize
2.26 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch unique_field_targets-661606-28.patch. Unable to apply patch. See the log in the details link for more information. View

Patched against 7.x-2.x

There you go.

Mohammed J. Razem’s picture

FileSize
2.27 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch unique_field_targets-661606-29.patch. Unable to apply patch. See the log in the details link for more information. View

Sorry.. Forgot to rename a variable. Here's the correct patch.

Niklas Fiekas’s picture

Intresting ... integrate with Unique field module ...

How would I decide for my custom field type, if the value is unique? I'd still like something like hook_feeds_node_processor_unique() or another hook to do that. Maybe Unique field could then implement it rather than doing "module_exists" in Feeds?

Lirkun’s picture

Tnx for great patch chromix and Mohammed J. Razem also =)

I used it for D 7.7 and it works great.

Rowan Price’s picture

Thank you so much, Chromix! And everyone else for their ideas here.

Once I applied the patch against the feeds module instead of unique field it worked like a charm :0)

dsnoeck’s picture

Thanks, this patch works also well for me.

amacias’s picture

Thanks, the patch works perfectly if you choose as your unique field a cck field of the node, but not if you select the title. Is there any quick fix in the patch so title can also be chosen?

chromix’s picture

The best quick-fix I could think of would be creating a required "Title" CCK field, and then have the Auto Nodetitle module use its value to set the real node title. Otherwise, it would obviously be better for Unique Field to work on node titles, but that's an issue for that module.

amacias’s picture

Thanks chromix, that worked perfectly!

rylowry@gmail.com’s picture

I installed the Unique Field module and the patch in #29. It worked perfectly for me. I'm running Drupal 7.9 and Feeds 7.x-2.0-alpha4. I needed to have 2 different feeds updating the same nodes, and this did the trick.

itai_biren’s picture

Hi,
I'm using drupal 7 and try to patch this file (#29) but i cant
i get an error:
can't find file to patch at line 5

skyoo8’s picture

Hi, I have installed Unique Field module and the patch in #29. But I can not import new node anymore, its shows "There are no new nodes." If I disable Unique Field, its works fine again.

Also, If I choose Update existing nodes instead of Do not update existing nodes, my data got messed up, feeds update the wrong node.

Need some help please.

johnv’s picture

Patch #29 applies correctly and works fine with latest feeds and unique_field.

@itai_biren, #28, are you sure you are in the right directory when applying the patch?
@skyoo8 #29, are you still having problems? we need some more details to be able to help you.

Lasac’s picture

@chromix: i cant get your patch working. What version of unique field did you use? I want unique titles. So i first tried setting the title to be unique, since unique fields supports that. Didn't work.

Then i tried your suggestion to create a cck, title copy field, and set the unique check on that. Tried it. Didn't work.

Lasac’s picture

Ah, i see we have a extra unique checkbox in the mapping page. Great, works as expected! Awesome. Let's work it out more, like multiple unique fields, and the unique title options. And then implement it in the feeds module. I love that this fix uses another module. This is how modules should work together. chromix++

sassafrass’s picture

I installed the patch and the module Unique Field. And this works if I select one field as a unique target but not if I select multiple fields. Is that as intended?

chromix’s picture

From comment #22 above:

For the sake of simplicity, this patch will only work in cases where the node has a single unique field and not multiple unique fields working in combination.

It's definitely possible to use multiple fields, but it adds a decent amount of complexity.

jamesdixon’s picture

I can confirm #21 works to get unique target working for multiple columns. Here's my implementation as an example. I'm using hook_feeds_after_parse inside a custom module. This is a quick and dirty solution:

<?php
// the columns that will form the unique target
define('MASTER_PARTNUM', 'part number');
define('PART_TYPE', 'part desc'); 
define('YEAR', 'year');
define('MAKE', 'make');
define('MODEL', 'model');
define('SUB_MODEL', 'sub_model'); 
define('LITER', 'liter'); 
// the new GUID is a result of the combined above fields transformed into
// an md5 hash
define('APP_GUID', 'guid');

function import_app_feeds_after_parse(FeedsImporter $importer, FeedsSource $source) {
   // my feed was using the CSV Parser, this if statement could be removed
   if (get_class($source->importer->parser) == 'FeedsCSVParser') {

    foreach ($source->batch->items as $item_key => &$item) {
      // can uncomment below line to find out column indexes in $item
      //watchdog('debug', var_export($item, true));
      // make sure we're dealing with an item that has all the required fields
      if (!empty($item[MASTER_PARTNUM]) && !empty($item[PART_TYPE]) &&
          !empty($item[YEAR]) && !empty($item[MAKE]) &&
          !empty($item[MODEL]) && !empty($item[LITER])) {
          // combine the unique target columns into one field
          $unique_app_combo = $item[MASTER_PARTNUM] . $item[PART_TYPE] .
            $item[YEAR] . $item[MAKE] . $item[MODEL] .
            $item[SUB_MODEL] . $item[LITER];
          // create the new guid source
          $item[APP_GUID] = md5($unique_app_combo);
      }
    }
  }
}
?>

Afterwards you'd need to add a mapping from source 'guid' to GUID, and click the unique target checkbox.

Hopefully this helps someone.

ambientdrup’s picture

I've installed this patch and the Unique Fields module but on my Feeds importer Node processor mapping I still only see one unique column with a checkbox next to the GUID field. That existed before the patch. So this patch does not seem to be working for me.

-Trevor

macdee’s picture

The patch at #29 worked well for me.

@ambientdrup Did you "Choose the fields that should be unique" under "Unique Field restrictions" in admin/structure/types/manage/--your content type--?

ambientdrup’s picture

Correct. Edit your content type, click on the Unique Field restrictions tab and then choose the fields via the checkboxes.

Best-

Trevor

Masala’s picture

Tell me understand please. To properly import I need to have a unique field 'Ubercart:' Model / SKU. In the standard settings do not checkmark the "unique" for SKU. This patch will help me? I installed #29 but did not see the changes.
I have no SKU fields in "Unique Field restrictions" in admin/structure/types/manage/product

ayalon’s picture

The path in #22 has a bug:


        default:
          if (module_exists('unique_field')) {
            $scope = variable_get('unique_field_scope_'. $content_type, UNIQUE_FIELD_SCOPE_TYPE);
            $nid = array_shift(unique_field_match_value($target, array(array('value' => $value)), variable_get('unique_field_scope_'. $target, $scope)));
          }

The variable $content_type is not defined in this context. It should be $target.

rond49’s picture

Can't one delete his/her own comment??

OLD ACCOUNT USE ID 169175 INSTEAD’s picture

The patch in #29 worked for me, thank you!

I'm not sure if it's related, but I had to change my Title and Description source mappings to xpath expressions after installing the Unique Field module and installing this patch, title and description were no longer options in the Source drop-down. It still works, I just had to make that change. Either way, it's working and I'm ecstatic. :)

lunk_rat’s picture

Did this get committed to 7.x dev? I see this option now and I didn't patch anything.

lunk_rat’s picture

Ignore #53; I was just seeing it on the Title field.

a.ross’s picture

It hasn't been committed yet. The title field can now be selected as a unique field, regardless of the unique_field settings.

a.ross’s picture

Status: Needs review » Needs work

I've manually applied the patch of #29, and it appears to still be working just fine. Not sure if that could collide with the Title field optionally being unique before this patch though.

a.ross’s picture

Status: Needs work » Needs review
FileSize
2.7 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch unique_field_targets-661606-57.patch. Unable to apply patch. See the log in the details link for more information. View

Attaching properly formatted patch rolled against the current revision.

selim13’s picture

FileSize
2.45 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch unique_field_targets-661606-58.patch. Unable to apply patch. See the log in the details link for more information. View

Thanks for the patch a.ross. I've tried it with 7.x-2.0-alpha4 and had pretty interesting results. Only one or part of nodes created on import, when unique_field module was enabled, even when no fields were marked as unique. The reason for me was, that unique_field_match_value returned aleady created node id, when GUID was provided as $target. So it just continuously updated the first one created node or I think so.
I've made realy quick fix that ensures that only field name taken from unique_field module will pass to unique_field_match_value.
I'm not sure if this is correct way, but it works for me.
Attaching patch (git diff) for 7.x-2.0-alpha4 but it should apply to current dev.

Sorry for my English =)

a.ross’s picture

You generally shouldn't really map anything to GUID, because it is system managed. Also, are you sure you didn't forget to set the field "unique" in the mapper itself?

Anyway, I didn't have the issues you describe. In my case it was working just fine.

That said, I don't think your patch might break things, but it's probably redundant code.

litvinova_yana’s picture

Comment #7:
Super! A good decision.

johnv’s picture

@litvinova_yana,
IMO comment #7-#11 describes the 'out-of-the-box' solution, provided by feeds.
However, it only works if the feeds is the only source of the objects. It won't work if you create an object manually, and amend/overwrite it with the feeds, since the GUID is not yet in the GUID-table.
In that case, comments/patches #22-#29 can be used.

Jason Dean’s picture

Unique Field module plus #29 is working well for me.

Now my challenge is to get this working with the feeds mapper for node_reference (References module). Using the example provided by johnv in this issue:

- Feed 1: Create nodes of Content type A, setting the GUID/URL:
Book Title e.g. 'XML programming'
ISBN field_isbn e.g. '0-7645-3829-2'
ISBN GUID X e.g. '0-7645-3829-2

- Feed 2: Create nodes of Content type B, getting the GUID/URL of Feed 1:
Author Title e.g. 'B. Benz'
Author GUID X e.g. 'B. Benz'
Book field_book_ref (Node reference by Feeds GUID) e.g. '0-7645-3829-2

This shows how a node reference field can map to Feeds GUID in a 'normal' setup (i.e. when Unique Field module is not used).

But using Unique Field module and #29 patch, Feeds GUID is not set when Feed 1 is imported. So referencing by Feeds GUID in Feed 2 cannot be used.

I'm not sure whether this is strictly an issue for Feeds or References, but I guess references.feeds.inc would need to be modified to use the unique field defined by Unique Field module instead?

litvinova_yana’s picture

@johnv,
thanks for given explanation!

But I wrote about comment #7, because I liked logic of desicion, given in it, and it went well with my project.

Best regards.

OLD ACCOUNT USE ID 169175 INSTEAD’s picture

FileSize
2.26 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch unique_field_targets-661606-64.patch. Unable to apply patch. See the log in the details link for more information. View

I've updated the patch in #29 for 7.x-2.0-alpha5. I've tested it but this is my very first patch (following these instructions) so please be nice if it's not correct. :)

a.ross’s picture

Patches should be rolled against -dev, which I already did in #58

OLD ACCOUNT USE ID 169175 INSTEAD’s picture

Sorry, that's what I thought, but -dev doesn't show up as a branch option, I just have:

ws460:feeds lrobeson$ git branch -a
* 7.x-2.x
  remotes/origin/6.x-1.x
  remotes/origin/7.x-2.x
  remotes/origin/HEAD -> origin/master
  remotes/origin/master
a.ross’s picture

Well, the 7.x-2.x-dev is just a release name. It comes from the 7.x-2.x branch, just like all 7.x-2.x releases (like 7.x-2.1). For stable releases, you should see tags corresponding to releases somewhere in the commit log.

selim13’s picture

Reply for #59.
Hello a.ross. Forgive me for long responce.

You generally shouldn't really map anything to GUID, because it is system managed.

Feeds module was used to create nodes from XML file. XML file could be updated, so nodes need to be updated too. So GUID was mapped to unique identifier field of XML feed and was set as "unique", which worked fine.

Also, are you sure you didn't forget to set the field "unique" in the mapper itself?

I did't set it neither in content type settings nor in mapper. Just enabling unique_field module causes imports with "unique" GUID mappings to fail.

The problem is, that when GUID is mapped and marked as "unique" it passes to unique_field_match_value() which is nothing to do with guids but returns some value (in my case it is a full list of node id's). So importing enitity (node) is considered as exisiting and overrides the most earliest one.

Another possible problem is that sometimes unique mappings from other modules can pass as targets to unique_field_match_value(). This probably happens only when mappings have 'optional_unique' => TRUE, but actually not being handled for uniqueness and pass to default case in switch ($target) in existingEntityId(). As with GUID this returns full list of node id's in this case and fails import.

So there still should be some simple check, that the target passed to unique_field_match_value() is actually a field. Possibly simpler that mine, for example by checking "field_" prefix in target's name.

Again, sorry for my English =)

a.ross’s picture

Status: Needs review » Needs work

Hmm, I think I understand most of what you're saying. First of all, mapping to GUID and marking unique does work, but as far as I can remember you should avoid mapping anything to the GUID field, because Drupal normally manages it.

Now it seems that the previous patch in this issue conflicts with the "uniqueness" that is provided by Feeds natively, which is what I was afraid of. I would expect that it also conflicts with the title field (which can already be set as unique in the current revision of Feeds).
I guess then this issue needs more work.

Lastly, I don't think the check should be as simple as preg_match('/^field_.+$/', $machine_name). The reason is simple; other modules may add fields that don't start with "field_"

Robin Millette’s picture

a.ross said

but as far as I can remember you should avoid mapping anything to the GUID field, because Drupal normally manages it.

I've been using feed api and then feeds for ages and generally needed to specify guid mapping myself. Say you're scrapping an html page with xpath parser, what do you think happens with the guid? I just grepped the source and can't see any special handling of guid. Maybe you're thinking of the md5 hash used internally?

On the other hand, if you could point to anywhere explaining how "Drupal manages the GUID", I would be very grateful.

a.ross’s picture

I've read it before in posts on d.o., though I can't find it right now. However, I just looked in the database, and there is no GUID column in the node table, only in the feed table, so I guess it's specific to the feeds module.

I guess then I was wrong, so I take that back. My apologies. However, it doesn't affect this issue at all, in that the unique_fields integration collides with Feeds' native handling of unique fields.

iAugur’s picture

#64 worked for me! Thanks.
I would be keen to see this make it into the next release of the module or something along these lines that supports unique fields.

We wanted a unique field on Product Displays on a Commerce site and using GUID is OK if you re-use the same Feed Importer to update nodes. But we had one that imported all node values initially and another to update statuses. Using the updater created new nodes as the GUID is unique only in the context of the feed importer not the target.
This pathc and http://drupal.org/project/unique_field was the solution thanks.
GUID is used by Feeds to track mapping for a specific importer not across importers.

There is a similar discussion here #1233142: remove ->condition('feed_nid', $source->feed_nid) from existingEntityId().

jjclint’s picture

Status: Needs review » Needs work

The patch in #64 didn't help me.

I have two similar unattached importers I've mapped the title in each to be unique with the help of unique field module (ver. 7.x-1.0-rc1). I crawl a url with an xpath fetcher and create a node from it with one importer, I then xpath a different url that has the exact same title with the second importer, but instead of updating the node fields like it should it just creates a new node with the same title.

Cottser’s picture

FileSize
2.27 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch unique_field_targets-661606-74.patch. Unable to apply patch. See the log in the details link for more information. View

Rerolled #64 against latest dev.

Cottser’s picture

FileSize
646 bytes
2.23 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch unique_field_targets-661606-75.patch. Unable to apply patch. See the log in the details link for more information. View

This patch removes the unnecessary $info = field_info_field($name) line inside the module_exists(), since it's already set just inside the foreach.

twistor’s picture

This patch implements unique fields on behalf of the unique_field module. hook_feeds_node_processor_unique() and the 7.x alternative needs to be implemented first. Then this patch should go into the unique_field module.

Cottser’s picture

Status: Needs work » Needs review
FileSize
589 bytes
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch unique_field-feeds_integration.patch. Unable to apply patch. See the log in the details link for more information. View
1.35 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch unique_field_targets-661606-77.patch. Unable to apply patch. See the log in the details link for more information. View

I agree, this will never get committed as is. Here's an attempt at adding hook_feeds_node_processor_unique() - @twistor I wasn't sure what you meant by the 7.x alternative.

Since the proposed hook is in the early stages, I'm not going to post a patch in the unique_field issue queue yet. Attached are two separate patches, one for Feeds and one for Unique field to show an example hook implementation. On the importing side this should work almost the same as #64 or #74/75, but I've corrected the call to unique_field_match_value() to work per content type if that's how you've configured unique_field.

In this patch, the hook looks like this:

function hook_field_feeds_node_processor_unique($value, FeedsSource $source, $target, $config)

I decided to pass $this->config from FeedsNodeProcessor, otherwise the hook implementer has to use $source->importer->processor->config['content_type'] if they want to access the content type. I couldn't see the full $source object via devel dpm() either. The 'feeds_source' query in FeedsNodeProcessor::existingEntityId() uses $this->config['content_type'] so I thought it was fair, but we could certainly drop the argument as well.

Right now the Feeds patch just shows/hides "Unique" checkboxes in the Feeds mapping UI based on whether any modules implement this hook or not. I'm not sure how to improve this other than creating a separate hook. The previous patch directly on top of Feeds (#74) was able to show/hide checkboxes based on what fields were configured with unique_field.

Cottser’s picture

I probably should have named the first patch feeds-unique_field_targets-661606-77.patch to be a bit clearer. The first patch is against Feeds, second is against Unique field.

The implementation of hook_feeds_node_processor_unique() should have a docblock too, of course.

jjclint’s picture

Status: Needs work » Needs review

Applied both patches and it's still creating 2 different nodes with the exact same title when I try to update using 2 different importers.

iAugur’s picture

JJClint: Just a thought - have you checked the unique target checkbox on the mapping screen for your unique field?
So long as that is checked and the field is unique it should update not duplicate.

jjclint’s picture

@iAugur: I've ticked the check boxes for unique title on the mappers UI screen and it's also a unique field in the content screen. The only other thing I can think of is that maybe something is wrong because I'm using the feeds tamper module as well, but when I go into the sql tables for my unique fields nothing seems wrong.
I'm really banging my head hard on this one.

p.s

After I've updated feeds to the latest dev version and applied the patches, unique check boxes appeared on the mappers UI screen next to fields which I haven't setup to be unique via the unique module (content/structure/unique_field), I'm assuming it's the doing of my feeds update to latest dev version.

a.ross’s picture

You can already set the title field as unique in the latest dev. No need to apply this patch (#75) if that's all you require. Also the patch needs work, or perhaps another approach like Cottser did in #77.

jjclint’s picture

@a.ross:
A. I was using the patches in #77

B. The built-in unique field option for title that comes with feeds latest dev. version doesn't work when one uses two different importers for the same node.
Example: Importer 1 should create the node and importer 2 should update some of it's fields, instead importer 2 creates a whole new node with the same title field.

EDIT:
I've just tried to achieve the above result with a fresh install of drupal 7.x with the latest dev. version of feeds (unpatched), without unique field module, using 2 different xpath crawlers with a unique title field and failed (again).

Cottser’s picture

@jjclint - That sounds pretty straightforward and Feeds should be able to accomplish that out of the box. As @a.ross said you may want to try without any patches. Check title as unique in both your importers, and double check your node processor settings under Update existing nodes.

From FeedsNodeProcessor.inc:

case 'title':
  $nid = db_query("SELECT nid FROM {node} WHERE title = :title", array(':title' => $value))->fetchField();
jjclint’s picture

@Cottser: I've checked all of the things you've mentioned and I still can't get it to work. I'm not sure what's wrong, I can upload my test site to a live server, but not for long.

edit:
Checking the sql tables for node I've noticed that one of the nodes which I thought had the same title actually has three
tags that got xpathed into the title field.
trimming the white space around the title field fixed my problem, anyway sorry for the fuss.

scottrigby’s picture

Status: Needs review » Needs work

I think #78 needs some work.

A hook to set optional_unique in field_feeds_processor_targets_alter() makes sense, but it should probably allow the implementation to specify which field (on the specific entity_type/bundle_name) should be unique, rather than allowing any of the fields to be unique if the hook is implemented.

About an implementation patch… it's not really necessary for this patch to Feeds - but as a proof of concept, it would be better to integrate with the Field validation module, as Unique fields only supports nodes. See:
#1094582-10: Support (Entity) Fields in general, not only on nodes
#1319396: Join forces with Field Validation

scottrigby’s picture

Title: Support unique targets in mappers / hook_feeds_node_processor_unique() » Support unique targets in mappers
Status: Needs work » Needs review
FileSize
3.85 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch field_validation-feeds-unique-target-661606-87.patch. Unable to apply patch. See the log in the details link for more information. View
787 bytes
PASSED: [[SimpleTest]]: [MySQL] 3,917 pass(es). View

Ok so I took a stab at this. Two patches attached, one for Feeds and the other for Field validation.

In the Feeds patch I added a more generic hook_feeds_existing_entity_id() to FeedsProcessor::existingEntityId() for the existing ID part of this issue, which should handle any entity type rather than just nodes.

In the other attached Field validation patch, I'm implementing this new hook.

For the unique target option, I don't think we need any additional hook, since we already have hook_feeds_processor_targets_alter(). Instead I'm also implementing that hook in the Field validation module (with another helper function for that module, which simplifies these hook implementations).

I can add this patch to the Field validation queue tomorrow, unless anyone sees a problem with these changes I'm not considering?

johnv’s picture

IMO this should be in a separate field_validation.feeds.inc file.

Cottser’s picture

@scottrigby - At a glance the patch looks good. After throwing together my patch I also thought it should work for more than just nodes, so I'm glad you went that way. I ended up creating a small custom module that uses hook_node_validate(), hook_node_presave() and hook_feeds_presave() to validate and skip duplicates for node functions and Feeds imports.

Jason Dean’s picture

I've been using Unique Fields module and patch from #57 successfully ... until I tried running my feeds importer via Drush.

The patch in the Drush issue (also #57, strangely) runs the importer from Drush fine, but it doesn't respect the unique field set in the importer mapping. This results in lots of duplicates.

Looking at the code for the Drush method, it's just using feeds_batch Feeds API function:

<?php
// Set the batch operations.
  $batch = array(
    'title' => dt('Importing !feed_name', array('!feed_name' => $feed_name)),
    'operations' => array(
      array('feeds_batch', array('import', $feed_name, $feed_nid)),
    ),
  );
  batch_set($batch);
  drush_backend_batch_process();
?>

So I can't see why it doesn't respect the mapped unique field?

scottrigby’s picture

Thanks for the feedback so far!

@johnv - About the field_validation patch moving Feeds hook implementations to an include - that sounds fine, except for the field_validation_get_fields() helper function, which IMO is generally and should be in the .module file no matter what. But where it's contrib hook implementations live is probably best left up to that module maintainer's preference, so I just added them to it's .module file until told otherwise =)

Although… are you suggesting we use path discovery (like CTools plugins) when invoking feeds_existing_entity_id()? If so, AFAIK that wouldn't gain us anything here.

@Cottser - Cool thanks!

@pushka (using Unique Fields) & @Cottser (re your custom module) - Yeah the nice thing about this new Feeds hook is… since Feeds should not be responsible for enforcing unique field values, it doesn't care how that gets accomplished. Any module (whether Field validation, Unique Fields, or other validation modules (contrib or custom) can accomplish this by implementing two hooks:

  1. hook_feeds_processor_targets_alter() for specifying which field you are validating as unique, and add the unique target option for selection on the mapper form (the Field validation patch from #87 is just an example, but one that works for validating using that module)
  2. hook_feeds_existing_entity_id() for providing the existing entity ID for the specified unique field during import

How everyone following this issue can help

So the Feeds patch in #87 is what mainly needs review now. The main differences between that (ultimately, a patch to add hook_feeds_existing_entity_id()) and the Feeds patch in #77 are:

  • Moved the hook invocation (adds more general entity suport, including but not limited to nodes)
  • Changed the hook name (for clarity)
  • The hook invocation now passes $entity_type, because (unlike the node-specific version in #77) $config will not always contain the entity type (though I kept $config anyway, since it may be generally useful for other implementations)
rbosscher’s picture

scottrigby, thank you very much for #87
Applied them both succesfully. And it worked like charm

I have a specific usecase where the SKU of a product is not really the unique identifier (the SKU can change from the receiving part) so i created a field product_id wich i can set now as an unique field. So this patch is working perfectly.

I just needed to change in the commerce_feeds module. so still working on the best solution. but it works right now.

I hope this gets implementen quick :)

Thanks again!

scottrigby’s picture

@rbosscher glad to hear it!

By the way, I also added a companion issue in the Field validation queue: #1705386: Feeds integration (Field validation)

twistor’s picture

Status: Needs review » Needs work

Glad to see this coming along!

Ideas/Suggestions:

  • Marking unique targets should really be a separate operation from adding them. The idea is that you would look at the targets and mark them appropriately. Not just create a bunch of arrays with $targets[$field_name]['optional_unique'] = TRUE;. The problem is that many modules set something like, $targets[$field_name:column_key]That said, there needs to be a hook_feeds_processor_targets() where targets are added, then the alter makes a lot more sense. This should be a separate issue.
  • I would like to see another array key in the config, maybe, "unique callbacks" that modules can add a callback function to, rather than hook_feeds_existing_entity_id(). This would allow some control over ordering, and it would allow us to only call as many functions as necessary until we get a value.

If the callback idea seems way off, I'm not convinced, and would like to hear objections.

twistor’s picture

Jason Dean’s picture

In testing #87 (and moving away from Unique Fields method), I tried this:

  • reinstalled Feeds 2.0-alpha-5
  • installed Field Validation
  • applied both patches
  • added validation rule to my unique field
  • edited importer and checked that field mapping set to unique target (it was, automatically)

So far so good, but importing (using either Feeds node import button or Drush) doesn't update any nodes; it imports new duplicates every time.

Perhaps I'm being a bit optimistic expecting my importer (originally configured using Unique Fields method) to just adapt? Or maybe the new method isn't fully implemented for Feeds node processing yet?

Great to see progress on this issue though - thanks!

tmsimont’s picture

is the patch in 87 applied to dev or alpha 5? It doesn't seem to work on the dev branch. I'd try to patch it myself but there seems to be significant differences between the dev branch and alpha 5. Can you clarify? I'd like to get it in dev if possible because I'm also looking to get this patch working: http://drupal.org/node/1454666#comment-6168760

scottrigby’s picture

@pushka & @tmsimont:

The patches in #87 are against the 7.x-2.x branches of each module:

  • feeds-unique-target-661606-87.patch is against Feeds 7.x-2.x
  • field_validation-feeds-unique-target-661606-87.patch is against Field validation 7.x-2.x

For me this imports appropriately (does not duplicate unique targets) on new entity imports. I'm also patching my Feeds 7.x-2.x with #1033202-69: [Meta] Generic entity processor, but afaik this patch shouldn't rely on that at all.

tmsimont’s picture

Ok thanks -- I got the modules installed, applied both patches sucessfuly, set up an integer field to be unique, was able to check the box on the importer config page to make it a "unique target," but if i run the import twice, the second run duplicates all of my nodes. I have the importer set to "replace nodes" so I'm not sure why it would duplicate all of them... I don't think this is working... Could it be because I'm using an integer field?

scottrigby’s picture

@tmsimont mine is also an integer field, so I don't think that would be the problem.

scottrigby’s picture

@twistor re #95, separating these issues:

  1. #1706026: Create a hook_feeds_processor_targets(). makes sense. Once that's farther along, the example patch for Field validation can be changed to account for that new hook. But we can follow up there (and in the Field validation queue) when that lands - and in the meantime use the existing hook_feeds_processor_targets_alter() (since the update will be backwards compatible anyway!). Though that shouldn't hold up this issue, right?
  2. Regarding hook_feeds_existing_entity_id(), I don't completely understand what you mean by a "unique callbacks" array key in the config (you don't mean that the callbacks would be selected on the processor settings form?).

I can definitely see wanting to avoid invoking another hook inside a method, and normally I would have just extended the class, except that the IDs then wouldn't apply to other extensions of that class, like FeedsNodeProcessor. I guess Feeds could define another CTools plugin, and expect implementing modules to define their own class with a specific method (or callback) then get the first unique ID that's returned? Something like:

         case 'guid':
           $entity_id = $query->condition('guid', $value)->execute()->fetchField();
           break;
+        default:
+          // Bring in unique ID plugins.
+          foreach (ctools_get_plugins('feeds', 'unique') as $plugin) {
+            // Pick the first ID returned by a plugin-defined class.
+            if (!isset($entity_id)) {
+              $class = ctools_plugin_get_class($plugin, 'handler');
+              $entity_id = $class->uniqueID($value, $source, $this->entityType(), $this->config);
+            }
+          }
       }
       if (isset($entity_id)) {

Or just stick with a traditional hook invocation, like in the patch as-is. What do you think? Is this making sense with what you were trying to say?

tmsimont’s picture

FileSize
3.86 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch field_validation-feeds-unique-target-661606-103.patch. Unable to apply patch. See the log in the details link for more information. View

@scottrigby -- it looks like the "bundle name" is not assigned properly in the field_validation_feeds_existing_entity_id() code in the field_validation.module patch.

I've got an importer named "moe_data_importer"

When I run it, $rules_fields_unique is an empty array after field_validation_get_fields takes "node" and "moe_data_importer" as its first two arguments ($entity_type and $bundle_name).

The "bundle" should be the node type, not the machine name of the importer. It's probably working for you if you have given your importer the same machine name as your node type.

The attached patch fixes this by using $config['content_type'] instead of $source->id

(it patches the current 7.x-2.x branch instead of patching on top of your patch, it should be used as a replacement to the patch you had in #87)

scottrigby’s picture

@tmsimont just to clarify, you're talking about the example Field validation patch (not the proposed Feeds patch).
However, you're right! I had named my importer the same as my bundle name, and saw that in $source->id.
Unfortunately, $config['content_type'] only applies to nodes, so that's not an acceptable solution.

@twistor, so this is kind of a blocker IMO…
Within existingEntityId() we need to get the bundle somehow, and pass that as a param in the alter function (or the plugin invocation per #102, or some other method to allow other modules to provide the unique ID during processing).

But currently we have no way to reliably get the bundle type from there yet - unless we assume the bundle type must be set in $config (or defined by extension classes manually somehow) and made available in some definitive way.

Proposal: Ok how about adding FeedsProcessor::bundleType() (an equivalent of entityType()). FeedsProcessorNode::bundleType() can return $config['content_type'], and other entity modules extending FeedsProcessor can add a bundle selection to the settings form, and return their own $config option in their own implementation of bundleType(). That way in FeedsProcessor::existingEntityId() we can just do $bundle_name = $this->bundleType();.

Do you think that makes sense? If so I can whip up a patch that covers all Feeds-supported entities.

(as a sidenote, the patch in #1033202-69: [Meta] Generic entity processor adds the bundle type selection to the Feeds processor config form automatically for entity modules that set their bundle type column to 'required' => TRUE in their Entity metadata info. That patch would just need to be extended to add FeedsEntityProcessor::bundleType() you think the new method idea is appropriate?).

tmsimont’s picture

i like the FeedsProcessor::bundleType() idea -- i'm not sure i fully understand your sidenote.. but i like the bundleType() idea :)

twistor’s picture

#1711648: Abstract bundle handling. this is turning out to be a very fruitful issue.

@scottrigby

What I was proposing, poorly, was this:

/**
 * Implements hook_feeds_processor_targets_alter().
 *
 * @see field_feeds_processor_targets_alter().
 */
function field_validation_feeds_processor_targets_alter(&$targets, $entity_type, $bundle_name) {
  $rules_fields_unique = field_validation_get_fields($entity_type, $bundle_name, 'field_validation_unique_validator');
  foreach ($rules_fields_unique as $field_name) {
    // Add the unique target option.
    $targets[$field_name]['optional_unique'] = TRUE;
    $targets[$field_name]['unique_callbacks'][] = 'callback_func_name';
  }
}

// In the processor. Very sudo code.
        default:
          foreach ($callbacks as $callback) {
            if (func_exists($callback)) {
              $entity_id = $callback(arguments...);
              if ($entity_id) {
                break;
              }
            }
          }

Also, the above code was borrowed from the patch in #103. It illustrates something else I mentioned. Unique settings should not be set for every field that field_validation is aware of. The contents of $targets should be checked for possible unique fields. That's the major reason I proposed the hook_feeds_processor_targets() patch.

It's similar in style to the set_target callbacks we already use, but it allows for multiple callbacks.

scottrigby’s picture

FileSize
1.05 KB
FAILED: [[SimpleTest]]: [MySQL] 3,915 pass(es), 0 fail(s), and 42 exception(s). View

@twistor - ah ok… that makes more sense actually, because only optional_unique targets will ever need to be checked.

Though I don't think we need to set an array of unique callbacks, just one . Because when invoking, even if multiple implementations of hook_feeds_processor_targets_alter() alter the same target, when invoking we'd still have to loop over them anyway and break at the first one we get. So each alter may as well just add/overwrite a single $targets[$field_name]['unique_callback'] and the processor can call whichever one ends up winning =)

But we still need to pass the bundle to our callback, so it can do a proper field check :p But how? Suggestions so far:

  1. We could do that using a new method like I suggested earlier (some method like FeedsProcessor::bundleName()).
  2. Or - since we're only missing this information in FeedsProcessor::existingEntityId() - we could expect implementations of hook_feeds_processor_targets_alter() to also pass the bundle name right back through $targets by reference, like: $targets[$field_name]['bundle_name'] = $bundle_name; (that may seem redundant, but otherwise we can't reliably ensure we have the bundle name when invoking the callback.

I'm attaching a patch for #2 (only because #1 would be more refactoring in Feeds module, and I would like your input on that direction before working on that). Do you have other ideas about definitively getting the bundle name here?

Here's an updated example for the hook implementation (with the bundle name issue in mind), to match the attached Feeds patch:

/**
 * Implements hook_feeds_processor_targets_alter().
 *
 * @see field_feeds_processor_targets_alter().
 */
function field_validation_feeds_processor_targets_alter(&$targets, $entity_type, $bundle_name) {
  $rules_fields_unique = field_validation_get_fields($entity_type, $bundle_name, 'field_validation_unique_validator');
  foreach ($rules_fields_unique as $field_name) {
    // Add the unique target option.
    $targets[$field_name]['optional_unique'] = TRUE;
    $targets[$field_name]['unique_callback'] = 'field_validation_feeds_existing_entity_id';
    $targets[$field_name]['bundle_name'] = $bundle_name;
  }
}

/**
 * Callback for Feeds processor unique entity ID.
 *
 * @param string $target
 *   The unique field name, containing the value to be checked.
 * @param int|string $value
 *   The unique field value to be checked.
 * @param string $entity_type
 *   Entity type for the entity to be processed.
 * @param string $bundle_name
 *   Bundle name for the entity to be processed.
 *
 * @return int
 *   Feeds processor existing entity ID.
 *
 * @see field_validation_feeds_processor_targets_alter()
 * @see FeedsProcessor::existingEntityId()
 */
function field_validation_feeds_existing_entity_id($target, $value, $entity_type, $bundle_name) {
  $rules_fields_unique = field_validation_get_fields($entity_type, $bundle_name, 'field_validation_unique_validator');
  if (in_array($target, $rules_fields_unique)) {
    // Get unique entity ID from unique field value. Note that we are not using
    // field_validation_unique_validator::validate() because we don't want to
    // fire set_error().
    $query = new EntityFieldQuery();
    $result = $query->entityCondition('entity_type', $entity_type)
      ->entityCondition('bundle', $bundle_name)
      ->fieldCondition($target, 'value', $value, '=')
      // Run the query as user 1.
      ->addMetaData('account', user_load(1))
      ->execute();
    if (isset($result[$entity_type])) {
      $ids = array_keys($result[$entity_type]);
      return reset($ids);
    }
  }
}
scottrigby’s picture

@twistor, sorry I somehow missed your link to he new issue: #1711648: Abstract bundle handling..

I guess this issue will now be dependent on that one, and we can sort out how the bundle name method should best be implemented there :)

scottrigby’s picture

FileSize
4.34 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch field_validation-feeds-unique-target-661606-107.patch. Unable to apply patch. See the log in the details link for more information. View

Adding the Field validation example from #107 here for reference. Will still follow up on the bundleName() method in the other issue.

g089h515r806’s picture

I have test latest patch from #107 and #109,
Then i have the same issue with #97, "but importing doesn't update any nodes; it imports new duplicates every time."

g089h515r806’s picture

Status: Needs work » Needs review
FileSize
2.56 KB
PASSED: [[SimpleTest]]: [MySQL] 3,917 pass(es). View
943 bytes
FAILED: [[SimpleTest]]: [MySQL] 4,503 pass(es), 0 fail(s), and 84 exception(s). View

Here are latst patch.
1, I think that we do not need "unique_callbacks" at here,"unique_callback" is enough.What we need is there is a callback function which could get the entity_id for us.One callback function is enough at here.We do no need an array.

2,I have remove the function of field_validation_get_fields, we could use :

$unique_rules = ctools_export_load_object('field_validation_rule', 'conditions', array('entity_type' => $entity_type, 'bundle' => $bundle_name, 'field_name' => $target, 'validator' => 'field_validation_unique_validator'));

and

$rules = ctools_export_load_object('field_validation_rule', 'conditions', array('entity_type' => $entity_type, 'bundle' => $bundle_name, 'validator' => 'field_validation_unique_validator'));

directly. We could add more conditions for ctools_export_load_object at here.

3, Put the code of feeds integration into a seperate file "field_validation.feeds.inc", we could earn a little performance at here.

4 Another small change is use '$unique_rule->col' instead of 'value':

 ->fieldCondition($target, $unique_rule->col, $value, '=')

The patch works at our drupal sites.

7wonders’s picture

#111 is working good for me

Jason Dean’s picture

Yep #111 working well for me too :)

twistor’s picture

Status: Needs review » Needs work

The field_validation code needs to go in the field_validation module.

Re: array of unique callbacks.
This IS needed, otherwise, multiple unique callbacks will overwrite eachother. It doesn't complicate the implementation at all. We just loop over the callback and break on the first one that returns a value.

g089h515r806’s picture

Status: Needs work » Needs review
FileSize
1018 bytes
FAILED: [[SimpleTest]]: [MySQL] 3,919 pass(es), 0 fail(s), and 84 exception(s). View

latest patch, works.

g089h515r806’s picture

To test the patch "feeds-unique-target-661606-115.patch" with field validation, you could download "field_validation.feeds.inc"
at http://drupal.org/node/1705386#comment-6481224 .

PeiPeiH’s picture

In case anyone's using #22 for D6, there seems to be an issue when using Feeds Tamper. My CCK field accepts multiple values, but each should be unique. I used explode, and this caused the unique check to fail. The nodes are uploaded anyway.

zeezhao’s picture

Please can someone clarify the right code to use? I currently installed:

field_validation-7.x-2.x-dev [2012-10-07 -- "7.x-2.1+7-dev" - the latest]
feeds-7.x-2.0-alpha7 + feeds-unique-target-661606-115.patch
unique_field-7.x-1.0-rc1

[edit]
-Created new rule via: admin/structure/field_validation/add when using: Entity Type:
Node, Bundle Name: Product [from ubercart]
- Can see now see the extra fields
- But unique field still got imported.
- Using "Unique Values" with "global scope".

Thanks

Jason Dean’s picture

@zeezhao

Not sure if this is the answer, but I couldn't select the unique field in my importer mappings when using current Feeds feeds-7.x-2.0-alpha7.

http://drupal.org/node/1705386#comment-6676640

Also, not sure you need Field Validation and Unique Field modules. Field Validation is the one to use with the recent patches here.

scottrigby’s picture

@zeezhao and @pushka - can you test #115 with feeds-7.x-2.x-dev? It is not intended for feeds-7.x-2.0-alpha7.

zeezhao’s picture

@scottrigby - thanks for your help.

Now got a step further, getting error "PDOException: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry..."

but I was expecting it to update the duplicate node because I have the following on "Settings for Node processor": "Update existing nodes (slower than replacing them) "

Please confirm if I still need to use unique_field module, and set the field on the content type, as I also did this. When I did not, import loaded duplicate anyway. Thanks

Note: I am also using feeds_tamper & custom_feeds_tamper_term_hierarchy modules on a totally different field [ for taxonomy hierarchy]

g089h515r806’s picture

You have to increase the weight of "field validation" module in systm table. Then clear the cache.
In feeds alpha7, function field_feeds_processor_targets_alter has been splitted into several functios:
number_feeds_processor_targets_alter,
text_feeds_processor_targets_alter,

they run after field_validation_feeds_processor_targets_alter.

We should make sure that field_validation_feeds_processor_targets_alter run at the last place.

I try to use following code:

function field_validation_module_implements_alter(&$implementations, $hook) {
//drupal_set_message('123456');
  if ($hook == 'feeds_processor_targets_alter') {
  drupal_set_message('123456');
    $group = $implementations['field_validation'];
    unset($implementations['field_validation']);
    $implementations['field_validation'] = $group;
  }
}

It does not work.

twistor’s picture

W.M.’s picture

@TamboWeb

Does the GUID has to be a Drupal field ?! Or is it a feeds module thing ?! Thanks.

SGhosh’s picture

Version: 7.x-2.x-dev » 7.x-2.0-alpha7
Status: Needs review » Active

I am using - 7.x-2.0-alpha7. I need to use a custom field as unique field for mapping. Which is the patch i should be using? Any other module to be used along with it?

The patch in this documentation - Use field other than nid as unique node field in feeds for 7.x-2.x - applies on a field.inc file in feeds/mappers which doesn't exist there.

scottrigby’s picture

Version: 7.x-2.0-alpha7 » 7.x-2.x-dev
Status: Active » Needs review

@SGhosh please see #120

agileadam’s picture

FileSize
1.04 KB
PASSED: [[SimpleTest]]: [MySQL] 3,915 pass(es). View

I have an updated version of the #115 patch from g089h515r806. This should apply cleanly to the latest version of 7.x-2.x-dev.

This updated patch eliminates some PHP notices coming from undefined index for $targets[$target]['unique_callbacks'] and $targets[$target]['bundle_name']. I have done minimal testing (few changes from #115; working fine for me) so please test it out.

Note: I am using the same code in 7.x-2.0-alpha7 but the line numbers are different. I put the default switch case just after the guid case. It should be around line 605.

Here's the diff between #115 and #127 (also, the line numbers changed due to other code changes in dev branch):

-          $callbacks = $targets[$target]['unique_callbacks'];
-          $bundle_name = $targets[$target]['bundle_name'];
-          if (!empty($callbacks) && !empty($bundle_name)) {
+          if (isset($targets[$target]['unique_callbacks']) && isset($targets[$target]['bundle_name'])) {
+            $callbacks = $targets[$target]['unique_callbacks'];
+            $bundle_name = $targets[$target]['bundle_name'];
johnv’s picture

In the past, I have patch #22 applied, and installed unique_field for that. Later patches only apply to field_validation.
I can't get it working again to unique_field again. Any help? Or should I switch from unique_field to field_validation?Thanks.

gaurav.pahuja’s picture

Version: 7.x-2.x-dev » 6.x-1.x-dev

Thanks #22 chromix for patch for 6.x. But this patch is not working if I try to make title as unique field but not some CCK created field.

idflood’s picture

Version: 6.x-1.x-dev » 7.x-2.x-dev

I quickly tested the patch in #127 and it seems to work nicely. The patch applies cleanly and in these 12 lines I don't see any code formatting issue.

@gaurav.pahuja19... : patch are usually made for the latest version and then backported if possible.

presleyd’s picture

FileSize
12.47 KB

I applied the patch in 127 on current dev. I used Field Validation to set a unique field on the content type. When I create a new node of that type from the node/add form, the validator successfully errors on a duplicate in that field. In the Feeds mapper I see the new UI option to make that field a unique target and used it to set that field as unique.

When I run the feeds importer though, duplicates are still being created. The originals were not added via Feeds but I thought that was the point of this patch.

liquidcms’s picture

wow.. this is a pretty long thread. i have read through all of this (but will re-read next) but wondering why this is so complicated. perhaps i am missing something here:

- why is any other module required besides Feeds? i do not understand why field validation or unique field should be required for this basic task.
- why are ALL fields (not just title, nid) not selectable as unique?

i must be missing something here as it seems like you should be able to select any fields you want to create a unique key.

also, can someone explain how guid is used. i see it is an autoincrement field in the feeds_item table but i am able to assign it something like title. as per some of the suggestions above i tried creating a hash of title+fieldX and setting to guid (since i am able to set guid as unique) but this did not work (will look at this again).

scottrigby’s picture

@liquidcms you can – there is a hook hook_feeds_processor_targets_alter() which Unique fields Field validation is simply implementing. You can write a module that implements that for whichever fields you wish. It's not a good idea to make that an option for most fields out of the box however, because many will have no guarantee of being unique.

liquidcms’s picture

nope, my bad. i set up the map to guid field incorrectly. so then this does work.. simply. so again not sure what this thread is about.. lol..

so this is what i did:

- added a Source: unique (note: unique is not a column in my CSV)
- map this to Target: GUID and set as unique

in custom module:

function mymodule_feeds_after_parse(FeedsSource $source, FeedsParserResult $result) {
  // use GUID as Unique field by mapping Title+Author to GUID
  foreach ($result->items as $key => $row) {
    $result->items[$key]['unique'] = md5($row['title'] . $row['field_author_users'][0]);
  }
}

i think this should also be doable with Feeds Tamper but couldn't get that to work.

johnv’s picture

@liquidcms, indeed, you could use GUID to make any field unique, but ONLY for nodes created by this very feed.
It does not work your nodes are originated from another source (initial upload, another feeds, manually created by a user).
See #11, #14.

liquidcms’s picture

@johnv, ahh, got it. thanks for the clarification.

oryx’s picture

Hello everyone, I read this thread with the greatest attention since I am having the exact same need (use feeds to update nodes which were not created by feeds). However, I couldn't figure out how to make the patch work : I'm using 7.x-2.x-dev, I applied patch #127, but where should I put the file field_validation.feeds.inc generated by field_validation-feeds-unique-target-661606-111.patch (#111)? In the plugins directory?

Thanks for your help !

marktheshark’s picture

Isn't it time there was a general consensus about how to update entities not created via Feeds?

I'm guessing this is a use case for many people.

The documentation for this probably needs updating.

Please advise which patch to use, so that I can assist in testing, thank you.

emilyf’s picture

Spent a few hours trying to go through this thread and test everything out so I sure hope this benefits others. Here is what I had to do to get this working:

- Using Field Validation 7-x.2-3 release for this testing (the dev version gave me a WSOD, but I will get back to that and try to do a patch)
- Then as mentioned in #116 (http://drupal.org/node/661606#comment-6481228), I downloaded the zip file from the field validation issue queue and added it to my field validation module directory (http://drupal.org/node/1705386#comment-6481224), direct link to this download is http://drupal.org/files/field_validation.feeds_.zip
- I attempted to apply the feeds patch from #111 but it failed, so I manually patched my plugins/FeedsProcessor.inc b/plugins/FeedsProcessor.inc

Doing all of this worked for me.
For anyone needing other help on how to actually use all of this, once you do the above and enable field validation module, go into the content type > Manage Fields that you want to set a unique field on. Now you will have a 'validate' link next to every field. click that for the field you want to change. Then apply the unique validator. Now create your feed importer. When you map a source to this field, you will have a unique checkbox. Note that as with all of these, by default it will be off. You not only have to click "update" and check the box, but if you don't then save the overall screen it will not retain your changes.

Also for anyone wondering, I am using the generic entity processor feeds branch (which is likely why the patch failed for me). So it is working with that too. I am going to see if I can create some more patches that are up to date and will post back.

marktheshark’s picture

Can you explain your use case? Are you using Feeds to update pre-existing entities? Thank you

emilyf’s picture

Actually, the feeds patch in #111 works and applies cleanly on both the current feeds 7.x-2.x branch and the 1033202-entity-processor branch.

emilyf’s picture

My use case:

I have a content type called "show". In this content type, I have a textfield that stores a value. This particular textfield needs to be globally unique in my system (imagine it like a serial number). Let's call this text field "field_serial_number". So in order to tell Drupal this always has to be unique, I installed Field Validation module. I then set up field validation module as I described in #139

go into the content type > Manage Fields that you want to set a unique field on. Now you will have a 'validate' link next to every field. click that for the field you want to change. Then apply the unique validator.

Now, in addition to this, I have an RSS feed from another website. That RSS feed stores some additional information that I want to import into my Drupal system. In fact, I want to overwrite the Title field of my content type "show" with the Title element of that RSS feed. Trouble is, I don't want it to create new nodes. This RSS feed has a tag that has a matching number to my field_serial_number. So I needed a way for my feed importer to know this already existed in my system. This is what this thread is all about.

So, I then patched feeds with #111 and then followed instructions here to update field_validation inc file: http://drupal.org/node/1705386#comment-6481224

Now I set up a new feed importer, and when I map the RSS source field that contains my matching serial number to my field_serial_number I now get a "unique" flag on this field. I check that off, save the feed importer. In addition, you do not set a GUID on this new feed mapper, you don't need it.

Now, when I run my import, when the RSS feed serial number matches an existing serial number in my field_serial_number, that node gets updated. No new nodes are created.

marktheshark’s picture

Thanks for your thorough explanation.

I will try to follow your advice for updating stock values for pre-existing product entities, with the SKU being the unique field.

Do you have any handling for nonexistent serial numbers? Do they get ignored, or do they trigger the creation of new nodes?

emilyf’s picture

They trigger the creation of new nodes in my use case.

One other note: I have now tested this workflow using both the feed importer node processor and the entity processor node (that comes with the new feeds branch related to #1033202: [Meta] Generic entity processor) with success on both.

marktheshark’s picture

In my case the nonexistent field values will need to be skipped... I'll have to look into that.

emilyf’s picture

@marktheshark - you can probably just use feeds tamper for that - there is the option in that to skip upon a certain type of match. see the documentation.

marktheshark’s picture

Thanks, there are also more ideas in this thread for anyone interested in the same functionality (i.e. update only, do not create new entities).

marktheshark’s picture

I tried your steps, but in the end they didn't solve my problem.

Turns out Drupal Commerce Products somehow ensure the uniqueness of the Product SKU and the option in the importer to mark as unique is already there.

Actually installing Field Validation and the patches broke the form for enabling the uniqueness giving a 500 error code in the Ajax request.

I reverted everything back to how it was, marked the SKU as unique and then the import worked, updating the stock levels of the desired products.

Now my only remaining problem is how to force updates only, for which I will look into Feeds Tamper, or some patch that will expose this option in the importer form.

Thanks for your feedback.

Summit’s picture

Hi @marktheshark, interested in your findings!
Greetings, Martijn

emilyf’s picture

@marktheshark - I have found a use case where I need the same thing as you, how to force updates only. Have you done any further research on this?

marktheshark’s picture

Well, I was successful in updating commerce products, however the processor I was using (Commerce Products Multi) currently is losing all images attached to the entity being updated...

So, still some stuff to fix.

As for the ability to update only, I added the definition of the SKIP_NEW option and modified the Processor to skip creation in the processing loop when the entity_id was found not to already exist on the system.

marktheshark’s picture

I have been asking this around in many places, but there is no answer:

Is anybody else losing images associated with entities when updating them via Feeds?

I can't be the only one having this problem...

emilyf’s picture

@marktheshark - can you post a patch or added code for you skip_new option with the processor...i would definitely find this useful to reuse and maybe we can start to work on getting something committed...sounds like there are quite a few people on this thread who might benefit from that.

marktheshark’s picture

FileSize
9.09 KB

I uninstalled the commerce feeds multitype module and relied on plain commerce feeds. I am attaching the modified files for your viewing (search for FEEDS_SKIP_NEW to find the changes easily), I will elaborate on the changes tomorrow, but they are basically this patch adapted to the Drupal Commerce Products plugin.

I'm very happy that the changes worked, I was able to update multiple types of products without actually needing the multi plugin, all nonexistent SKUs were skipped and no nasty side-effects. :-)

emilyf’s picture

FileSize
1.61 KB

@marktheshark here is another alternative...

Attached is a feeds tamper that should do what me and marktheshark have been discussing. To use:
- put this file in your feeds_tamper/plugins folder
- You need latest dev version of feeds_tamper, clear cache
- rename file to entity_skip_new.inc

Here's how it works:
- apply it to the field that you need to match on. So if you are lining up an element in a feed with field_serial_number, put this tamper on that field.

- if it does not see a matching value then it does not create a new node
- if it sees a matching value it will update the node

Also, I had to choose 'skip hash check' on my feed.

Would love some feedback on if this works for others

emilyf’s picture

Issue summary: View changes

Update issue with d.o standard issue template

johnv’s picture

Issue summary: View changes

Added quicklink to comments with latest patches.

Lasac’s picture

@151 for the lost images when updating you will have to check https://drupal.org/node/1107522 and https://drupal.org/node/2049341

miltonsp’s picture

See #159.

kingandy’s picture

Patch in #127 worked for me by itself, the current (7.x-2.3) release of Field Validation already has the feeds API hooks implemented so there was no need to patch anything else. Just installed FV and configured a unique condition, and bam.

miltonsp’s picture

In reference to:
Currently you can choose any field as a unique target using the GUID mapping element. This method has the following limitation:
- this GUID is specific for each importer, so you cannot use multiple importers to update the same node. They will double up.

Note that the Title (Target Field) has no such limitation and will be unique.

If you have another field that is unique (field_xyz), you can do the following:

1. Install 'field_validation' module.
2. Install patch #127 to the 'feeds' module.
3. Add a validation rule to field_xyz to enforce uniqueness.
4. In the Feeds/Node-Processor/Mapping screen mark the field_xyz target configuration as unique. Unmark uniquenes for the others.
5. You are done. Do the import with multiple mappers without creating duplicates.

Jason Dean’s picture

@emilyf

Your Feeds Tamper plugin #155 is working perfectly for me. I haven't ticked the option to skip hash check and it seems fine so far...

Thanks a lot! :)

Jason Dean’s picture

Issue summary: View changes

Added a more non-technical introduction.

JayShoe’s picture

This discussion was started on December 16, 2009. It's no November 11, 2013. Is there any commit or dev version that incorporates some or all of these concepts?

My use case is simple, I have a "VenueID' as a custom field in my content type venues. And I have a VenueID in my feed import (csv). I need to either add or update existing nodes based on this match.

If this isn't available in any committed version - I'll try the patches. But it seems like this use case is very common.

mstef’s picture

Patch is great - the only issue I have, and I'm not sure if this is the correct thread to add this to, is that say your entity has 10 fields and 1 primary/unique key and you have 100 records fully inserted in to the database.

Then, you use Feeds to import data via a CSV that contains the primary-key/unique-field and just one other column/field that you want to update. All records will be updated with that one column/field but the other 9 fields will return to their default value since they are not present in the update import. I think the Entity processor needs to be changed so then when an update is being done, ONLY the present fields are updated, and the others are left as they stand (ie, merge verses update).

mstef’s picture

I also can't seem to find any difference in FeedsProcessor.inc between "replace" and "update".

mstef’s picture

For those looking for unique support with Data - I wrote a small module that takes advantage of this patch and provides the integration: #2174373: Add support for Feeds "unique" field updating/replacing.

twistor’s picture

Issue summary: View changes
Status: Needs review » Needs work
Issue tags: +Needs tests

This needs to be updated. FeedsProcessor now provides a bundle() method, so we don't need the $bundle_name hack.

This also needs tests, at least one.

joachim’s picture

+++ b/plugins/FeedsProcessor.inc
@@ -621,6 +621,18 @@ abstract class FeedsProcessor extends FeedsPlugin {
+          if (isset($targets[$target]['unique_callbacks']) && isset($targets[$target]['bundle_name'])) {

AFAICT this is adding new properties to hook_feeds_processor_targets_alter(). Therefore this needs to be documented in the api.php file.

bradjones1’s picture

FileSize
965 bytes
PASSED: [[SimpleTest]]: [MySQL] 5,080 pass(es). View

This still needs tests but this is re-rolled to incorporate feedback from #165.

Also re: #166, we're not changing the function signature of hook_feeds_processor_targets_alter(), but we do define a signature for a callback, so not sure if that needs to be documented anywhere.

twistor’s picture

The docs need to be updated, as well as, an example callback provided in feeds.api.php.

Anybody’s picture

Status: Needs work » Needs review

As already stated out in #161 it would be nice to see some progress here and finally finish this thread after FIVE years ;). I'll set this to needs review. Who's the right person to update the docs and do further checks?

MegaChriz’s picture

FileSize
6.85 KB
PASSED: [[SimpleTest]]: [MySQL] 5,080 pass(es). View
6.68 KB

I've started with updating the docs and writing a test. Attached is what I currently got. The test doesn't pass yet and I haven't figured out yet why.

From the previous patch I also changed this:

+++ b/plugins/FeedsProcessor.inc
@@ -761,6 +761,17 @@ abstract class FeedsProcessor extends FeedsPlugin {
+              if (function_exists($callback) && $entity_id = $callback($target, $value, $this->entityType(), $this->bundle())) {

To this:

if (is_callable($callback) && $entity_id = call_user_func_array($callback, array($target, $value, $this->entityType(), $this->bundle()))) {

This way callbacks that live in classes (e.g., Test::callback()) also work instead of only procedural functions (e.g., callback()).

MegaChriz’s picture

FileSize
7.28 KB
FAILED: [[SimpleTest]]: [MySQL] 5,178 pass(es), 3 fail(s), and 0 exception(s). View
442 bytes

I forgot to add a reference to the test FeedsMapperUniqueTestCase in the file feeds.info.

New patch.

Status: Needs review » Needs work

The last submitted patch, 171: feeds.unique-target-661606-171.patch, failed testing.

WillowDigit’s picture

I had this patch working at one stage, and now it does not. Using the latest dev release of field_validation which already contains the needed patch, and the latest dev release of feeds patched with #171. Tried some other configurations also.

This thread is becoming very long and from some of the comments I can see that I am not the only one getting lost.

raul_drupal_dev’s picture

Hi community

I made a video explaining how fix this issue. its made in spanish but is easy to follow. If you've got any question feel free to ask me!. Also I made a pack with the modules already parched. just plug&play!

The link --> http://drupalia.cat/videotutorial/drupal-7-crear-un-campo-unico-customiz...

Please feedbak is welcome!!

MegaChriz’s picture

Status: Needs work » Needs review
FileSize
7.3 KB
PASSED: [[SimpleTest]]: [MySQL] 5,183 pass(es). View
3.4 KB

The attached patch completes the tests I was working on in #170/#171. In comparison with the previous patch, there are no changes in the Feeds module itself, only in the tests. Technical details: the unique callback was moved from the test class "FeedsMapperUniqueTestCase" to the test module. The test class couldn't be loaded during runtime, because it inherits from the class "DrupalWebTestCase" which is only available when the simpletest module is enabled.

@raul_drupal_dev
Great video. :) I couldn't understand much of the words of it, but I think the steps provided of how to install the patch are clear enough. Thanks for proving that the patch works.

chrowe’s picture

This seems to not work with link fields. Any ideas why?

MegaChriz’s picture

@chrowe
The feeds target name for the link field is different compared to other fields, because the link field consists of two values instead of one ('url' and 'title'), see feeds/mappers/link.inc. Assuming you are using the Field validation module: the Field validation module does not take this into account, it assumes that each feeds target for a field is equal to the field's machine name and this is not the case for the link field.

If you want to make the link field a unique target, you should do the following:

  1. In your own module, implement hook_feeds_processor_targets_alter() and for the feeds target of your link field, set 'optional_unique' to TRUE and define a callback function for 'unique_callbacks'.
  2. Implement the callback function in which you query the database to look if the provided link already exists. The callback function should return the entity ID if the link was found or NULL otherwise.

The details plus an example are included with the patch in #175. Check feeds.api.php after applying the patch.

hairqles’s picture

FileSize
7.94 KB
FAILED: [[SimpleTest]]: [MySQL] 5,181 pass(es), 0 fail(s), and 367 exception(s). View

Hi,

I've tested the patch and it works as I was expected, so great job! Thank you!

On the other hand the current implementation prevents me to define my own logic to determine if the GUID or the URL field is unique or not. My usecase is that the GUID field should be unique globally.

So I've made a few changes in the code. You can check that in the attached patch file.

Status: Needs review » Needs work

The last submitted patch, 178: feeds-unique-target-661606-178.patch, failed testing.

MegaChriz’s picture

Status: Needs work » Needs review
Related issues: +#1539224: Add support for unique fields to be unique site wide

@hairqles
Uhm, making the GUID globally unique seem to be a total different issue. In fact, there is already an issue about this: #1539224: Add support for unique fields to be unique site wide. Lets keep this issue focussed.

Exploratus’s picture

#175 Worked for me. Field validation was not working until I installed the patch. I am using field validation, Feeds 7.x-2.0-alpha8, and Patch #175 to ensure imports are mapped to existing nodes based on a unique field.

twistor’s picture

Issue tags: -Needs tests
FileSize
8.49 KB
PASSED: [[SimpleTest]]: [MySQL] 5,632 pass(es). View
twistor’s picture

FileSize
8.58 KB
PASSED: [[SimpleTest]]: [MySQL] 5,632 pass(es). View

Normalize hook parameters.

twistor’s picture

FileSize
2.83 KB
8.73 KB
FAILED: [[SimpleTest]]: [MySQL] 5,631 pass(es), 3 fail(s), and 0 exception(s). View

Status: Needs review » Needs work

The last submitted patch, 184: feeds-unique-target-661606-184.patch, failed testing.

twistor’s picture

Status: Needs work » Needs review
FileSize
3.34 KB
9.69 KB
PASSED: [[SimpleTest]]: [MySQL] 5,632 pass(es). View

I think this is ready.

MegaChriz’s picture

Assigned: Unassigned » MegaChriz

Thanks for the update! I'll plan to take a closer look at the patch in #186 next Thursday. For now, I saw one thing in the code which I have a question about:

+++ b/feeds.api.php
@@ -369,5 +380,40 @@ function my_module_form_callback($mapping, $target, $form, $form_state) {
+  $query = new EntityFieldQuery();
+  $result = $query->entityCondition('entity_type', $entity_type)
+                  ->entityCondition('bundle', $bundle)
+                  ->fieldCondition($field_name, $column, $values)
+                  ->execute();

+++ b/tests/feeds_tests.module
@@ -205,3 +213,20 @@ function feeds_tests_mapper_form($mapping, $target, $form, $form_state) {
+  $query = new EntityFieldQuery();
+  $result = $query->entityCondition('entity_type', $entity_type)
+                  ->entityCondition('bundle', $bundle)
+                  ->fieldCondition('field_alpha', 'value', $values)
+                  ->execute();

I'm not sure, but isn't the extra indenting here against coding standards? I don't think it is anywhere marked up in core like that.
Example from field.api.php:

$query = new EntityFieldQuery();
$found = $query
  ->fieldCondition($prior_field['field_name'], 'value', $lost_keys)
  ->range(0, 1)
  ->execute();
lquessenberry’s picture

Hey guys, I can't get the settings to stick when I select them.Only local images are allowed. I applied patch 186 to the newest dev version of feeds.

twistor’s picture

@lquessenberry, what are you trying to make unique? You have to click save after changing the settings.

twistor’s picture

lquessenberry’s picture

I was away this weekend. Let me see if I can provide a screenshot this time. I can't get it to save for some reason.

lquessenberry’s picture

http://puu.sh/akOS9/096f047e85.png Here is a screenshot. I am using an entity and not a node. Might that be a problem? I check the radio button and hit save, and then it returns that it was saved and nothing is selected.

lquessenberry’s picture

Wait a minute. I don't think I set a field to be unique with the unique fields module. I was mapping ot a GUID. I will turn that off and turn a field to be unique in my entity. I will see if that works.

MegaChriz’s picture

I'm halfway of reviewing the patch in #186 (found a few minor code issues so far). For now, I just want to warn users that this patch does not work with Field validation module. This is not because the patch in #186 is wrong, but because the implementation of the Field validation module is based on an earlier patch, so Field validation must be updated once a fix for this issue gets in.

Site builders (using Field validation module): temporary use the patch in #175.
Developers (not using Field validation module): use the patch in #186.

I plan to post a patch for Field validation module here when I'm done with reviewing #186.

MegaChriz’s picture

Assigned: MegaChriz » Unassigned
FileSize
9.67 KB
PASSED: [[SimpleTest]]: [MySQL] 6,041 pass(es). View
3.16 KB
3.65 KB

After I modified field_validation.feeds.inc based on the code example in feeds.api.php I can confirm the patch in #186 works!

I made a few minor modifications. This doesn't change much functionality, it's just a little cleanup.

+++ b/feeds.api.php
@@ -369,5 +380,40 @@ function my_module_form_callback($mapping, $target, $form, $form_state) {
+  $query = new EntityFieldQuery();
+  $result = $query->entityCondition('entity_type', $entity_type)
+                  ->entityCondition('bundle', $bundle)
+                  ->fieldCondition($field_name, $column, $values)
+                  ->execute();

+++ b/tests/feeds_tests.module
@@ -205,3 +213,20 @@ function feeds_tests_mapper_form($mapping, $target, $form, $form_state) {
+  $query = new EntityFieldQuery();
+  $result = $query->entityCondition('entity_type', $entity_type)
+                  ->entityCondition('bundle', $bundle)
+                  ->fieldCondition('field_alpha', 'value', $values)
+                  ->execute();

I changed the indentation of those to match with Drupal core's code style (see also #187).

+++ b/plugins/FeedsProcessor.inc
@@ -751,23 +751,40 @@ abstract class FeedsProcessor extends FeedsPlugin {
+    static $targets = array();

I turned this into a drupal_static(), as I think this could else cause problems in automated tests. Imagine you have two tests with the same importer ID and they both involve testing unique ID's. Without drupal_static() the second test would receive the mapping targets from the first test.

Attached are three files:

  • New patch for this issue.
  • An interdiff to see the changes with the previous patch.
  • A patch for Field validation. This patch could be dropped in the Field validation issue queue once a fix for this issue gets in. The patch' name has 'do-not-test' added to prevent the testbot from testing the patch. You as users are welcome to try that patch!

Looks ready to me.

manojbisht_drupal’s picture

FileSize
938 bytes
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch feeds_unique_target.patch. Unable to apply patch. See the log in the details link for more information. View

Hello,

I have created a custom field in one of my content type, and I wants to add it in feeds mapping for node processor as unique field.

As suggested by some people.

I have downloaded the unique_field module and after that applied the patch mentioned in #57.
But the patch didnot work.

as there is not a file field.inc in mappers folder.
so I edited the patch and again commiting it.

Please review it.

Meanwhile, I have also downloaded the fresh install of feeds both alpha and development, no one has field.inc file.

manojbisht_drupal’s picture

FileSize
830 bytes
PASSED: [[SimpleTest]]: [MySQL] 5,930 pass(es). View

Sorry for above patch it will not work.

Uploading new patch.

The last submitted patch, 196: feeds_unique_target.patch, failed testing.

MegaChriz’s picture

@manojbisht_drupal
Sorry, your piece of code should go in Unique Field module, not in the Feeds module. See the issue summary:

Since Feeds should not be responsible for enforcing unique field values, it doesn't care how that gets accomplished.

To get this to work with Unique field module:

  1. Checkout or download the latest Feeds development version.
  2. Apply the patch from #195 to the Feeds module.
  3. In your own module or in the Unique field module, implement hook_feeds_processor_targets_alter() and for the feeds target of each node field, set 'optional_unique' to TRUE and define a callback function for 'unique_callbacks'.
  4. Implement the callback function in which you query the database to look if the node already exists based on the given field. The callback function should return the node ID if the field was found or NULL otherwise.

The details plus an example are included with the patch. Check feeds.api.php after applying the patch.
You can also take Field validation's implementation as an example. In this case, be sure to apply the in #195 provided patch for Field validation module first.

MegaChriz’s picture

  • twistor committed f22a0f7 on 7.x-2.x
    Issue #661606 by MegaChriz, twistor, Cottser, scottrigby,...
twistor’s picture

Status: Needs review » Fixed

Congratulations everybody! Thanks for all of your hard work.

MegaChriz’s picture

Great! I've posted the patch that is needed for Field validation in #1705386-18: Feeds integration (Field validation). This is the same patch that I provided here in #195.

MegaChriz’s picture

I've also updated the docs for how and when to use this feature.

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.

knowledges33ker’s picture

MegaChiz, I'm trying to follow the documentation you linked to in #204.

I've got multiple feeds importing nodes into multiple content types. The problem is that some stories exist in two (or more) feeds and therefore end up duplicated, triplicate (or worse) once all the feeds have run. Since I'm using a different importer for each feed, the existing stories aren't being recognized as needing to be updated or skipped and they are simply being created as if they are new nodes.

After installing the field validation module and applying the patch from #1705386-18: Feeds integration (Field validation) to the Field validation module and setting a unique value in the content type(s), I get to the last step "Go to the mapping settings of your Feeds importer, add the field you want to be unique and set this field as an unique target."

But on the mapping settings tab of the Feeed importer, after I add the field I want to be unique (a link field imported from the feed), I don't have the option to set that field as unique from with the feeds interface.

Where have I gone wrong?

GerZah’s picture

Is it possible that the "Use other field as an unique target in Feeds 7.x-2.x" feature broke in Feeds 7.x-2.0-alpha8?

I'm using the simplest possible other unique field, namely an integer ID. After updating Feeds to 7.x-2.0-alpha8, it won't respect another import using the same numerical ID as existing node that should be updated, but actually create a new node – with the same numerical ID.

The latter is extremely disturbing: Feeds actually creates a second node using the same numerical ID that is supposed to be unique. Albeit, field_validation works fine – I am not able to edit and save that node: Validation kicks in and won't let me re-use the identical ID.

Forget what I said. — Sorry, something went wrong and I ended up with 7.x-2.0-alpha8 instead of 7.x-2.x-dev.

GerZah’s picture

Status: Closed (fixed) » Needs work
GerZah’s picture

Status: Needs work » Closed (fixed)
joachim’s picture

I'm trying to get support for this into Data module (#2174373: Add support for Feeds "unique" field updating/replacing) and I don't understand the reason for this:

+++ b/plugins/FeedsProcessor.inc
@@ -751,23 +751,40 @@ abstract class FeedsProcessor extends FeedsPlugin {
+        if (!is_array($value)) {
+          $value = array($value);
+        }

Why should the value always be turned into an array?
From Data module's perspective, this just means it has to be extracted out of the array in the unique_callback.

MegaChriz’s picture

@joachim
This is done for two reasons:

  1. To be consistent with the target callback, for which the given value(s) is also always an array (see my_module_set_target() in feeds.api.php).
  2. Because values coming from the source can be single-valued (string, integer) or multi-valued (array). By casting the value to an array, it's certain that the "unique"-callback always receives an array. This simplifies the callback implementation as it no longer have to perform that check. For more information see #2093651: Simplify target callbacks. and the accompanying change record.
codev0’s picture

Hi.
After patching, on localhost everything is good, but after placing in to shared hosting I have error.

Recoverable fatal error: Object of class FeedsSource could not be converted to string in DatabaseStatementBase->execute() (line 2171 of /includes/database/database.inc).

I applied this patch https://www.drupal.org/files/issues/feeds-unique-target-661606-195.patch
Is this patch available from module dev version?

UPD: I forgot upload updated field_validation. Sorry, my bad.

phendji-cap’s picture

@ Dakanca

Use the version of Dev. directly because the patch already applied to this version

codev0’s picture

@phendji-cap
I tried use dev version, it breaks my site.

joelpittet’s picture

@dakanca can you open up a new issue for the "breaks my site" issue and link it here? Maybe put more detail into what error message you are getting from PHP?

brandonc503’s picture

*deleted as i cant remove account

yatendrasingh121’s picture

Hi,

Patch attached in #195 does not work for me. Becuase I am using 7.x-2.0-alpha7 version of feeds . I am attaching a patch for 7.x-2.0-alpha7 version.

kingandy’s picture

As of today the code does not appear to have made its way into the main branch (7.x-2.x-alpha9) - if you want the unique target mappers you'll need to download the latest -dev release.

dasginganinja’s picture

I can confirm what kingandy has said.

marcolz’s picture

Patch from #217 updated for 2.0-alpha9 attached.

yatendrasingh121’s picture

Patch attached in #220 does not work for 2.0-alpha9 . Attaching correct patch.

memcinto’s picture

7.x-2.0-beta1 is now out as of 11-July. Hoping for a new patch ;-) THANKS!

dasginganinja’s picture

@twistor, is there any way this can get rolled up into the next 7.x beta release?

twistor’s picture

This is already in beta1.

MegaChriz’s picture

Issue summary: View changes

The patch from #195 is already part of the 7.x-2.0-beta1 release. See commit. See also the code from this release.

See the docs for how to use this feature.

dasginganinja’s picture

+1 Thanks guys.

memcinto’s picture

re #225 EXCELLENT!! Thank you so much!!

adrien.felipe’s picture

Using the same feature from the 7.x-2.0-beta1 release, with a Commerce Order and Commerce order ID as unique, it does not work.
I get a duplicated entry SQL error.
Should it work? Should we extend the patch or functionnality?

MegaChriz’s picture

@wayaslij
Sounds like an issue in the Commerce Feeds module, if you are using that. Or, if you use the Feeds entity processor module, see #2004762: Entity ID as unique target does not work.

Anyway, this issue is about being able to use any target as unique identifier using a custom module by specifying a callback on the target definition. The Field validation module implements such a callback. This issue is not about unique targets in general. If a specific unique target does not work, please open a new issue. If the unique target is provided by a module other than Feeds, open an issue in the queue for that module (instead of Feeds' queue).

SeanA’s picture

Issue summary: View changes