For an example on how to use this feature, see the documentation.
Problem/Motivation
Currently you can choose any field as a unique target using the GUID mapping element. This method has the following limitations:
- this GUID is specific for each importer, so you cannot use multiple importers to update the same node.
- this GUID is stored in its own table, so nodes that are created from another source than Feeds, cannot be updated via Feeds.
Currently there is a hook to allow specifying any field as a unique target (GUID) when mapping is to implement, using hook_feeds_processor_targets_alter()
. However, there is no hook allowing implementing modules to provide an existing entity ID for the specified unique field during Feeds import.
Proposed resolution
Add a new hook: hook_feeds_existing_entity_id()
[2013-07-25] Comments #111 and #127 contains the latest patches on Feeds and Field Validation modules.
User interface changes
None. The mapping UI will still respect unique target selection based on implementations of hook_feeds_processor_targets_alter()
. However, this new hook will allow modules who specify that unique target to now ensure they import properly (do not duplicate).
API changes
Since Feeds should not be responsible for enforcing unique field values, it doesn't care how that gets accomplished. Any module (whether Field validation (general entity support), Unique Fields (supports nodes only), or other validation modules (contrib or custom) can accomplish this by implementing two hooks:
hook_feeds_processor_targets_alter()
for specifying which field it is validating as unique, and add the unique target option for selection on the mapper form (the Field validation patch from #87 is a working example for validating using that module)hook_feeds_existing_entity_id()
for providing the existing entity ID for the specified unique field during import (that patch also contains an example implementation)
Original report by lunk_rat
Is it possible to use any CCK field as a unique target? I have a CCK filed that is unique to each node and I would like to update existing nodes based on a unique source field in a CSV file, using my CCK field as a unique target.
I have been trying to figure this out but no luck yet.
Thanks!
Comments
Comment #1
alex_b CreditAttribution: alex_b commentedThis is not possible in the current implementation. Could be added though.
For simple text/number_integer/number_decimal fields (mapped with mappers/content.inc) this would mean:
1) Declare mapping targets unique in content_feeds_node_processor_targets_alter()
2) Add a new hook that is invoked in
FeedsNodeProcessor::existingItemId()
in the switch statement's 'default' state. The hook would ask all implementers for an existing $nid for a given $source_item / $source combination (in analogy toFeedsNodeProcessor::existingItemId()
).3) Implement this hook in content.inc for all fields.
Comment #2
tomcatuk CreditAttribution: tomcatuk commentedActually, I'm already doing pretty much exactly what linkovitch is asking. One (possibly) important note - all the nodes I'm updating were initially created from the .csv using the importer. I'd be amazed if this worked on nodes NOT created by feeds.
I've got a "unique" CCK field, and in my .csv it's got a column identifier of course so I can use mappings. First I mapped this unique field to the CCK field I want it to go into (so far so obvious). I then mapped the SAME field from the .csv to "GUID" and ticked the "unique target" checkbox. The field values aren't actually GUIDs of course.
I've got a bad feeling Alex will say this is the wrong thing to do, but I can say it's working for me - I've got 9000 nodes in that table and uploading a modified .csv updates the existing nodes, doesn't create duplicates.
Comment #5
TamboWeb CreditAttribution: TamboWeb commentedI am at a complete loss here. From what I understand ....
From #1 above,
part 1.-
It adds the checkbox for unique target to the mapped fields. So far so good.
Now Part 2. Add a new hook to check if node already exists; This code checks for nid and guid.
So to check mytargetid, I would need to add to the code above another case branch to check for mytargetid.
Where do I implement this?
Is is in the content.inc file, hard coded to the FeedsNodeProcessor.inc or somewhere else?
Or do I create the hook?, how?
Thanks.
Comment #6
twistor CreditAttribution: twistor commentedsubscribe
Comment #7
TamboWeb CreditAttribution: TamboWeb commentedI found a solution to my problem without the need to re-code as in #5 above.
In my application, I am using Feeds plus the XML parser for feeds. I am importing a non-standard XML file with lots of CCK fields. ( Real Estate Listings )
For some reason I overlooked the comments of XML parser readme file, my mistake.
While you are mapping the XML fields to the CCK content type, you can do the following:
Then next time the xml feed comes in, any records that have a GUID match either get skip or get updated if there are any changes. This works for me since all properties have a unique listing number.
Furthermore, since I need the listing number in the cck content type, I was able to assign it to both the cck field and the GUID.
Hope this helps.
Comment #8
lunk rat CreditAttribution: lunk rat commentedI did exactly as you explain to solve my problem.
Comment #9
alex_b CreditAttribution: alex_b commented#7 is a good solution and may be applicable in most cases where people are looking for a CCK field or similar as unique target.
#5 hook_feeds_node_processor_unique() needs to be invoked from the existingItemId() method FeedsNodeProcessor.inc
Comment #10
ccoppen CreditAttribution: ccoppen commentedSubscribing, but going to try out the custom additions above. If it works, I'll make a patch.
I have nodes I imported from a MySQL db, but now I need to update them with info from a SQL Server db which does not have the same fields to use as unique targets.
I need this ASAP, so if someone else has figured it out, let me know. Otherwise, I'm working on it.
Comment #11
alex_b CreditAttribution: alex_b commentedAnother reason why #7 is generally a good solution is because "guid" and "url" are indexed.
If we offer e. g. CCK fields as potentially unique targets it won't perform out of the box, adding an index to those fields would be practically required - problem is, this is
a) paramount to hacking a module
b) given a) offering a UI that assumes that you will know and fix the implicit performance issue is a bad practice.
That said, there are a lot more potential 3rd party mapping targets that would be equipped with the right indexes to offer a unique flag. This means I would still accept a patch that introduces a hook_feeds_node_processor_unique(). However, I won't accept a patch using this new functionality for CCK mappers or similar 3rd party mappers in Feeds that don't have the proper indexes to function as a unique mapping target, in these cases #7 is the proper workaround.
Comment #12
alex_b CreditAttribution: alex_b commentedComment #13
smscotten CreditAttribution: smscotten commentedHere's what I don't get: what is the purpose of having a unique target that isn't in a nodetype? Unless I am totally misunderstanding this, it only forces uniqueness in the set of new nodes being created, not uniqueness among the set of all nodes of that type.
Comment #14
alex_b CreditAttribution: alex_b commentedIt forces uniqueness among all nodes imported by the given importer from a given source. This ticket is about allowing node properties exposed by mappers to be unique targets - e. g. with this feature you could ensure that specific values on a CCK field can only occur once in a set of imported nodes.
Comment #15
smscotten CreditAttribution: smscotten commentedI'm sorry Alex, but I don't understand your comment. I don't believe I was hijacking the thread but explaining why the feature is important.
But perhaps I'm misunderstanding the scope of the feature entirely.
Does a unique target only constrain the data coming in during a specific import? That is to say, is there any facility in Feeds to check to see that the unique target is not only unique in the import but that it will also be unique in the list of nodes of that type?
I'm wondering if this feature means what I thought it did: that it can be used to differentiate between "create new" and "update existing" when bringing new node data in.
From "ensure that specific values on a CCK field can only occur once in a set of imported nodes" (emphasis obviously mine) I'm suspecting that there is no checking of existing nodes.
So if I have a CSV file with two rows, and GUID for each one is "XYZ123" and the import runs every day (even though the CSV file doesn't change), then every day I will get one new node, and at the end of the week I'll have seven nodes with XYZ123 (If I set GUID both to GUID and to another field).
What I am looking for is a way to make there be only one node with whatever field set to XYZ123 no matter how many times the feed importer runs.
If the module's design indicates I would end up with one row, then the ability to use a CCK field for a unique target is critical. If I would end up with seven I don't see how making CCK fields into unique targets is even remotely useful.
(If after a week I'd have fourteen XYZ123 nodes then I've completely misunderstood everything about this module.)
If you or someone would indicate which of these descriptions fits the unique target behavior, I'd appreciate it and be better able to participate in this discussion in a more constructive manner. Thank you!
Comment #16
EvanDonovan CreditAttribution: EvanDonovan commentedDoes this mean that a unique target isn't actually checked for uniqueness between multiple source feeds, but only ensures that the items from a single feed are unique? I need something that is unique across all nodes of a given type, not just those from a particular source.
My apologies if I'm misunderstanding or further leading this thread off topic.
It looks like from the code in the existingItemId method of FeedsNodeProcessor that the query looks at feed_nid as well as id and either guid and url. That suggests to me then that there is the possibility of duplicates being created across separate feeds.
Comment #17
EvanDonovan CreditAttribution: EvanDonovan commentedSorry for derailing this issue - I decided that #998426: Unique targets ignored unless nodes have same parent feed node was a better place to take the discussion from #13 on. This issue can return back to its original purpose - creating a hook for defining new unique targets. Theoretically, the ideal solution for the issue I describe in #998426: Unique targets ignored unless nodes have same parent feed node might require the hook described in this issue though.
Comment #18
EvanDonovan CreditAttribution: EvanDonovan commentedTo answer the question in #5, I think what Alex is suggesting is something like:
2)
3) And then in your module, implement the hook with something like:
Obviously for CCK, the queries would be more complex, and you would want to use a switch statement.
Does that make sense? Does the idea of a hook_feeds_processor_unique_value() sound like something that could get in to the actual module? If so, that would mean that I wouldn't have to do what I do in #998426-9: Unique targets ignored unless nodes have same parent feed node as a hack.
Comment #19
EvanDonovan CreditAttribution: EvanDonovan commentedSetting this to "needs review" so that the proposed hook could be reviewed, prior to me writing an actual patch.
Comment #20
EvanDonovan CreditAttribution: EvanDonovan commentedI think that I am going to need to follow the approach in my comment #18 for the project I am currently working on, since I have a case in which the feed doesn't have unique URLs. Rather the uniqueness is determined by a combination of title, description, and a custom tag called fp:provider.
I'm thinking that I will have to do the following:
1) Define a new source field that is an md5 hash of the title + description + fp:provider tag.
2) Define a new target field that is unique, and which is stored in its own database table, on which the foreign key is the nid (uniqueness between feeds is not an issue; I want this to be global).
3) Change the code of the FeedsNodeProcessor to have a hook_feeds_processor_unique_value().
4) Implement that hook to query the table into which my hashes will get stored.
Does this sound like a good strategy? Is this the only way to accomplish what I would like to do, or is there an easier way?
Comment #21
EvanDonovan CreditAttribution: EvanDonovan commentedActually, I think I really only need to do #1, and then set that as the GUID field. I'll see if that works.
Comment #22
chromix CreditAttribution: chromix commentedI decided to take this on using the Unique Field module, since it takes care of the uniqueness validation out of the box. My patch works directly with the module's settings variables to figure out which CCK fields should be used as unique target fields. For the sake of simplicity, this patch will only work in cases where the node has a single unique field and not multiple unique fields working in combination. Either way, thanks to everyone else who contributed code to this ticket. Enjoy!
Comment #23
msimanga CreditAttribution: msimanga commentedThank you #22 chromix, worked like a charm. I vote for inclusion in the next release of feeds.
Comment #24
itai_biren CreditAttribution: itai_biren commentedHi,
I'm using drupal 7 and try to patch this file (#22) but i cant
i get an error:
can't find file to patch at line 5
Comment #25
EvanDonovan CreditAttribution: EvanDonovan commentedI believe all the code in this issue was against 6.x. I'm not sure if there are substantial differences in the architecture of 7.x. I think we should keep this against 6.x, for now, since patches are still getting committed to that branch. If you wanted to re-roll for 7.x, though, that would be cool, though, I'm sure.
Comment #26
lhugg CreditAttribution: lhugg commentedReply to #22:
In my testing, the Unique Field module fails if any of the original nodes created by feeds is deleted. In our test case, nine node were imported, and used a unique path to the original source as the unique field. Additional runs of feeds did not re-add the nodes. BUT, if just a single one of the nine nodes was deleted, then the entire nine were re-created in the next run.
BTW, we are using Drupal 6 with Feeds 6.x-1.0-beta11, and the included Node Processor.
I don't understand why Feeds doesn't directly address duplicates itself. It's seems like such an essential function.
Comment #27
Niklas Fiekas CreditAttribution: Niklas Fiekas commentedSubscribe.
Comment #28
Mohammed J. RazemPatched against 7.x-2.x
There you go.
Comment #29
Mohammed J. RazemSorry.. Forgot to rename a variable. Here's the correct patch.
Comment #30
Niklas Fiekas CreditAttribution: Niklas Fiekas commentedIntresting ... integrate with Unique field module ...
How would I decide for my custom field type, if the value is unique? I'd still like something like
hook_feeds_node_processor_unique()
or another hook to do that. Maybe Unique field could then implement it rather than doing "module_exists" in Feeds?Comment #31
Lirkun CreditAttribution: Lirkun commentedTnx for great patch chromix and Mohammed J. Razem also =)
I used it for D 7.7 and it works great.
Comment #32
rowanprice CreditAttribution: rowanprice commentedThank you so much, Chromix! And everyone else for their ideas here.
Once I applied the patch against the feeds module instead of unique field it worked like a charm :0)
Comment #33
dsnoeck CreditAttribution: dsnoeck commentedThanks, this patch works also well for me.
Comment #34
amacias CreditAttribution: amacias commentedThanks, the patch works perfectly if you choose as your unique field a cck field of the node, but not if you select the title. Is there any quick fix in the patch so title can also be chosen?
Comment #35
chromix CreditAttribution: chromix commentedThe best quick-fix I could think of would be creating a required "Title" CCK field, and then have the Auto Nodetitle module use its value to set the real node title. Otherwise, it would obviously be better for Unique Field to work on node titles, but that's an issue for that module.
Comment #36
amacias CreditAttribution: amacias commentedThanks chromix, that worked perfectly!
Comment #37
rylowry@gmail.com CreditAttribution: rylowry@gmail.com commentedI installed the Unique Field module and the patch in #29. It worked perfectly for me. I'm running Drupal 7.9 and Feeds 7.x-2.0-alpha4. I needed to have 2 different feeds updating the same nodes, and this did the trick.
Comment #38
itai_biren CreditAttribution: itai_biren commentedHi,
I'm using drupal 7 and try to patch this file (#29) but i cant
i get an error:
can't find file to patch at line 5
Comment #39
skyoo8 CreditAttribution: skyoo8 commentedHi, I have installed Unique Field module and the patch in #29. But I can not import new node anymore, its shows "There are no new nodes." If I disable Unique Field, its works fine again.
Also, If I choose Update existing nodes instead of Do not update existing nodes, my data got messed up, feeds update the wrong node.
Need some help please.
Comment #40
johnvPatch #29 applies correctly and works fine with latest feeds and unique_field.
@itai_biren, #28, are you sure you are in the right directory when applying the patch?
@skyoo8 #29, are you still having problems? we need some more details to be able to help you.
Comment #41
Michsk CreditAttribution: Michsk commented@chromix: i cant get your patch working. What version of unique field did you use? I want unique titles. So i first tried setting the title to be unique, since unique fields supports that. Didn't work.
Then i tried your suggestion to create a cck, title copy field, and set the unique check on that. Tried it. Didn't work.
Comment #42
Michsk CreditAttribution: Michsk commentedAh, i see we have a extra unique checkbox in the mapping page. Great, works as expected! Awesome. Let's work it out more, like multiple unique fields, and the unique title options. And then implement it in the feeds module. I love that this fix uses another module. This is how modules should work together. chromix++
Comment #43
sassafrass CreditAttribution: sassafrass commentedI installed the patch and the module Unique Field. And this works if I select one field as a unique target but not if I select multiple fields. Is that as intended?
Comment #44
chromix CreditAttribution: chromix commentedFrom comment #22 above:
It's definitely possible to use multiple fields, but it adds a decent amount of complexity.
Comment #45
jamesdixon CreditAttribution: jamesdixon commentedI can confirm #21 works to get unique target working for multiple columns. Here's my implementation as an example. I'm using hook_feeds_after_parse inside a custom module. This is a quick and dirty solution:
Afterwards you'd need to add a mapping from source 'guid' to GUID, and click the unique target checkbox.
Hopefully this helps someone.
Comment #46
ambientdrup CreditAttribution: ambientdrup commentedI've installed this patch and the Unique Fields module but on my Feeds importer Node processor mapping I still only see one unique column with a checkbox next to the GUID field. That existed before the patch. So this patch does not seem to be working for me.
-Trevor
Comment #47
macdee CreditAttribution: macdee commentedThe patch at #29 worked well for me.
@ambientdrup Did you "Choose the fields that should be unique" under "Unique Field restrictions" in admin/structure/types/manage/--your content type--?
Comment #48
ambientdrup CreditAttribution: ambientdrup commentedCorrect. Edit your content type, click on the Unique Field restrictions tab and then choose the fields via the checkboxes.
Best-
Trevor
Comment #49
Masala CreditAttribution: Masala commentedTell me understand please. To properly import I need to have a unique field 'Ubercart:' Model / SKU. In the standard settings do not checkmark the "unique" for SKU. This patch will help me? I installed #29 but did not see the changes.
I have no SKU fields in "Unique Field restrictions" in admin/structure/types/manage/product
Comment #50
ayalon CreditAttribution: ayalon commentedThe path in #22 has a bug:
The variable $content_type is not defined in this context. It should be $target.
Comment #51
rond49 CreditAttribution: rond49 commentedCan't one delete his/her own comment??
Comment #52
OldAccount CreditAttribution: OldAccount commentedThe patch in #29 worked for me, thank you!
I'm not sure if it's related, but I had to change my Title and Description source mappings to xpath expressions after installing the Unique Field module and installing this patch, title and description were no longer options in the Source drop-down. It still works, I just had to make that change. Either way, it's working and I'm ecstatic. :)
Comment #53
lunk rat CreditAttribution: lunk rat commentedDid this get committed to 7.x dev? I see this option now and I didn't patch anything.
Comment #54
lunk rat CreditAttribution: lunk rat commentedIgnore #53; I was just seeing it on the Title field.
Comment #55
a.ross CreditAttribution: a.ross commentedIt hasn't been committed yet. The title field can now be selected as a unique field, regardless of the unique_field settings.
Comment #56
a.ross CreditAttribution: a.ross commentedI've manually applied the patch of #29, and it appears to still be working just fine. Not sure if that could collide with the Title field optionally being unique before this patch though.
Comment #57
a.ross CreditAttribution: a.ross commentedAttaching properly formatted patch rolled against the current revision.
Comment #58
selim13 CreditAttribution: selim13 commentedThanks for the patch a.ross. I've tried it with 7.x-2.0-alpha4 and had pretty interesting results. Only one or part of nodes created on import, when unique_field module was enabled, even when no fields were marked as unique. The reason for me was, that unique_field_match_value returned aleady created node id, when GUID was provided as $target. So it just continuously updated the first one created node or I think so.
I've made realy quick fix that ensures that only field name taken from unique_field module will pass to unique_field_match_value.
I'm not sure if this is correct way, but it works for me.
Attaching patch (git diff) for 7.x-2.0-alpha4 but it should apply to current dev.
Sorry for my English =)
Comment #59
a.ross CreditAttribution: a.ross commentedYou generally shouldn't really map anything to GUID, because it is system managed. Also, are you sure you didn't forget to set the field "unique" in the mapper itself?
Anyway, I didn't have the issues you describe. In my case it was working just fine.
That said, I don't think your patch might break things, but it's probably redundant code.
Comment #60
litvinova_yana CreditAttribution: litvinova_yana commentedComment #7:
Super! A good decision.
Comment #61
johnv@litvinova_yana,
IMO comment #7-#11 describes the 'out-of-the-box' solution, provided by feeds.
However, it only works if the feeds is the only source of the objects. It won't work if you create an object manually, and amend/overwrite it with the feeds, since the GUID is not yet in the GUID-table.
In that case, comments/patches #22-#29 can be used.
Comment #62
Jason Dean CreditAttribution: Jason Dean commentedUnique Field module plus #29 is working well for me.
Now my challenge is to get this working with the feeds mapper for node_reference (References module). Using the example provided by johnv in this issue:
This shows how a node reference field can map to Feeds GUID in a 'normal' setup (i.e. when Unique Field module is not used).
But using Unique Field module and #29 patch, Feeds GUID is not set when Feed 1 is imported. So referencing by Feeds GUID in Feed 2 cannot be used.
I'm not sure whether this is strictly an issue for Feeds or References, but I guess references.feeds.inc would need to be modified to use the unique field defined by Unique Field module instead?
Comment #63
litvinova_yana CreditAttribution: litvinova_yana commented@johnv,
thanks for given explanation!
But I wrote about comment #7, because I liked logic of desicion, given in it, and it went well with my project.
Best regards.
Comment #64
OldAccount CreditAttribution: OldAccount commentedI've updated the patch in #29 for 7.x-2.0-alpha5. I've tested it but this is my very first patch (following these instructions) so please be nice if it's not correct. :)
Comment #65
a.ross CreditAttribution: a.ross commentedPatches should be rolled against -dev, which I already did in #58
Comment #66
OldAccount CreditAttribution: OldAccount commentedSorry, that's what I thought, but -dev doesn't show up as a branch option, I just have:
Comment #67
a.ross CreditAttribution: a.ross commentedWell, the 7.x-2.x-dev is just a release name. It comes from the 7.x-2.x branch, just like all 7.x-2.x releases (like 7.x-2.1). For stable releases, you should see tags corresponding to releases somewhere in the commit log.
Comment #68
selim13 CreditAttribution: selim13 commentedReply for #59.
Hello a.ross. Forgive me for long responce.
Feeds module was used to create nodes from XML file. XML file could be updated, so nodes need to be updated too. So GUID was mapped to unique identifier field of XML feed and was set as "unique", which worked fine.
I did't set it neither in content type settings nor in mapper. Just enabling unique_field module causes imports with "unique" GUID mappings to fail.
The problem is, that when GUID is mapped and marked as "unique" it passes to unique_field_match_value() which is nothing to do with guids but returns some value (in my case it is a full list of node id's). So importing enitity (node) is considered as exisiting and overrides the most earliest one.
Another possible problem is that sometimes unique mappings from other modules can pass as targets to unique_field_match_value(). This probably happens only when mappings have 'optional_unique' => TRUE, but actually not being handled for uniqueness and pass to default case in switch ($target) in existingEntityId(). As with GUID this returns full list of node id's in this case and fails import.
So there still should be some simple check, that the target passed to unique_field_match_value() is actually a field. Possibly simpler that mine, for example by checking "field_" prefix in target's name.
Again, sorry for my English =)
Comment #69
a.ross CreditAttribution: a.ross commentedHmm, I think I understand most of what you're saying. First of all, mapping to GUID and marking unique does work, but as far as I can remember you should avoid mapping anything to the GUID field, because Drupal normally manages it.
Now it seems that the previous patch in this issue conflicts with the "uniqueness" that is provided by Feeds natively, which is what I was afraid of. I would expect that it also conflicts with the title field (which can already be set as unique in the current revision of Feeds).
I guess then this issue needs more work.
Lastly, I don't think the check should be as simple as
preg_match('/^field_.+$/', $machine_name)
. The reason is simple; other modules may add fields that don't start with "field_"Comment #70
Robin Millette CreditAttribution: Robin Millette commenteda.ross said
I've been using feed api and then feeds for ages and generally needed to specify guid mapping myself. Say you're scrapping an html page with xpath parser, what do you think happens with the guid? I just grepped the source and can't see any special handling of guid. Maybe you're thinking of the md5 hash used internally?
On the other hand, if you could point to anywhere explaining how "Drupal manages the GUID", I would be very grateful.
Comment #71
a.ross CreditAttribution: a.ross commentedI've read it before in posts on d.o., though I can't find it right now. However, I just looked in the database, and there is no GUID column in the node table, only in the feed table, so I guess it's specific to the feeds module.
I guess then I was wrong, so I take that back. My apologies. However, it doesn't affect this issue at all, in that the unique_fields integration collides with Feeds' native handling of unique fields.
Comment #72
iAugur CreditAttribution: iAugur commented#64 worked for me! Thanks.
I would be keen to see this make it into the next release of the module or something along these lines that supports unique fields.
We wanted a unique field on Product Displays on a Commerce site and using GUID is OK if you re-use the same Feed Importer to update nodes. But we had one that imported all node values initially and another to update statuses. Using the updater created new nodes as the GUID is unique only in the context of the feed importer not the target.
This pathc and http://drupal.org/project/unique_field was the solution thanks.
GUID is used by Feeds to track mapping for a specific importer not across importers.
There is a similar discussion here #1233142: remove ->condition('feed_nid', $source->feed_nid) from existingEntityId().
Comment #73
jjclint CreditAttribution: jjclint commentedThe patch in #64 didn't help me.
I have two similar unattached importers I've mapped the title in each to be unique with the help of unique field module (ver. 7.x-1.0-rc1). I crawl a url with an xpath fetcher and create a node from it with one importer, I then xpath a different url that has the exact same title with the second importer, but instead of updating the node fields like it should it just creates a new node with the same title.
Comment #74
star-szrRerolled #64 against latest dev.
Comment #75
star-szrThis patch removes the unnecessary
$info = field_info_field($name)
line inside themodule_exists()
, since it's already set just inside the foreach.Comment #76
twistor CreditAttribution: twistor commentedThis patch implements unique fields on behalf of the unique_field module. hook_feeds_node_processor_unique() and the 7.x alternative needs to be implemented first. Then this patch should go into the unique_field module.
Comment #77
star-szrI agree, this will never get committed as is. Here's an attempt at adding hook_feeds_node_processor_unique() - @twistor I wasn't sure what you meant by the 7.x alternative.
Since the proposed hook is in the early stages, I'm not going to post a patch in the unique_field issue queue yet. Attached are two separate patches, one for Feeds and one for Unique field to show an example hook implementation. On the importing side this should work almost the same as #64 or #74/75, but I've corrected the call to
unique_field_match_value()
to work per content type if that's how you've configured unique_field.In this patch, the hook looks like this:
I decided to pass
$this->config
from FeedsNodeProcessor, otherwise the hook implementer has to use$source->importer->processor->config['content_type']
if they want to access the content type. I couldn't see the full $source object via devel dpm() either. The 'feeds_source' query inFeedsNodeProcessor::existingEntityId()
uses$this->config['content_type']
so I thought it was fair, but we could certainly drop the argument as well.Right now the Feeds patch just shows/hides "Unique" checkboxes in the Feeds mapping UI based on whether any modules implement this hook or not. I'm not sure how to improve this other than creating a separate hook. The previous patch directly on top of Feeds (#74) was able to show/hide checkboxes based on what fields were configured with unique_field.
Comment #78
star-szrI probably should have named the first patch feeds-unique_field_targets-661606-77.patch to be a bit clearer. The first patch is against Feeds, second is against Unique field.
The implementation of hook_feeds_node_processor_unique() should have a docblock too, of course.
Comment #79
jjclint CreditAttribution: jjclint commentedApplied both patches and it's still creating 2 different nodes with the exact same title when I try to update using 2 different importers.
Comment #80
iAugur CreditAttribution: iAugur commentedJJClint: Just a thought - have you checked the unique target checkbox on the mapping screen for your unique field?
So long as that is checked and the field is unique it should update not duplicate.
Comment #81
jjclint CreditAttribution: jjclint commented@iAugur: I've ticked the check boxes for unique title on the mappers UI screen and it's also a unique field in the content screen. The only other thing I can think of is that maybe something is wrong because I'm using the feeds tamper module as well, but when I go into the sql tables for my unique fields nothing seems wrong.
I'm really banging my head hard on this one.
p.s
After I've updated feeds to the latest dev version and applied the patches, unique check boxes appeared on the mappers UI screen next to fields which I haven't setup to be unique via the unique module (content/structure/unique_field), I'm assuming it's the doing of my feeds update to latest dev version.
Comment #82
a.ross CreditAttribution: a.ross commentedYou can already set the title field as unique in the latest dev. No need to apply this patch (#75) if that's all you require. Also the patch needs work, or perhaps another approach like Cottser did in #77.
Comment #83
jjclint CreditAttribution: jjclint commented@a.ross:
A. I was using the patches in #77
B. The built-in unique field option for title that comes with feeds latest dev. version doesn't work when one uses two different importers for the same node.
Example: Importer 1 should create the node and importer 2 should update some of it's fields, instead importer 2 creates a whole new node with the same title field.
EDIT:
I've just tried to achieve the above result with a fresh install of drupal 7.x with the latest dev. version of feeds (unpatched), without unique field module, using 2 different xpath crawlers with a unique title field and failed (again).
Comment #84
star-szr@jjclint - That sounds pretty straightforward and Feeds should be able to accomplish that out of the box. As @a.ross said you may want to try without any patches. Check title as unique in both your importers, and double check your node processor settings under Update existing nodes.
From FeedsNodeProcessor.inc:
Comment #85
jjclint CreditAttribution: jjclint commented@Cottser: I've checked all of the things you've mentioned and I still can't get it to work. I'm not sure what's wrong, I can upload my test site to a live server, but not for long.
edit:
Checking the sql tables for node I've noticed that one of the nodes which I thought had the same title actually has three
tags that got xpathed into the title field.
trimming the white space around the title field fixed my problem, anyway sorry for the fuss.
Comment #86
scottrigbyI think #78 needs some work.
A hook to set
optional_unique
infield_feeds_processor_targets_alter()
makes sense, but it should probably allow the implementation to specify which field (on the specific entity_type/bundle_name) should be unique, rather than allowing any of the fields to be unique if the hook is implemented.About an implementation patch… it's not really necessary for this patch to Feeds - but as a proof of concept, it would be better to integrate with the Field validation module, as Unique fields only supports nodes. See:
#1094582-10: Support (Entity) Fields in general, not only on nodes
#1319396: Join forces with Field Validation
Comment #87
scottrigbyOk so I took a stab at this. Two patches attached, one for Feeds and the other for Field validation.
In the Feeds patch I added a more generic
hook_feeds_existing_entity_id()
toFeedsProcessor::existingEntityId()
for the existing ID part of this issue, which should handle any entity type rather than just nodes.In the other attached Field validation patch, I'm implementing this new hook.
For the unique target option, I don't think we need any additional hook, since we already have
hook_feeds_processor_targets_alter()
. Instead I'm also implementing that hook in the Field validation module (with another helper function for that module, which simplifies these hook implementations).I can add this patch to the Field validation queue tomorrow, unless anyone sees a problem with these changes I'm not considering?
Comment #89
johnvIMO this should be in a separate field_validation.feeds.inc file.
Comment #90
star-szr@scottrigby - At a glance the patch looks good. After throwing together my patch I also thought it should work for more than just nodes, so I'm glad you went that way. I ended up creating a small custom module that uses hook_node_validate(), hook_node_presave() and hook_feeds_presave() to validate and skip duplicates for node functions and Feeds imports.
Comment #91
Jason Dean CreditAttribution: Jason Dean commentedI've been using Unique Fields module and patch from #57 successfully ... until I tried running my feeds importer via Drush.
The patch in the Drush issue (also #57, strangely) runs the importer from Drush fine, but it doesn't respect the unique field set in the importer mapping. This results in lots of duplicates.
Looking at the code for the Drush method, it's just using feeds_batch Feeds API function:
So I can't see why it doesn't respect the mapped unique field?
Comment #92
scottrigbyThanks for the feedback so far!
@johnv - About the field_validation patch moving Feeds hook implementations to an include - that sounds fine, except for the
field_validation_get_fields()
helper function, which IMO is generally and should be in the .module file no matter what. But where it's contrib hook implementations live is probably best left up to that module maintainer's preference, so I just added them to it's .module file until told otherwise =)Although… are you suggesting we use path discovery (like CTools plugins) when invoking
feeds_existing_entity_id()
? If so, AFAIK that wouldn't gain us anything here.@Cottser - Cool thanks!
@pushka (using Unique Fields) & @Cottser (re your custom module) - Yeah the nice thing about this new Feeds hook is… since Feeds should not be responsible for enforcing unique field values, it doesn't care how that gets accomplished. Any module (whether Field validation, Unique Fields, or other validation modules (contrib or custom) can accomplish this by implementing two hooks:
hook_feeds_processor_targets_alter()
for specifying which field you are validating as unique, and add the unique target option for selection on the mapper form (the Field validation patch from #87 is just an example, but one that works for validating using that module)hook_feeds_existing_entity_id()
for providing the existing entity ID for the specified unique field during importHow everyone following this issue can help
So the Feeds patch in #87 is what mainly needs review now. The main differences between that (ultimately, a patch to add
hook_feeds_existing_entity_id()
) and the Feeds patch in #77 are:$entity_type
, because (unlike the node-specific version in #77)$config
will not always contain the entity type (though I kept $config anyway, since it may be generally useful for other implementations)Comment #93
rbosscher CreditAttribution: rbosscher commentedscottrigby, thank you very much for #87
Applied them both succesfully. And it worked like charm
I have a specific usecase where the SKU of a product is not really the unique identifier (the SKU can change from the receiving part) so i created a field product_id wich i can set now as an unique field. So this patch is working perfectly.
I just needed to change in the commerce_feeds module. so still working on the best solution. but it works right now.
I hope this gets implementen quick :)
Thanks again!
Comment #94
scottrigby@rbosscher glad to hear it!
By the way, I also added a companion issue in the Field validation queue: #1705386: Feeds integration (Field validation)
Comment #95
twistor CreditAttribution: twistor commentedGlad to see this coming along!
Ideas/Suggestions:
$targets[$field_name]['optional_unique'] = TRUE;
. The problem is that many modules set something like,$targets[$field_name:column_key]
That said, there needs to be ahook_feeds_processor_targets()
where targets are added, then the alter makes a lot more sense. This should be a separate issue.hook_feeds_existing_entity_id()
. This would allow some control over ordering, and it would allow us to only call as many functions as necessary until we get a value.If the callback idea seems way off, I'm not convinced, and would like to hear objections.
Comment #96
twistor CreditAttribution: twistor commentedAccompanying issue. #1706026: Create a hook_feeds_processor_targets()..
Comment #97
Jason Dean CreditAttribution: Jason Dean commentedIn testing #87 (and moving away from Unique Fields method), I tried this:
So far so good, but importing (using either Feeds node import button or Drush) doesn't update any nodes; it imports new duplicates every time.
Perhaps I'm being a bit optimistic expecting my importer (originally configured using Unique Fields method) to just adapt? Or maybe the new method isn't fully implemented for Feeds node processing yet?
Great to see progress on this issue though - thanks!
Comment #98
tmsimont CreditAttribution: tmsimont commentedis the patch in 87 applied to dev or alpha 5? It doesn't seem to work on the dev branch. I'd try to patch it myself but there seems to be significant differences between the dev branch and alpha 5. Can you clarify? I'd like to get it in dev if possible because I'm also looking to get this patch working: http://drupal.org/node/1454666#comment-6168760
Comment #99
scottrigby@pushka & @tmsimont:
The patches in #87 are against the 7.x-2.x branches of each module:
For me this imports appropriately (does not duplicate unique targets) on new entity imports. I'm also patching my Feeds 7.x-2.x with #1033202-69: [Meta] Generic entity processor, but afaik this patch shouldn't rely on that at all.
Comment #100
tmsimont CreditAttribution: tmsimont commentedOk thanks -- I got the modules installed, applied both patches sucessfuly, set up an integer field to be unique, was able to check the box on the importer config page to make it a "unique target," but if i run the import twice, the second run duplicates all of my nodes. I have the importer set to "replace nodes" so I'm not sure why it would duplicate all of them... I don't think this is working... Could it be because I'm using an integer field?
Comment #101
scottrigby@tmsimont mine is also an integer field, so I don't think that would be the problem.
Comment #102
scottrigby@twistor re #95, separating these issues:
hook_feeds_processor_targets_alter()
(since the update will be backwards compatible anyway!). Though that shouldn't hold up this issue, right?hook_feeds_existing_entity_id()
, I don't completely understand what you mean by a "unique callbacks" array key in the config (you don't mean that the callbacks would be selected on the processor settings form?).I can definitely see wanting to avoid invoking another hook inside a method, and normally I would have just extended the class, except that the IDs then wouldn't apply to other extensions of that class, like
FeedsNodeProcessor
. I guess Feeds could define another CTools plugin, and expect implementing modules to define their own class with a specific method (or callback) then get the first unique ID that's returned? Something like:Or just stick with a traditional hook invocation, like in the patch as-is. What do you think? Is this making sense with what you were trying to say?
Comment #103
tmsimont CreditAttribution: tmsimont commented@scottrigby -- it looks like the "bundle name" is not assigned properly in the
field_validation_feeds_existing_entity_id()
code in the field_validation.module patch.I've got an importer named "moe_data_importer"
When I run it,
$rules_fields_unique
is an empty array afterfield_validation_get_fields
takes "node" and "moe_data_importer" as its first two arguments ($entity_type
and$bundle_name
).The "bundle" should be the node type, not the machine name of the importer. It's probably working for you if you have given your importer the same machine name as your node type.
The attached patch fixes this by using
$config['content_type']
instead of$source->id
(it patches the current 7.x-2.x branch instead of patching on top of your patch, it should be used as a replacement to the patch you had in #87)
Comment #104
scottrigby@tmsimont just to clarify, you're talking about the example Field validation patch (not the proposed Feeds patch).
However, you're right! I had named my importer the same as my bundle name, and saw that in
$source->id
.Unfortunately,
$config['content_type']
only applies to nodes, so that's not an acceptable solution.@twistor, so this is kind of a blocker IMO…
Within
existingEntityId()
we need to get the bundle somehow, and pass that as a param in the alter function (or the plugin invocation per #102, or some other method to allow other modules to provide the unique ID during processing).But currently we have no way to reliably get the bundle type from there yet - unless we assume the bundle type must be set in $config (or defined by extension classes manually somehow) and made available in some definitive way.
Proposal: Ok how about adding
FeedsProcessor::bundleType()
(an equivalent ofentityType()
).FeedsProcessorNode::bundleType()
can return$config['content_type']
, and other entity modules extendingFeedsProcessor
can add a bundle selection to the settings form, and return their own $config option in their own implementation ofbundleType()
. That way inFeedsProcessor::existingEntityId()
we can just do$bundle_name = $this->bundleType();
.Do you think that makes sense? If so I can whip up a patch that covers all Feeds-supported entities.
(as a sidenote, the patch in #1033202-69: [Meta] Generic entity processor adds the bundle type selection to the Feeds processor config form automatically for entity modules that set their bundle type column to
'required' => TRUE
in their Entity metadata info. That patch would just need to be extended to addFeedsEntityProcessor::bundleType()
you think the new method idea is appropriate?).Comment #105
tmsimont CreditAttribution: tmsimont commentedi like the
FeedsProcessor::bundleType()
idea -- i'm not sure i fully understand your sidenote.. but i like thebundleType()
idea :)Comment #106
twistor CreditAttribution: twistor commented#1711648: Abstract bundle handling. this is turning out to be a very fruitful issue.
@scottrigby
What I was proposing, poorly, was this:
Also, the above code was borrowed from the patch in #103. It illustrates something else I mentioned. Unique settings should not be set for every field that field_validation is aware of. The contents of $targets should be checked for possible unique fields. That's the major reason I proposed the hook_feeds_processor_targets() patch.
It's similar in style to the set_target callbacks we already use, but it allows for multiple callbacks.
Comment #107
scottrigby@twistor - ah ok… that makes more sense actually, because only optional_unique targets will ever need to be checked.
Though I don't think we need to set an array of unique callbacks, just one . Because when invoking, even if multiple implementations of
hook_feeds_processor_targets_alter()
alter the same target, when invoking we'd still have to loop over them anyway and break at the first one we get. So each alter may as well just add/overwrite a single$targets[$field_name]['unique_callback']
and the processor can call whichever one ends up winning =)But we still need to pass the bundle to our callback, so it can do a proper field check :p But how? Suggestions so far:
FeedsProcessor::bundleName()
).FeedsProcessor::existingEntityId()
- we could expect implementations ofhook_feeds_processor_targets_alter()
to also pass the bundle name right back through $targets by reference, like:$targets[$field_name]['bundle_name'] = $bundle_name;
(that may seem redundant, but otherwise we can't reliably ensure we have the bundle name when invoking the callback.I'm attaching a patch for #2 (only because #1 would be more refactoring in Feeds module, and I would like your input on that direction before working on that). Do you have other ideas about definitively getting the bundle name here?
Here's an updated example for the hook implementation (with the bundle name issue in mind), to match the attached Feeds patch:
Comment #108
scottrigby@twistor, sorry I somehow missed your link to he new issue: #1711648: Abstract bundle handling..
I guess this issue will now be dependent on that one, and we can sort out how the bundle name method should best be implemented there :)
Comment #109
scottrigbyAdding the Field validation example from #107 here for reference. Will still follow up on the
bundleName()
method in the other issue.Comment #110
g089h515r806 CreditAttribution: g089h515r806 commentedI have test latest patch from #107 and #109,
Then i have the same issue with #97, "but importing doesn't update any nodes; it imports new duplicates every time."
Comment #111
g089h515r806 CreditAttribution: g089h515r806 commentedHere are latst patch.
1, I think that we do not need "unique_callbacks" at here,"unique_callback" is enough.What we need is there is a callback function which could get the entity_id for us.One callback function is enough at here.We do no need an array.
2,I have remove the function of field_validation_get_fields, we could use :
and
directly. We could add more conditions for ctools_export_load_object at here.
3, Put the code of feeds integration into a seperate file "field_validation.feeds.inc", we could earn a little performance at here.
4 Another small change is use '$unique_rule->col' instead of 'value':
The patch works at our drupal sites.
Comment #112
7wonders CreditAttribution: 7wonders commented#111 is working good for me
Comment #113
Jason Dean CreditAttribution: Jason Dean commentedYep #111 working well for me too :)
Comment #114
twistor CreditAttribution: twistor commentedThe field_validation code needs to go in the field_validation module.
Re: array of unique callbacks.
This IS needed, otherwise, multiple unique callbacks will overwrite eachother. It doesn't complicate the implementation at all. We just loop over the callback and break on the first one that returns a value.
Comment #115
g089h515r806 CreditAttribution: g089h515r806 commentedlatest patch, works.
Comment #116
g089h515r806 CreditAttribution: g089h515r806 commentedTo test the patch "feeds-unique-target-661606-115.patch" with field validation, you could download "field_validation.feeds.inc"
at http://drupal.org/node/1705386#comment-6481224 .
Comment #117
Anonymous (not verified) CreditAttribution: Anonymous commentedIn case anyone's using #22 for D6, there seems to be an issue when using Feeds Tamper. My CCK field accepts multiple values, but each should be unique. I used explode, and this caused the unique check to fail. The nodes are uploaded anyway.
Comment #118
zeezhao CreditAttribution: zeezhao commentedPlease can someone clarify the right code to use? I currently installed:
field_validation-7.x-2.x-dev [2012-10-07 -- "7.x-2.1+7-dev" - the latest]
feeds-7.x-2.0-alpha7 + feeds-unique-target-661606-115.patch
unique_field-7.x-1.0-rc1
[edit]
-Created new rule via: admin/structure/field_validation/add when using: Entity Type:
Node, Bundle Name: Product [from ubercart]
- Can see now see the extra fields
- But unique field still got imported.
- Using "Unique Values" with "global scope".
Thanks
Comment #119
Jason Dean CreditAttribution: Jason Dean commented@zeezhao
Not sure if this is the answer, but I couldn't select the unique field in my importer mappings when using current Feeds feeds-7.x-2.0-alpha7.
http://drupal.org/node/1705386#comment-6676640
Also, not sure you need Field Validation and Unique Field modules. Field Validation is the one to use with the recent patches here.
Comment #120
scottrigby@zeezhao and @pushka - can you test #115 with feeds-7.x-2.x-dev? It is not intended for feeds-7.x-2.0-alpha7.
Comment #121
zeezhao CreditAttribution: zeezhao commented@scottrigby - thanks for your help.
Now got a step further, getting error "PDOException: SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry..."
but I was expecting it to update the duplicate node because I have the following on "Settings for Node processor": "Update existing nodes (slower than replacing them) "
Please confirm if I still need to use unique_field module, and set the field on the content type, as I also did this. When I did not, import loaded duplicate anyway. Thanks
Note: I am also using feeds_tamper & custom_feeds_tamper_term_hierarchy modules on a totally different field [ for taxonomy hierarchy]
Comment #122
g089h515r806 CreditAttribution: g089h515r806 commentedYou have to increase the weight of "field validation" module in systm table. Then clear the cache.
In feeds alpha7, function field_feeds_processor_targets_alter has been splitted into several functios:
number_feeds_processor_targets_alter,
text_feeds_processor_targets_alter,
they run after field_validation_feeds_processor_targets_alter.
We should make sure that field_validation_feeds_processor_targets_alter run at the last place.
I try to use following code:
It does not work.
Comment #123
twistor CreditAttribution: twistor commentedThe above is why we need, #1706026: Create a hook_feeds_processor_targets().
Comment #124
W.M. CreditAttribution: W.M. commented@TamboWeb
Does the GUID has to be a Drupal field ?! Or is it a feeds module thing ?! Thanks.
Comment #125
SGhosh CreditAttribution: SGhosh commentedI am using - 7.x-2.0-alpha7. I need to use a custom field as unique field for mapping. Which is the patch i should be using? Any other module to be used along with it?
The patch in this documentation - Use field other than nid as unique node field in feeds for 7.x-2.x - applies on a field.inc file in feeds/mappers which doesn't exist there.
Comment #126
scottrigby@SGhosh please see #120
Comment #127
agileadamI have an updated version of the #115 patch from g089h515r806. This should apply cleanly to the latest version of 7.x-2.x-dev.
This updated patch eliminates some PHP notices coming from undefined index for $targets[$target]['unique_callbacks'] and $targets[$target]['bundle_name']. I have done minimal testing (few changes from #115; working fine for me) so please test it out.
Note: I am using the same code in 7.x-2.0-alpha7 but the line numbers are different. I put the default switch case just after the guid case. It should be around line 605.
Here's the diff between #115 and #127 (also, the line numbers changed due to other code changes in dev branch):
Comment #128
johnvIn the past, I have patch #22 applied, and installed unique_field for that. Later patches only apply to field_validation.
I can't get it working again to unique_field again. Any help? Or should I switch from unique_field to field_validation?Thanks.
Comment #129
gaurav.pahuja CreditAttribution: gaurav.pahuja commentedThanks #22 chromix for patch for 6.x. But this patch is not working if I try to make title as unique field but not some CCK created field.
Comment #130
idflood CreditAttribution: idflood commentedI quickly tested the patch in #127 and it seems to work nicely. The patch applies cleanly and in these 12 lines I don't see any code formatting issue.
@gaurav.pahuja19... : patch are usually made for the latest version and then backported if possible.
Comment #131
presleyd CreditAttribution: presleyd commentedI applied the patch in 127 on current dev. I used Field Validation to set a unique field on the content type. When I create a new node of that type from the node/add form, the validator successfully errors on a duplicate in that field. In the Feeds mapper I see the new UI option to make that field a unique target and used it to set that field as unique.
When I run the feeds importer though, duplicates are still being created. The originals were not added via Feeds but I thought that was the point of this patch.
Comment #132
liquidcms CreditAttribution: liquidcms commentedwow.. this is a pretty long thread. i have read through all of this (but will re-read next) but wondering why this is so complicated. perhaps i am missing something here:
- why is any other module required besides Feeds? i do not understand why field validation or unique field should be required for this basic task.
- why are ALL fields (not just title, nid) not selectable as unique?
i must be missing something here as it seems like you should be able to select any fields you want to create a unique key.
also, can someone explain how guid is used. i see it is an autoincrement field in the feeds_item table but i am able to assign it something like title. as per some of the suggestions above i tried creating a hash of title+fieldX and setting to guid (since i am able to set guid as unique) but this did not work (will look at this again).
Comment #133
scottrigby@liquidcms you can – there is a hook
hook_feeds_processor_targets_alter()
whichUnique fieldsField validation is simply implementing. You can write a module that implements that for whichever fields you wish. It's not a good idea to make that an option for most fields out of the box however, because many will have no guarantee of being unique.Comment #134
liquidcms CreditAttribution: liquidcms commentednope, my bad. i set up the map to guid field incorrectly. so then this does work.. simply. so again not sure what this thread is about.. lol..
so this is what i did:
- added a Source: unique (note: unique is not a column in my CSV)
- map this to Target: GUID and set as unique
in custom module:
i think this should also be doable with Feeds Tamper but couldn't get that to work.
Comment #135
johnv@liquidcms, indeed, you could use GUID to make any field unique, but ONLY for nodes created by this very feed.
It does not work your nodes are originated from another source (initial upload, another feeds, manually created by a user).
See #11, #14.
Comment #136
liquidcms CreditAttribution: liquidcms commented@johnv, ahh, got it. thanks for the clarification.
Comment #137
oryx CreditAttribution: oryx commentedHello everyone, I read this thread with the greatest attention since I am having the exact same need (use feeds to update nodes which were not created by feeds). However, I couldn't figure out how to make the patch work : I'm using 7.x-2.x-dev, I applied patch #127, but where should I put the file field_validation.feeds.inc generated by field_validation-feeds-unique-target-661606-111.patch (#111)? In the plugins directory?
Thanks for your help !
Comment #138
marktheshark CreditAttribution: marktheshark commentedIsn't it time there was a general consensus about how to update entities not created via Feeds?
I'm guessing this is a use case for many people.
The documentation for this probably needs updating.
Please advise which patch to use, so that I can assist in testing, thank you.
Comment #139
emilyf CreditAttribution: emilyf commentedSpent a few hours trying to go through this thread and test everything out so I sure hope this benefits others. Here is what I had to do to get this working:
- Using Field Validation 7-x.2-3 release for this testing (the dev version gave me a WSOD, but I will get back to that and try to do a patch)
- Then as mentioned in #116 (http://drupal.org/node/661606#comment-6481228), I downloaded the zip file from the field validation issue queue and added it to my field validation module directory (http://drupal.org/node/1705386#comment-6481224), direct link to this download is http://drupal.org/files/field_validation.feeds_.zip
- I attempted to apply the feeds patch from #111 but it failed, so I manually patched my plugins/FeedsProcessor.inc b/plugins/FeedsProcessor.inc
Doing all of this worked for me.
For anyone needing other help on how to actually use all of this, once you do the above and enable field validation module, go into the content type > Manage Fields that you want to set a unique field on. Now you will have a 'validate' link next to every field. click that for the field you want to change. Then apply the unique validator. Now create your feed importer. When you map a source to this field, you will have a unique checkbox. Note that as with all of these, by default it will be off. You not only have to click "update" and check the box, but if you don't then save the overall screen it will not retain your changes.
Also for anyone wondering, I am using the generic entity processor feeds branch (which is likely why the patch failed for me). So it is working with that too. I am going to see if I can create some more patches that are up to date and will post back.
Comment #140
marktheshark CreditAttribution: marktheshark commentedCan you explain your use case? Are you using Feeds to update pre-existing entities? Thank you
Comment #141
emilyf CreditAttribution: emilyf commentedActually, the feeds patch in #111 works and applies cleanly on both the current feeds 7.x-2.x branch and the 1033202-entity-processor branch.
Comment #142
emilyf CreditAttribution: emilyf commentedMy use case:
I have a content type called "show". In this content type, I have a textfield that stores a value. This particular textfield needs to be globally unique in my system (imagine it like a serial number). Let's call this text field "field_serial_number". So in order to tell Drupal this always has to be unique, I installed Field Validation module. I then set up field validation module as I described in #139
Now, in addition to this, I have an RSS feed from another website. That RSS feed stores some additional information that I want to import into my Drupal system. In fact, I want to overwrite the Title field of my content type "show" with the Title element of that RSS feed. Trouble is, I don't want it to create new nodes. This RSS feed has a tag that has a matching number to my field_serial_number. So I needed a way for my feed importer to know this already existed in my system. This is what this thread is all about.
So, I then patched feeds with #111 and then followed instructions here to update field_validation inc file: http://drupal.org/node/1705386#comment-6481224
Now I set up a new feed importer, and when I map the RSS source field that contains my matching serial number to my field_serial_number I now get a "unique" flag on this field. I check that off, save the feed importer. In addition, you do not set a GUID on this new feed mapper, you don't need it.
Now, when I run my import, when the RSS feed serial number matches an existing serial number in my field_serial_number, that node gets updated. No new nodes are created.
Comment #143
marktheshark CreditAttribution: marktheshark commentedThanks for your thorough explanation.
I will try to follow your advice for updating stock values for pre-existing product entities, with the SKU being the unique field.
Do you have any handling for nonexistent serial numbers? Do they get ignored, or do they trigger the creation of new nodes?
Comment #144
emilyf CreditAttribution: emilyf commentedThey trigger the creation of new nodes in my use case.
One other note: I have now tested this workflow using both the feed importer node processor and the entity processor node (that comes with the new feeds branch related to #1033202: [Meta] Generic entity processor) with success on both.
Comment #145
marktheshark CreditAttribution: marktheshark commentedIn my case the nonexistent field values will need to be skipped... I'll have to look into that.
Comment #146
emilyf CreditAttribution: emilyf commented@marktheshark - you can probably just use feeds tamper for that - there is the option in that to skip upon a certain type of match. see the documentation.
Comment #147
marktheshark CreditAttribution: marktheshark commentedThanks, there are also more ideas in this thread for anyone interested in the same functionality (i.e. update only, do not create new entities).
Comment #148
marktheshark CreditAttribution: marktheshark commentedI tried your steps, but in the end they didn't solve my problem.
Turns out Drupal Commerce Products somehow ensure the uniqueness of the Product SKU and the option in the importer to mark as unique is already there.
Actually installing Field Validation and the patches broke the form for enabling the uniqueness giving a 500 error code in the Ajax request.
I reverted everything back to how it was, marked the SKU as unique and then the import worked, updating the stock levels of the desired products.
Now my only remaining problem is how to force updates only, for which I will look into Feeds Tamper, or some patch that will expose this option in the importer form.
Thanks for your feedback.
Comment #149
Summit CreditAttribution: Summit commentedHi @marktheshark, interested in your findings!
Greetings, Martijn
Comment #150
emilyf CreditAttribution: emilyf commented@marktheshark - I have found a use case where I need the same thing as you, how to force updates only. Have you done any further research on this?
Comment #151
marktheshark CreditAttribution: marktheshark commentedWell, I was successful in updating commerce products, however the processor I was using (Commerce Products Multi) currently is losing all images attached to the entity being updated...
So, still some stuff to fix.
As for the ability to update only, I added the definition of the SKIP_NEW option and modified the Processor to skip creation in the processing loop when the entity_id was found not to already exist on the system.
Comment #152
marktheshark CreditAttribution: marktheshark commentedI have been asking this around in many places, but there is no answer:
Is anybody else losing images associated with entities when updating them via Feeds?
I can't be the only one having this problem...
Comment #153
emilyf CreditAttribution: emilyf commented@marktheshark - can you post a patch or added code for you skip_new option with the processor...i would definitely find this useful to reuse and maybe we can start to work on getting something committed...sounds like there are quite a few people on this thread who might benefit from that.
Comment #154
marktheshark CreditAttribution: marktheshark commentedI uninstalled the commerce feeds multitype module and relied on plain commerce feeds. I am attaching the modified files for your viewing (search for FEEDS_SKIP_NEW to find the changes easily), I will elaborate on the changes tomorrow, but they are basically this patch adapted to the Drupal Commerce Products plugin.
I'm very happy that the changes worked, I was able to update multiple types of products without actually needing the multi plugin, all nonexistent SKUs were skipped and no nasty side-effects. :-)
Comment #155
emilyf CreditAttribution: emilyf commented@marktheshark here is another alternative...
Attached is a feeds tamper that should do what me and marktheshark have been discussing. To use:
- put this file in your feeds_tamper/plugins folder
- You need latest dev version of feeds_tamper, clear cache
- rename file to entity_skip_new.inc
Here's how it works:
- apply it to the field that you need to match on. So if you are lining up an element in a feed with field_serial_number, put this tamper on that field.
- if it does not see a matching value then it does not create a new node
- if it sees a matching value it will update the node
Also, I had to choose 'skip hash check' on my feed.
Would love some feedback on if this works for others
Comment #155.0
emilyf CreditAttribution: emilyf commentedUpdate issue with d.o standard issue template
Comment #155.1
johnvAdded quicklink to comments with latest patches.
Comment #156
Michsk CreditAttribution: Michsk commented@151 for the lost images when updating you will have to check https://drupal.org/node/1107522 and https://drupal.org/node/2049341
Comment #157
miltonsp CreditAttribution: miltonsp commentedSee #159.
Comment #158
kingandy CreditAttribution: kingandy commentedPatch in #127 worked for me by itself, the current (7.x-2.3) release of Field Validation already has the feeds API hooks implemented so there was no need to patch anything else. Just installed FV and configured a unique condition, and bam.
Comment #159
miltonsp CreditAttribution: miltonsp commentedIn reference to:
Currently you can choose any field as a unique target using the GUID mapping element. This method has the following limitation:
- this GUID is specific for each importer, so you cannot use multiple importers to update the same node. They will double up.
Note that the Title (Target Field) has no such limitation and will be unique.
If you have another field that is unique (field_xyz), you can do the following:
1. Install 'field_validation' module.
2. Install patch #127 to the 'feeds' module.
3. Add a validation rule to field_xyz to enforce uniqueness.
4. In the Feeds/Node-Processor/Mapping screen mark the field_xyz target configuration as unique. Unmark uniquenes for the others.
5. You are done. Do the import with multiple mappers without creating duplicates.
Comment #160
Jason Dean CreditAttribution: Jason Dean commented@emilyf
Your Feeds Tamper plugin #155 is working perfectly for me. I haven't ticked the option to skip hash check and it seems fine so far...
Thanks a lot! :)
Comment #160.0
Jason Dean CreditAttribution: Jason Dean commentedAdded a more non-technical introduction.
Comment #161
JayShoe CreditAttribution: JayShoe commentedThis discussion was started on December 16, 2009. It's no November 11, 2013. Is there any commit or dev version that incorporates some or all of these concepts?
My use case is simple, I have a "VenueID' as a custom field in my content type venues. And I have a VenueID in my feed import (csv). I need to either add or update existing nodes based on this match.
If this isn't available in any committed version - I'll try the patches. But it seems like this use case is very common.
Comment #162
mstef CreditAttribution: mstef commentedPatch is great - the only issue I have, and I'm not sure if this is the correct thread to add this to, is that say your entity has 10 fields and 1 primary/unique key and you have 100 records fully inserted in to the database.
Then, you use Feeds to import data via a CSV that contains the primary-key/unique-field and just one other column/field that you want to update. All records will be updated with that one column/field but the other 9 fields will return to their default value since they are not present in the update import. I think the Entity processor needs to be changed so then when an update is being done, ONLY the present fields are updated, and the others are left as they stand (ie, merge verses update).
Comment #163
mstef CreditAttribution: mstef commentedI also can't seem to find any difference in FeedsProcessor.inc between "replace" and "update".
Comment #164
mstef CreditAttribution: mstef commentedFor those looking for unique support with Data - I wrote a small module that takes advantage of this patch and provides the integration: #2174373: Add support for Feeds "unique" field updating/replacing.
Comment #165
twistor CreditAttribution: twistor commentedThis needs to be updated. FeedsProcessor now provides a bundle() method, so we don't need the $bundle_name hack.
This also needs tests, at least one.
Comment #166
joachim CreditAttribution: joachim commentedAFAICT this is adding new properties to hook_feeds_processor_targets_alter(). Therefore this needs to be documented in the api.php file.
Comment #167
bradjones1This still needs tests but this is re-rolled to incorporate feedback from #165.
Also re: #166, we're not changing the function signature of hook_feeds_processor_targets_alter(), but we do define a signature for a callback, so not sure if that needs to be documented anywhere.
Comment #168
twistor CreditAttribution: twistor commentedThe docs need to be updated, as well as, an example callback provided in feeds.api.php.
Comment #169
AnybodyAs already stated out in #161 it would be nice to see some progress here and finally finish this thread after FIVE years ;). I'll set this to needs review. Who's the right person to update the docs and do further checks?
Comment #170
MegaChriz CreditAttribution: MegaChriz commentedI've started with updating the docs and writing a test. Attached is what I currently got. The test doesn't pass yet and I haven't figured out yet why.
From the previous patch I also changed this:
To this:
This way callbacks that live in classes (e.g.,
Test::callback()
) also work instead of only procedural functions (e.g.,callback()
).Comment #171
MegaChriz CreditAttribution: MegaChriz commentedI forgot to add a reference to the test FeedsMapperUniqueTestCase in the file feeds.info.
New patch.
Comment #173
WillowDigit CreditAttribution: WillowDigit commentedI had this patch working at one stage, and now it does not. Using the latest dev release of field_validation which already contains the needed patch, and the latest dev release of feeds patched with #171. Tried some other configurations also.
This thread is becoming very long and from some of the comments I can see that I am not the only one getting lost.
Comment #174
raul_drupal_dev CreditAttribution: raul_drupal_dev commentedHi community
I made a video explaining how fix this issue. its made in spanish but is easy to follow. If you've got any question feel free to ask me!. Also I made a pack with the modules already parched. just plug&play!
The link --> http://drupalia.cat/videotutorial/drupal-7-crear-un-campo-unico-customiz...
Please feedbak is welcome!!
Comment #175
MegaChriz CreditAttribution: MegaChriz commentedThe attached patch completes the tests I was working on in #170/#171. In comparison with the previous patch, there are no changes in the Feeds module itself, only in the tests. Technical details: the unique callback was moved from the test class "FeedsMapperUniqueTestCase" to the test module. The test class couldn't be loaded during runtime, because it inherits from the class "DrupalWebTestCase" which is only available when the simpletest module is enabled.
@raul_drupal_dev
Great video. :) I couldn't understand much of the words of it, but I think the steps provided of how to install the patch are clear enough. Thanks for proving that the patch works.
Comment #176
chrowe CreditAttribution: chrowe commentedThis seems to not work with link fields. Any ideas why?
Comment #177
MegaChriz CreditAttribution: MegaChriz commented@chrowe
The feeds target name for the link field is different compared to other fields, because the link field consists of two values instead of one ('url' and 'title'), see feeds/mappers/link.inc. Assuming you are using the Field validation module: the Field validation module does not take this into account, it assumes that each feeds target for a field is equal to the field's machine name and this is not the case for the link field.
If you want to make the link field a unique target, you should do the following:
TRUE
and define a callback function for 'unique_callbacks'.NULL
otherwise.The details plus an example are included with the patch in #175. Check feeds.api.php after applying the patch.
Comment #178
hairqles CreditAttribution: hairqles commentedHi,
I've tested the patch and it works as I was expected, so great job! Thank you!
On the other hand the current implementation prevents me to define my own logic to determine if the GUID or the URL field is unique or not. My usecase is that the GUID field should be unique globally.
So I've made a few changes in the code. You can check that in the attached patch file.
Comment #180
MegaChriz CreditAttribution: MegaChriz commented@hairqles
Uhm, making the GUID globally unique seem to be a total different issue. In fact, there is already an issue about this: #1539224: Add support for unique fields to be unique site wide. Lets keep this issue focussed.
Comment #181
Exploratus CreditAttribution: Exploratus commented#175 Worked for me. Field validation was not working until I installed the patch. I am using field validation, Feeds 7.x-2.0-alpha8, and Patch #175 to ensure imports are mapped to existing nodes based on a unique field.
Comment #182
twistor CreditAttribution: twistor commentedComment #183
twistor CreditAttribution: twistor commentedNormalize hook parameters.
Comment #184
twistor CreditAttribution: twistor commentedComment #186
twistor CreditAttribution: twistor commentedI think this is ready.
Comment #187
MegaChriz CreditAttribution: MegaChriz commentedThanks for the update! I'll plan to take a closer look at the patch in #186 next Thursday. For now, I saw one thing in the code which I have a question about:
I'm not sure, but isn't the extra indenting here against coding standards? I don't think it is anywhere marked up in core like that.
Example from field.api.php:
Comment #188
lquessenberry CreditAttribution: lquessenberry commentedHey guys, I can't get the settings to stick when I select them. I applied patch 186 to the newest dev version of feeds.
Comment #189
twistor CreditAttribution: twistor commented@lquessenberry, what are you trying to make unique? You have to click save after changing the settings.
Comment #190
twistor CreditAttribution: twistor commentedComment #191
lquessenberry CreditAttribution: lquessenberry commentedI was away this weekend. Let me see if I can provide a screenshot this time. I can't get it to save for some reason.
Comment #192
lquessenberry CreditAttribution: lquessenberry commentedhttp://puu.sh/akOS9/096f047e85.png Here is a screenshot. I am using an entity and not a node. Might that be a problem? I check the radio button and hit save, and then it returns that it was saved and nothing is selected.
Comment #193
lquessenberry CreditAttribution: lquessenberry commentedWait a minute. I don't think I set a field to be unique with the unique fields module. I was mapping ot a GUID. I will turn that off and turn a field to be unique in my entity. I will see if that works.
Comment #194
MegaChriz CreditAttribution: MegaChriz commentedI'm halfway of reviewing the patch in #186 (found a few minor code issues so far). For now, I just want to warn users that this patch does not work with Field validation module. This is not because the patch in #186 is wrong, but because the implementation of the Field validation module is based on an earlier patch, so Field validation must be updated once a fix for this issue gets in.
Site builders (using Field validation module): temporary use the patch in #175.
Developers (not using Field validation module): use the patch in #186.
I plan to post a patch for Field validation module here when I'm done with reviewing #186.
Comment #195
MegaChriz CreditAttribution: MegaChriz commentedAfter I modified field_validation.feeds.inc based on the code example in feeds.api.php I can confirm the patch in #186 works!
I made a few minor modifications. This doesn't change much functionality, it's just a little cleanup.
I changed the indentation of those to match with Drupal core's code style (see also #187).
I turned this into a
drupal_static()
, as I think this could else cause problems in automated tests. Imagine you have two tests with the same importer ID and they both involve testing unique ID's. Withoutdrupal_static()
the second test would receive the mapping targets from the first test.Attached are three files:
Looks ready to me.
Comment #196
manojbisht_drupal CreditAttribution: manojbisht_drupal commentedHello,
I have created a custom field in one of my content type, and I wants to add it in feeds mapping for node processor as unique field.
As suggested by some people.
I have downloaded the unique_field module and after that applied the patch mentioned in #57.
But the patch didnot work.
as there is not a file field.inc in mappers folder.
so I edited the patch and again commiting it.
Please review it.
Meanwhile, I have also downloaded the fresh install of feeds both alpha and development, no one has field.inc file.
Comment #197
manojbisht_drupal CreditAttribution: manojbisht_drupal commentedSorry for above patch it will not work.
Uploading new patch.
Comment #199
MegaChriz CreditAttribution: MegaChriz commented@manojbisht_drupal
Sorry, your piece of code should go in Unique Field module, not in the Feeds module. See the issue summary:
To get this to work with Unique field module:
TRUE
and define a callback function for 'unique_callbacks'.NULL
otherwise.The details plus an example are included with the patch. Check feeds.api.php after applying the patch.
You can also take Field validation's implementation as an example. In this case, be sure to apply the in #195 provided patch for Field validation module first.
Comment #200
MegaChriz CreditAttribution: MegaChriz commentedComment #202
twistor CreditAttribution: twistor commentedCongratulations everybody! Thanks for all of your hard work.
Comment #203
MegaChriz CreditAttribution: MegaChriz commentedGreat! I've posted the patch that is needed for Field validation in #1705386-18: Feeds integration (Field validation). This is the same patch that I provided here in #195.
Comment #204
MegaChriz CreditAttribution: MegaChriz commentedI've also updated the docs for how and when to use this feature.
Comment #206
knowledges33ker CreditAttribution: knowledges33ker commentedMegaChiz, I'm trying to follow the documentation you linked to in #204.
I've got multiple feeds importing nodes into multiple content types. The problem is that some stories exist in two (or more) feeds and therefore end up duplicated, triplicate (or worse) once all the feeds have run. Since I'm using a different importer for each feed, the existing stories aren't being recognized as needing to be updated or skipped and they are simply being created as if they are new nodes.
After installing the field validation module and applying the patch from #1705386-18: Feeds integration (Field validation) to the Field validation module and setting a unique value in the content type(s), I get to the last step "Go to the mapping settings of your Feeds importer, add the field you want to be unique and set this field as an unique target."
But on the mapping settings tab of the Feeed importer, after I add the field I want to be unique (a link field imported from the feed), I don't have the option to set that field as unique from with the feeds interface.
Where have I gone wrong?
Comment #207
GerZah CreditAttribution: GerZah commentedIs it possible that the "Use other field as an unique target in Feeds 7.x-2.x" feature broke in Feeds 7.x-2.0-alpha8?I'm using the simplest possible other unique field, namely an integer ID. After updating Feeds to 7.x-2.0-alpha8, it won't respect another import using the same numerical ID as existing node that should be updated, but actually create a new node – with the same numerical ID.The latter is extremely disturbing: Feeds actually creates a second node using the same numerical ID that is supposed to be unique. Albeit, field_validation works fine – I am not able to edit and save that node: Validation kicks in and won't let me re-use the identical ID.Forget what I said. — Sorry, something went wrong and I ended up with 7.x-2.0-alpha8 instead of 7.x-2.x-dev.
Comment #208
GerZah CreditAttribution: GerZah commentedComment #209
GerZah CreditAttribution: GerZah commentedComment #210
joachim CreditAttribution: joachim commentedI'm trying to get support for this into Data module (#2174373: Add support for Feeds "unique" field updating/replacing) and I don't understand the reason for this:
Why should the value always be turned into an array?
From Data module's perspective, this just means it has to be extracted out of the array in the unique_callback.
Comment #211
MegaChriz CreditAttribution: MegaChriz commented@joachim
This is done for two reasons:
my_module_set_target()
in feeds.api.php).Comment #212
codev0 CreditAttribution: codev0 commentedHi.
After patching, on localhost everything is good, but after placing in to shared hosting I have error.
I applied this patch https://www.drupal.org/files/issues/feeds-unique-target-661606-195.patch
Is this patch available from module dev version?
UPD: I forgot upload updated field_validation. Sorry, my bad.
Comment #213
phendji-cap CreditAttribution: phendji-cap commented@ Dakanca
Use the version of Dev. directly because the patch already applied to this version
Comment #214
codev0 CreditAttribution: codev0 commented@phendji-cap
I tried use dev version, it breaks my site.
Comment #215
joelpittet@dakanca can you open up a new issue for the "breaks my site" issue and link it here? Maybe put more detail into what error message you are getting from PHP?
Comment #216
brandonc503 CreditAttribution: brandonc503 commented*deleted as i cant remove account
Comment #217
yatendrasingh121 CreditAttribution: yatendrasingh121 commentedHi,
Patch attached in #195 does not work for me. Becuase I am using 7.x-2.0-alpha7 version of feeds . I am attaching a patch for 7.x-2.0-alpha7 version.
Comment #218
kingandy CreditAttribution: kingandy commentedAs of today the code does not appear to have made its way into the main branch (7.x-2.x-alpha9) - if you want the unique target mappers you'll need to download the latest -dev release.
Comment #219
dasginganinjaI can confirm what kingandy has said.
Comment #220
marcolz CreditAttribution: marcolz as a volunteer commentedPatch from #217 updated for 2.0-alpha9 attached.
Comment #221
yatendrasingh121 CreditAttribution: yatendrasingh121 commentedPatch attached in #220 does not work for 2.0-alpha9 . Attaching correct patch.
Comment #222
memcinto CreditAttribution: memcinto commented7.x-2.0-beta1 is now out as of 11-July. Hoping for a new patch ;-) THANKS!
Comment #223
dasginganinja@twistor, is there any way this can get rolled up into the next 7.x beta release?
Comment #224
twistor CreditAttribution: twistor as a volunteer commentedThis is already in beta1.
Comment #225
MegaChriz CreditAttribution: MegaChriz at WebCoo commentedThe patch from #195 is already part of the 7.x-2.0-beta1 release. See commit. See also the code from this release.
See the docs for how to use this feature.
Comment #226
dasginganinja+1 Thanks guys.
Comment #227
memcinto CreditAttribution: memcinto commentedre #225 EXCELLENT!! Thank you so much!!
Comment #228
adrien.felipe CreditAttribution: adrien.felipe commentedUsing the same feature from the 7.x-2.0-beta1 release, with a Commerce Order and Commerce order ID as unique, it does not work.
I get a duplicated entry SQL error.
Should it work? Should we extend the patch or functionnality?
Comment #229
MegaChriz CreditAttribution: MegaChriz at WebCoo commented@wayaslij
Sounds like an issue in the Commerce Feeds module, if you are using that. Or, if you use the Feeds entity processor module, see #2004762: Entity ID as unique target does not work.
Anyway, this issue is about being able to use any target as unique identifier using a custom module by specifying a callback on the target definition. The Field validation module implements such a callback. This issue is not about unique targets in general. If a specific unique target does not work, please open a new issue. If the unique target is provided by a module other than Feeds, open an issue in the queue for that module (instead of Feeds' queue).
Comment #230
SeanA CreditAttribution: SeanA commented