Feeds saves a hash of source row data and skips processing the hash is the same. However, I'd like to be able to force an update for a particular batch job. Even if the source row hasn't changed, I may have changed the processing behavior with hook_feeds_after_parse() or Feeds Tamper module settings.

Files: 
CommentFileSizeAuthor
#23 feeds-skip-hash-check-1364116-23.patch1.41 KBosopolar
PASSED: [[SimpleTest]]: [MySQL] 2,562 pass(es).
[ View ]
#8 0002-Reverted-force-update-config-in-1152940-as-skip-hash.patch1.76 KBaxel.rutz
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch 0002-Reverted-force-update-config-in-1152940-as-skip-hash.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]
#7 0001-Implemented-1364116-Option-to-skip-hash-check-on-re-.patch1.91 KBaxel.rutz
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch 0001-Implemented-1364116-Option-to-skip-hash-check-on-re-.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]
#2 0001-Implement-1364116-Option-to-skip-hash-check-on-re-im.patch1.65 KBaxel.rutz
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch 0001-Implement-1364116-Option-to-skip-hash-check-on-re-im.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]

Comments

twistor’s picture

If you have changed the way the items get parsed then the hash of the items will change.

As a note, if you want to re-run an import you can delete the last two mappings and re-add them in the opposite order. Not ideal, I know, but can be useful.

Still a valid feature request.

axel.rutz’s picture

Status:Active» Needs review
StatusFileSize
new1.65 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch 0001-Implement-1364116-Option-to-skip-hash-check-on-re-im.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]

>If you have changed the way the items get parsed then the hash of the items will change.
Source code says: hash comes from source data and unchanged hash saves us from forther processing - which is a good thing in production.

I think this is an important DX issue and iterating a feed is a PITA without that so raising prio (feel free to readjust if i'm too self centered).

Here is a straightforward implementation which worksforme, please test.

As soon as this gets committed i will simplify my pending patch in #1152940: Feeds term import with hierarchy and weight where we needed the same for a two-pass import.

twistor’s picture

Status:Needs review» Needs work

#1152940: Feeds term import with hierarchy and weight got committed. Can you re-work this patch to remove the Term specific functionality?

axel.rutz’s picture

Status:Needs work» Needs review

> Can you re-work this patch to remove the Term specific functionality?
I don't see term specific functionality in #2...?

twistor’s picture

Status:Needs review» Needs work

FeedsTermProcessor has its own version of this patch added in #1152940: Feeds term import with hierarchy and weight. That code should be re-factored to use this patch.

axel.rutz’s picture

Status:Needs work» Needs review

Patch in minisprint branch:
* Implemented 1364116: Option to skip hash check on re-import - i reworked that to stay compatible for most scenarios by letting getHash return a fake hash so other importers can inherit that.
* Reverted force-update config in #1152940 as skip-hash-check #1364116 implements that
* also this fixes #1698076: Force update for TermProcessor should default to FALSE.

axel.rutz’s picture

StatusFileSize
new1.91 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch 0001-Implemented-1364116-Option-to-skip-hash-check-on-re-.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]
axel.rutz’s picture

StatusFileSize
new1.76 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch 0002-Reverted-force-update-config-in-1152940-as-skip-hash.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]
twistor’s picture

This should actually be 2 settings. Forced updating should be set on a per-source basis.

  1. Allow forced updating, on the importer page.
  2. On the source page, "Force update" checkbox.
twistor’s picture

Status:Needs review» Needs work
axel.rutz’s picture

Status:Needs work» Needs review

I don't agree moving the "force update" setting from "importer" to "source".
At least i don't understand your background.
For me this setting is an extension to the "update existing content" setting and so global to the processor.
Having this per source is a different issue to me which i now see no use case.

twistor’s picture

Status:Needs review» Needs work

@axel, I concede the point. My thought is that it's most useful for debuggin/developing, rather than a long term solution. I agree though, that it's more of an extension to "Update existing". So, then why don't we put it in that list :)

axel.rutz’s picture

Status:Needs work» Needs review

> it's more of an extension to "Update existing". So, then why don't we put it in that list :)

I already thought about that but "skip hash check" is orthogonal to the "update / replace" setting.

Incorporating in the list would multiply that out to 5 options like this reworded)

Unique target behavior:
* Create new items on every import
* Replace existing items on every import
* Update existing items (slower than replacing them, but keeps unmapped fields) on every import
* Replace existing items if source item changed
* Update existing items (slower than replacing them, but keeps unmapped fields) if source item changed

old wording:

* Do not update existing nodes
* Do not update existing nodes & [X] Skip hash check
* Update existing nodes (slower than replacing them) & [X] Skip hash check
* Do not update existing nodes & [ ] Skip hash check
* Update existing nodes (slower than replacing them) & [ ] Skip hash check

what do you think about this?

twistor’s picture

Status:Needs review» Patch (to be ported)
Issue tags:+Needs tests
slefevre1’s picture

I think it would be handy to have this option on the import page of each feed. We ran into a situation where we had a feed for a content type that was in a features module. We added some fields to the feature, but our feeds ran from cron before we updated the mapping, and feeds considered the nodes to have been updated even though it didn't care about the new fields when it ran.

It would have been convenient to run the feed on its /import page, checked the box to ignore hashes, and then just go on our merry way. We don't want the hashes skipped on each import, just when we screw things up :P

osopolar’s picture

Version:7.x-2.x-dev» 6.x-1.0-beta13
Issue summary:View changes
Status:Patch (to be ported)» Needs review
StatusFileSize
new1.49 KB
FAILED: [[SimpleTest]]: [MySQL] 2,562 pass(es), 0 fail(s), and 4 exception(s).
[ View ]

Here is the 6.x backport. It's only for FeedsNodeProcessor because the other processors aren't using hashes at all.

Status:Needs review» Needs work

The last submitted patch, 16: feeds-skip-hash-check-1364116-16.patch, failed testing.

osopolar’s picture

Version:6.x-1.0-beta13» 6.x-1.x-dev
Status:Needs work» Needs review

Can't see how the patch causes the test failure - I guess there is something else causing the test to fail. Maybe Changing the version to current dev helps.

The last submitted patch, 2: 0001-Implement-1364116-Option-to-skip-hash-check-on-re-im.patch, failed testing.

The last submitted patch, 7: 0001-Implemented-1364116-Option-to-skip-hash-check-on-re-.patch, failed testing.

The last submitted patch, 8: 0002-Reverted-force-update-config-in-1152940-as-skip-hash.patch, failed testing.

axel.rutz’s picture

@osopolar: the patch is frozen to beta and probably won't apply (which is a good feature ;-)

you might want to re-upload it with dev version set !

osopolar’s picture

StatusFileSize
new1.41 KB
PASSED: [[SimpleTest]]: [MySQL] 2,562 pass(es).
[ View ]

Patch for dev attached. The patch in #17 applied well, but there it seems that there is something wrong with the locale (language) mapper mapper. Let's see if the problem goes away by itself.

moonray’s picture

Status:Needs review» Reviewed & tested by the community

Works. The line numbers just changed slightly, but the patch still applies.