CAVEAT

the REST async queue seems to work just fine.
All my input here is in regards to SOAP async queue processing.

Synopsis

Asynchronous queue processing is a difficult problem, and the current approach falls well short.
SOAP queue processing needs to be scrapped and rewritten.

Current Status

Right now, everything gets dumped into a single queue, only to be tediously teased apart again during cron runtime.
Basic scenarios lead to critical failures. Queues of mixed updates, creates, deletes, or upserts fail. Upserts can lead to data loss if two different objects have the same key field name.

Proposed Solution

  • Each mapping should have its own queues, based on its configured settings.
    If deletes are to be synched, the mapping should have a corresponding delete queue.
    If creates and updates are to be synched, there's at least one more queue.
    Finally, if the mapping has a dedupe key configured, an additional upsert queue may be useful.
  • SF Push should do more work up front to figure out where queue items belong, rather than dumping them into a pile and delegating all the work to hook_cron.
  • There needs to be some error handling. Async push can fail for a variety of reasons - from network problems to data conflicts to validation failure. There needs to be some kind of configurable error handling policy, along with better reporting.
  • With multiple queues, a global limit on the number of items to process may be problematic. Mappings and operations need some kind of priority to determine which queues are processed first.
  • If using SOAP, always use upsert(), never create() or update().

Comments

aaronbauman created an issue. See original summary.

aaronbauman’s picture

Assigned: Unassigned » aaronbauman
aaronbauman’s picture

Issue summary: View changes
aaronbauman’s picture

aaronbauman’s picture

Assigned: aaronbauman » Unassigned
Status: Active » Needs review
aaronbauman’s picture

Status: Needs review » Needs work

This is still not right.

amaisano’s picture

Is this the reason why as soon as I check "Process Asynchronously" for my User mapping, it never pushes - even after multiple updates on Drupal? Not even when I manually choose "Push to Salesforce" from the Bulk Operations after I've updated it?

Push works flawlessly when that option is not checked. $update_list is being populated with the right object that needs pushing, but nothing ever happens with it :(

aaronbauman’s picture

Yup, probably

amaisano’s picture

Crud. I assume this affects the 7.x-3.1 too then (which I'm using). Good luck figuring it out - I'll follow closely.

aaronbauman’s picture

Title: salesforce_push_cron() is broken » salesforce_push_cron() is broken when using SOAP
Issue summary: View changes

Issue summary updated.

aaronbauman’s picture

Status: Needs work » Closed (won't fix)

7.x is no longer supported

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.