CAVEAT
the REST async queue seems to work just fine.
All my input here is in regards to SOAP async queue processing.
Synopsis
Asynchronous queue processing is a difficult problem, and the current approach falls well short.
SOAP queue processing needs to be scrapped and rewritten.
Current Status
Right now, everything gets dumped into a single queue, only to be tediously teased apart again during cron runtime.
Basic scenarios lead to critical failures. Queues of mixed updates, creates, deletes, or upserts fail. Upserts can lead to data loss if two different objects have the same key field name.
Proposed Solution
- Each mapping should have its own queues, based on its configured settings.
If deletes are to be synched, the mapping should have a corresponding delete queue.
If creates and updates are to be synched, there's at least one more queue.
Finally, if the mapping has a dedupe key configured, an additional upsert queue may be useful.
- SF Push should do more work up front to figure out where queue items belong, rather than dumping them into a pile and delegating all the work to hook_cron.
- There needs to be some error handling. Async push can fail for a variety of reasons - from network problems to data conflicts to validation failure. There needs to be some kind of configurable error handling policy, along with better reporting.
- With multiple queues, a global limit on the number of items to process may be problematic. Mappings and operations need some kind of priority to determine which queues are processed first.
- If using SOAP, always use upsert(), never create() or update().
Comments
Comment #2
aaronbaumanComment #3
aaronbaumanComment #4
aaronbaumanComment #5
aaronbaumanComment #6
aaronbaumanThis is still not right.
Comment #7
amaisano commentedIs this the reason why as soon as I check "Process Asynchronously" for my User mapping, it never pushes - even after multiple updates on Drupal? Not even when I manually choose "Push to Salesforce" from the Bulk Operations after I've updated it?
Push works flawlessly when that option is not checked. $update_list is being populated with the right object that needs pushing, but nothing ever happens with it :(
Comment #8
aaronbaumanYup, probably
Comment #9
amaisano commentedCrud. I assume this affects the 7.x-3.1 too then (which I'm using). Good luck figuring it out - I'll follow closely.
Comment #10
aaronbaumanIssue summary updated.
Comment #11
aaronbauman7.x is no longer supported