Done so far:
- track pull times per mapping (the less efficient, more straightforward method suggested in original post)
Still todo:
- encapsulate the pull info within SalesforceMapping, so that the pull and delete handlers don't have to deal with raw config values
- update delete handling to actually take advantage of per-mapping pull info, rather than just doing the same thing over and over when we have multiple mappings on the same sobject type.
Original post:
Last pull times are stored similarly to how there were in in 7.x-3.x but the pull queries are done differently. In 7.x-3.x one query was done per object type with the collected mapped fields all added to the query. In 8.x-3.x it looks like a query is done for each mapping. With the current code, if multiple mappings exist for the same object type, data could get missed in the sync.
The module should either track last pull times per mapping OR merge the query for multiple mappings into a single query like is done in 7.x-3.x and then loop over all mappings for the object type when processing. While more complicated, this latter option is more efficient (less calls to SF) and probably easier for enabling multiple mappings for the same object.
| Comment | File | Size | Author |
|---|---|---|---|
| #5 | salesforce-pull_info_per_mapping-2896608.patch | 8.66 KB | aaronbauman |
Comments
Comment #2
aaronbaumanintent was to perform one query per object type, not per mapping, same as 7.x-3.xre-reading the code now and i'm not sure about original intent.
The easy parts of making this efficiency improvement are:
* moving "getPullQuery" out of SalesforceMapping class
* changing signature of SalesforceEvents::PULL_QUERY to accept SF object type instead of SalesforceMapping
The hard part is reconciling different pull trigger date fields between mappings.
(This component is ignored in 7.x-3.x)
This might have to be a rather tedious process of de-tangling a multi-mapping result set and enqueuing by mapping.
Or
Moving the pull queue trigger off the mapping and into a separate per-object schema.
This is how 7.x-2.x and Springboard used to work, i think.
I'm not sure that this is worth it to save API calls, but I'm open to ideas.
Comment #3
tauno commentedIt looks like the intent is split between per object queries a-la 7.x-3.x (this is how the last pull and delete dates are tracked in salesforce.sobject_pull_info) and per-mapping (how the queries actually happen).
Keeping the queries per-mapping might allow for greater flexibility around handling complex dependencies (Contact type B references Contact type A for example) and the increase in API calls should be fairly minimal.
The immediate issue with the current approach is that if two mappings pull from the same Objects, mapping 2 will miss data since the last pull date will get updated when mapping 1 is processed rather than when mapping 2 was last run.
This isn't impacting my current use case, but seems critical to get fixed so folks aren't unexpectedly missing data.
Comment #4
aaronbaumanRight, that makes a lot of sense.
I guess the cleanest solution is to have one pull timestamp per mapping.
Comment #5
aaronbaumanIn this patch:
- renames update hooks (again) with 80XX prefix
- refactors pull info state vars (again) to keep track of pull info per-mapping
- updates salesforce mapping class, pull handler, and delete handler to use the new settings
This passes unit tests and works on my very brief local tests.
This could use more cleanup.
I'd like to:
- encapsulate the pull info within SalesforceMapping, so that the pull and delete handlers don't have to deal with raw config values
- update delete handling to actually take advantage of per-mapping pull info, rather than just doing the same thing over and over when we have multiple mappings on the same sobject type.
Comment #7
aaronbaumancommitted previous patch.
setting back to active and updated OP with remaining cleanup tasks.
Comment #8
aaronbauman8.x is no longer supported