I have a salesforce_mapped_object table with 400,000-odd records, which and attempts to update it to the latest stable release looked good on test sites, but once salesforce_mapping_update_8001() tried to fix that many entries, it was downhill.

What if, instead of loading, editing, and saving every mapped_object entity, we instead just used a SQL query to copy the values from the old columns to the new columns? Perhaps we would need to do the same for the revision table, but it would save a lot of entity business if this is truly just a field-column rename.

CommentFileSizeAuthor
#4 faster_sf_update_8001-2941754-4.patch2.59 KBgcb

Comments

gcb created an issue. See original summary.

aaronbauman’s picture

aaronbauman
he/him
commented

once salesforce_mapping_update_8001() tried to fix that many entries, it was downhill

do you mean it takes too long to complete? or it dies completely?
this update is using Drupal's batch api for the exact reasons you describe, but granted i have not tested on a 400K dataset.

Maybe include the equivalent sql commands in the comment, so that users can circumvent The Drupal Way&tm; when it fails.

gcb’s picture

gcb
he/him
Location Portland, OR
commented

Error was tough to parse:

Fatal error: Unknown: Cannot use output buffering in output buffering display handlers in Unknown on line 0
The external command could not be executed due to an application         [error]
error.
The command could not be executed successfully (returned: Ok             [error]
Ok

Fatal error: Unknown: Cannot use output buffering in output buffering
display handlers in Unknown on line 0
, code: 255)
Error while sending QUERY packet. PID=13319 Statement.php:59           [warning]
PDOException: SQLSTATE[HY000]: General error: 2006 MySQL server has      [error]

Perhaps the server simply turned the test site off, and it would have completed eventually. However, it was running for about 12 hours before it crashed: that's quite a long time to sit in maintenance mode.

I guess since this is an update hook designed to take the module out of a non-stable release to a stable, that might be a more reasonable ask.

gcb’s picture

gcb
he/him
Location Portland, OR
commented
StatusFileSize
new faster_sf_update_8001-2941754-4.patch2.59 KB

Just running it in sql still leaves you with an update that wants to run. How about I insert a comment that points to this patch, in case people want to use it?

gcb’s picture

gcb
he/him
Location Portland, OR
commented
Title: salesforce_mapping_update_8001 fails on large mapping object tables » salesforce_mapping_update_8001 too slow on large mapping object tables
aaronbauman’s picture

aaronbauman
he/him
commented

I'm good just adding a link to this issue.
I don't think we need to worry about tweaking this update hook too much, since it's probably just you and me who are using it.

gcb’s picture

gcb
he/him
Location Portland, OR
commented

No objections here.

aaronbauman’s picture

aaronbauman
he/him
commented
Status: Active » Closed (won't fix)

8.x is no longer supported

Now that this issue is closed, review the contribution record.

As a contributor, attribute any organization that helped you, or if you volunteered your own time.

Maintainers, credit people who helped resolve this issue.