Support for Drupal 7 is ending on 5 January 2025—it’s time to migrate to Drupal 10! Learn about the many benefits of Drupal 10 and find migration tools in our resource center.I have a very large site I inherited with over 400K field collection items (and over half a million nodes). I started the update at 2:15 today, and it's still running nearly four hours later. I can tell it's working as I'm now down to ~370K items, but I am very concerned about how long this is taking and there's no way to know how far along in the process it is. Are there any ways this can possibly be optimized?
| Comment | File | Size | Author |
|---|---|---|---|
| #9 | field_collection_update7002-bigdata.patch | 3.3 KB | itsekhmistro |
| |||
| #8 | 2344513-field_collection_update7002-bigdata.patch | 1.71 KB | Josh Waihi |
| #5 | field_collection_update_7002-2344513-3.patch | 1.21 KB | seanr |
Comments
Comment #1
mikemccaffreyI am seeing this issue as well. When running update 7002, my system is either hanging or crashing with a memory allocation error.
It seems that the cause of this is in the left joins that the update is doing between field_collection_item and the various field_data_field_something tables. The individual field_data tables don't have indexes on the field_something_value fields, so joining on that is very computationally intensive.
You may want to add the indexes yourself before running this update as a short term fix. It might also be good to have the update add indexes before proceeding with the join.
Comment #2
seanrThe attached patch definitely speeds up this update.
Comment #4
seanrFixed patch base path. BTW, all credit for this belongs to https://www.drupal.org/u/astierler
Comment #5
seanrComment #6
seanrWe've tested this at least a couple dozen times in a CI setup and have had no problems. It definitely speeds up the update substantially. Marking RTBC.
Comment #7
jmuzz CreditAttribution: jmuzz commentedComments should end in a full stop.
Can we resolve the doubt around deleting the index afterwards? On the one hand many have already run this update so in theory it shouldn't be changed in a way that will leave newly updated versions in a different state than the ones that updated before. On the other hand there is reason to have this index as indicated in #2213795: Add index to FIELDNAME_value column on field_collection field tables .
Comment #8
Josh Waihi CreditAttribution: Josh Waihi as a volunteer and at Acquia commentedWith large amounts of data, you can also run into issues with php memory attempting to retrieve all the data at once. I've revised the patch to perform deletes in batches of 1000. I've also changed the drupal_set_message call to watchdog which is more visible in drush log output.
Comment #9
itsekhmistro CreditAttribution: itsekhmistro at Adyax commentedHi, the patch in comment #8 doesn't follow the Drupal practice: big DB updates should be processed in batches.
Attached updated patches that performs field_collection_update_7002 in batches.
Note, highly recommended to run this update with drush for performance reasons.