The following error message has been greatly truncated as there are 607529 placeholders in that exception. Basically there is a batch update that's running via drush that is removing a bunch of content. That content is causing some removal tasks for some indexes, and this is the result. My proposed solution is to queue the items for deletion across several shutdowns. Not just the one. This may cause some other unexpected issues. but I don't have any other idea.

Performed update: martincounty_helper_update_7004                                                                           [ok]
The external command could not be executed due to an application error.                                                     [error]
Illegal string offset 'site' backend.inc:1014                                                                               [warning]
The command could not be executed successfully (returned:                                                                   [error]
DRUSH_BACKEND_OUTPUT_START>>>{"output":"","object":[],"error_status":0,"log":[{"type":"bootstrap","message":"Bootstrap to
phase 0.","timestamp":1431115806.3222,"memory":2509376,"error":null},{"type":"bootstrap","message":"Drush bootstrap phase :
_drush_bootstrap_drush()","timestamp":1431115806.3247,"memory":2656008,"error":null},{"type":"debug","message":"Cache MISS
cid:
6.6.0-alias-path--b71e067f3afde326615ffbe4782be6a4","timestamp":1431115806.3265,"memory":2665816,"error":null},{"type":"debug","message":"Cache
HIT cid:
6.6.0-commandfiles-0-2d954a4161f4eea371bb8168880fcaad","timestamp":1431115806.3312,"memory":2677736,"error":null},{"type":"bootstrap","message":"Bootstrap
to phase 0.","timestamp":1431115806.4094,"memory":6828288,"error":null},{"type":"bootstrap","message":"Bootstrap to phase
2.","timestamp":1431115806.4649,"memory":6832352,"error":null},{"type":"bootstrap","message":"Drush bootstrap phase :
_drush_bootstrap_drupal_root()","timestamp":1431115806.4735,"memory":7938904,"error":null},{"type":"notice","message":"Initialized
Drupal 7.36 root directory at
\/var\/www\/sites\/martincounty.dev\/www","timestamp":1431115806.4766,"memory":7940544,"error":null},{"type":"bootstrap","message":"Drush
bootstrap phase :
_drush_bootstrap_drupal_site()","timestamp":1431115806.4778,"memory":7946896,"error":null},{"type":"notice","message":"Initialized
Drupal site default at
sites\/default","timestamp":1431115806.478,"memory":7948664,"error":null},{"type":"debug","message":"Cache HIT cid:
6.6.0-install_profile-66ecfeb9791a023150773849f1550c5d","timestamp":1431115806.4815,"memory":7951568,"error":null},{"type":"debug","message":"Cache
HIT cid:
6.6.0-commandfiles-2-b6c91fb3e8acba720fae6a156d2c9808","timestamp":1431115806.4819,"memory":7963240,"error":null},{"type":"bootstrap","message":"Found
command: updatedb-batch-process
(commandfile=core)","timestamp":1431115806.5108,"memory":10012672,"error":null},{"type":"debug","message":"Calling hook
drush_core_updatedb_batch_process","timestamp":1431115806.7185,"memory":10985152,"error":null},{"type":"notice","message":"WD
php: Notice: Undefined index: name in system_requirements() (line 448 of
\/var\/www\/sites\/martincounty.dev\/www\/modules\/system\/system.install).","timestamp":1431115807.8211,"memory":23778520,"error":null},{"type":"notice","message":"WD
php: Notice: Undefined index: name in system_requirements() (line 469 of
\/var\/www\/sites\/martincounty.dev\/www\/modules\/system\/system.install).","timestamp":1431115807.8277,"memory":23781160,"error":null},{"type":"notice","message":"WD
php: Notice: Undefined index: name in system_requirements() (line 469 of
\/var\/www\/sites\/martincounty.dev\/www\/modules\/system\/system.install).","timestamp":1431115807.8294,"memory":23782840,"error":null},{"type":"notice","message":"WD
php: Notice: Undefined index: name in system_requirements() (line 469 of
\/var\/www\/sites\/martincounty.dev\/www\/modules\/system\/system.install).","timestamp":1431115807.8305,"memory":23784648,"error":null},{"type":"notice","message":"WD
php: Notice: Undefined index: name in system_requirements() (line 448 of
\/var\/www\/sites\/martincounty.dev\/www\/modules\/system\/system.install).","timestamp":1431115807.8331,"memory":23786344,"error":null},{"type":"notice","message":"WD
php: Notice: Undefined index: name in system_requirements() (line 448 of
\/var\/www\/sites\/martincounty.dev\/www\/modules\/system\/system.install).","timestamp":1431115807.8344,"memory":23788016,"error":null},{"type":"bootstrap","message":"Drush
bootstrap phase :
_drush_bootstrap_drupal_configuration()","timestamp":1431115807.8398,"memory":23693016,"error":null},{"type":"bootstrap","message":"Drush
bootstrap phase :
_drush_bootstrap_drupal_database()","timestamp":1431115807.8413,"memory":23697192,"error":null},{"type":"bootstrap","message":"Successfully
connected to the Drupal
database.","timestamp":1431115807.8417,"memory":23698224,"error":null},{"type":"bootstrap","message":"Drush bootstrap phase
:
_drush_bootstrap_drupal_full()","timestamp":1431115807.8422,"memory":23699264,"error":null},{"type":"debug","message":"Cache
HIT cid:
6.6.0-install_profile-66ecfeb9791a023150773849f1550c5d","timestamp":1431115808.1815,"memory":51733320,"error":null},{"type":"notice","message":"Undefined
index: name in system_requirements() (line 448 of
\/var\/www\/sites\/martincounty.dev\/www\/modules\/system\/system.install).","timestamp":1431115808.1818,"memory":51735760,"error":null},{"type":"notice","message":"Undefined
index: name in system_requirements() (line 469 of
\/var\/www\/sites\/martincounty.dev\/www\/modules\/system\/system.install).","timestamp":1431115808.1822,"memory":51737032,"error":null},{"type":"notice","message":"Undefined
index: name in system_requirements() (line 469 of
\/var\/www\/sites\/martincounty.dev\/www\/modules\/system\/system.install).","timestamp":1431115808.1825,"memory":51738304,"error":null},{"type":"notice","message":"Undefined
index: name in system_requirements() (line 469 of
\/var\/www\/sites\/martincounty.dev\/www\/modules\/system\/system.install).","timestamp":1431115808.1827,"memory":51739576,"error":null},{"type":"notice","message":"Undefined
index: name in system_requirements() (line 448 of
\/var\/www\/sites\/martincounty.dev\/www\/modules\/system\/system.install).","timestamp":1431115808.1836,"memory":51740848,"error":null},{"type":"notice","message":"Undefined
index: name in system_requirements() (line 448 of
\/var\/www\/sites\/martincounty.dev\/www\/modules\/system\/system.install).","timestamp":1431115808.1838,"memory":51742120,"error":null},{"type":"debug","message":"Cache
HIT cid:
6.6.0-install_profile-66ecfeb9791a023150773849f1550c5d","timestamp":1431115808.1841,"memory":51737976,"error":null},{"type":"debug","message":"Cache
HIT cid:
6.6.0-commandfiles-5-31db529743dc1883a0185c0a5a446b38","timestamp":1431115808.185,"memory":51794920,"error":null},{"type":"notice","message":"Executing
martincounty_helper_update_7004","timestamp":1431115809.2528,"memory":56835160,"error":null},{"type":"notice","message":"WD
features: Reverting martin_county_resource_ct \/
ds_field_settings.","timestamp":1431115809.4425,"memory":64075016,"error":null},{"type":"notice","message":"WD features:
Revert completed for martin_county_resource_ct \/
ds_field_settings.","timestamp":1431115809.5909,"memory":65143960,"error":null},{"type":"notice","message":"WD features:
Reverting martin_county_resource_ct \/
ds_layout_settings.","timestamp":1431115809.5933,"memory":65145096,"error":null},{"type":"notice","message":"WD features:
Revert completed for martin_county_resource_ct \/
ds_layout_settings.","timestamp":1431115809.6051,"memory":65446520,"error":null},{"type":"notice","message":"WD features:
Reverting martin_county_resource_ct \/
field_base.","timestamp":1431115809.6069,"memory":65447552,"error":null},{"type":"notice","message":"WD features: Revert
completed for martin_county_resource_ct \/
field_base.","timestamp":1431115809.9103,"memory":66866872,"error":null},{"type":"notice","message":"WD features: Reverting
martin_county_resource_ct \/
field_instance.","timestamp":1431115809.9125,"memory":66867920,"error":null},{"type":"notice","message":"WD features: Revert
completed for martin_county_resource_ct \/
field_instance.","timestamp":1431115810.5394,"memory":68111184,"error":null},{"type":"notice","message":"WD features:
Reverting martin_county_resource_ct \/
node.","timestamp":1431115810.5423,"memory":68112208,"error":null},{"type":"notice","message":"WD features: Revert completed
for martin_county_resource_ct \/
node.","timestamp":1431115814.1988,"memory":112076000,"error":null},{"type":"notice","message":"WD features: Reverting
martin_county_resource_ct \/
variable.","timestamp":1431115814.2015,"memory":112077040,"error":null},{"type":"notice","message":"WD features: Revert
completed for martin_county_resource_ct \/
variable.","timestamp":1431115814.4269,"memory":116631488,"error":null},{"type":"ok","message":"Performed update:
martincounty_helper_update_7004","timestamp":1431115823.7405,"memory":120030784,"error":null},{"type":"batch","message":"Batch
process has consumed in excess of 50% of available memory. Starting new
thread","timestamp":1431115823.7414,"memory":120031880,"error":null},{"type":"debug","message":"Returned from hook
drush_core_updatedb_batch_process","timestamp":1431115823.7417,"memory":120025808,"error":null},{"type":"notice","message":"Command
dispatch complete","timestamp":1431115823.7429,"memory":119960008,"error":null},{"type":"memory","message":"Peak memory
usage was 123.64
MB","timestamp":1431115823.7431,"memory":119954952,"error":null}],"error_log":[],"self":{"root":"\/var\/www\/sites\/martincounty.dev\/www","uri":"http:\/\/default","path-aliases":{"%root":"\/var\/www\/sites\/martincounty.dev\/www"}},"context":{"php":"\/usr\/bin\/php","php-options":"
-d magic_quotes_gpc=Off -d magic_quotes_runtime=Off -d
magic_quotes_sybase=Off","backend":"2","yes":true,"root":"\/var\/www\/sites\/martincounty.dev\/www","uri":"http:\/\/default"}}<<<DRUSH_BACKEND_OUTPUT_END<h1>Uncaught
exception thrown in shutdown function.</h1><p>PDOException: SQLSTATE[HY000]: General error: 2006 MySQL server has gone away:
DELETE FROM {search_api_task}
WHERE  (id IN  (:db_condition_placeholder_0, :db_condition_placeholder_1, :db_condition_placeholder_2,
:db_condition_placeholder_3, :db_condition_placeholder_4, :db_condition_placeholder_5, :db_condition_placeholder_6,
:db_condition_placeholder_7, :db_condition_placeholder_8, :db_condition_placeholder_9, :db_condition_placeholder_10,
:db_condition_placeholder_11, :db_condition_placeholder_12, :db_condition_placeholder_13, :db_condition_placeholder_14,
:db_condition_placeholder_15, :db_condition_placeholder_16, :db_condition_placeholder_17, :db_condition_placeholder_18


...

 [:db_condition_placeholder_607525] =&amp;gt; 3861411
    [:db_condition_placeholder_607526] =&amp;gt; 3861416
    [:db_condition_placeholder_607527] =&amp;gt; 3861421
    [:db_condition_placeholder_607528] =&amp;gt; 3861426
    [:db_condition_placeholder_607529] =&amp;gt; 3861431
)
 in search_api_server_tasks_delete() (line 1383 of
/var/www/sites/martincounty.dev/vendor/drupal/search_api/search_api.module).</p><hr />PHP Fatal error:  Uncaught exception
'PDOException' with message 'SQLSTATE[HY000]: General error: 2006 MySQL server has gone away' in
/var/www/sites/martincounty.dev/www/includes/database/database.inc:2171

Comments

drunken monkey’s picture

drunken monkey
he/him
Primary language German
Location Vienna, Austria
commented
Status: Active » Closed (won't fix)

First off, using code that will call API functions like search_api_server_tasks_delete() is discouraged, or even forbidden, in update hooks.
And in general, I think it is rare enough that a query tries to delete 600k tasks at once to have to optimize for this case. I'm not sure, what the exact sequence of events is that leads to all those deleted tasks, but maybe just try to avoid this in some way (e.g., by manually deleting tasks, or by disabling the server while executing this update) for your scenario.
Stretching this over multiple page requests would complicate the code a lot (thus potentially also introducing new problems), but probably only benefit a handful of users world-wide.

generalredneck’s picture

generalredneck
him/his
Primary language English
Location Texas, USA 🇺🇸
commented

First off, some clarification. I'm not using api functions directly in the hook. This module is performing them on shutdown. So yes you are right... greatly discouraged and NOT used in my update hooks.

I am perfectly ok with your reasons for not making the fix as you are most likely right that not very many people delete 600000 nodes generally...

But I did want to make sure people are aware it's a thing that does happen with big content changes.

kenorb’s picture

kenorb commented
Status: Closed (won't fix) » Closed (duplicate)
Related issues: +#2408727: Better handling of large numbers of tasks

#2408727: Better handling of large numbers of tasks