Better handling of large numbers of tasks

Remove all indexes from the server, then re-add them. That should do it.
Otherwise, just deleting them from the database should probably be fine, too.

PS: It seems you (like many others – it's really easy to misinterpret) are confused by the "Issue tags" field. As the guidelines state, they aren't meant for free text tags related to the issue, but only for specific categorization purposes, usually by module maintainers.
So, if you aren't sure your current usage is correct, please just leave the field empty.

Pretty sure this is a bug. I ran into this too, and have 14,938,677 rows in my search_api_task table. This has the very unfortunate side effect of really messing up node save performance. Any idea why this might be happening?

(Also, removing all indexes from the server and re-adding it is not really an option on a production site that takes forever to reindex)

Pretty sure this is a bug. I ran into this too, and have 14,938,677 rows in my search_api_task table. This has the very unfortunate side effect of really messing up node save performance. Any idea why this might be happening?

Depends on what kinds of operations those are. But, in general, some operation seems to be consistently failing, filling up this table. Do you maybe have an unreachable Solr server set up, or something? Try to look into your logs to see what the error might be.
In any case, depending on the kind of tasks, you can execute them manually (e.g., write a script that deletes the items in question from the server) and then truncate the search_api_task table.

I've similar problem (sorry if it's not the same). The indexing is stuck on Initializing when run from UI. I've increased limit of max_execution_time to 120, but it didn't help. It still processing a lot of data, making hundreds of requests, despite I've requested only 10 items to index.

I've used DTrace to see what the script is doing and half a minute of tracing generated millions of entries.

It seems like Search API keep trying to deleteItems() and keep SearchApiSolrService connect().

Here is my trace file: https://gist.github.com/kenorb/110d9e3f4b266f82c166

I'm using Solr 5.5.0, Drupal 7.39
My Solr doesn't have any indexed docs, so I'm not sure what Search API is trying to delete. It may attempt to remove some nodes which were on other instance and since I'm using dump from other server on my local on empty Solr instance, SAPI is confused.

Here is a dump after 120 second timeout which may be useful:

An AJAX HTTP error occurred. HTTP Result Code: 200

Fatal error:  Maximum execution time of 120 seconds exceeded in includes/common.inc on line 951
Stack trace:
  1. {main}() index.php:0
  2. menu_execute_active_handler($path = *uninitialized*, $deliver = *uninitialized*) index.php:21
  3. call_user_func_array:{includes/menu.inc:519}('system_batch_page', array ()) includes/menu.inc:519
  4. system_batch_page() includes/menu.inc:519
  5. _batch_page() modules/system/system.admin.inc:2379
  6. _batch_do() includes/batch.inc:80
  7. _batch_process() includes/batch.inc:161
  8. call_user_func_array:{includes/batch.inc:284}('_search_api_batch_indexing_callback', array (0 => class SearchApiIndex { protected $datasource = class SearchApiEntityDataSourceController { protected $entityInfo = array (...); protected $idKey = 'nid'; protected $bundleKey = 'type'; protected $bundles = array (...); protected $type = 'node'; protected $entityType = 'node'; protected $info = array (...); protected $table = 'search_api_item'; protected $itemIdColumn = 'item_id'; protected $indexIdColumn = 'index_id'; protected $changedColumn = 'changed' }; protected $server_object = class SearchApiServer { public $id = '19'; public $name = 'Local Solr Server'; public $machine_name = 'local_solr_server'; public $description = ''; public $class = 'search_api_solr_service'; public $options = array (...); public $enabled = '1'; protected $proxy = class SearchApiSolrService { ... }; protected $entityType = 'search_api_server'; protected $entityInfo = array (...); protected $idKey = 'id'; protected $nameKey = 'machine_name'; protected $statusKey = 'status'; protected $defaultLabel = FALSE; public $status = '3'; public $module = 'd7_search'; public $rdf_mapping = array (...) }; protected $callbacks = NULL; protected $processors = NULL; protected $added_properties = NULL; protected $fields = array (1 => array (...)); protected $fulltext_fields = array (); public $id = '10'; public $name = 'Site Index'; public $machine_name = 'site_index'; public $description = NULL; public $server = 'local_solr_server'; public $item_type = 'node'; public $options = array ('index_directly' => 1, 'cron_limit' => '1000', 'fields' => array (...), 'data_alter_callbacks' => array (...), 'processors' => array (...)); public $enabled = '1'; public $read_only = '0'; protected $entityType = 'search_api_index'; protected $entityInfo = array ('label' => 'Search index', 'controller class' => 'EntityAPIControllerExportable', 'metadata controller class' => FALSE, 'entity class' => 'SearchApiIndex', 'base table' => 'search_api_index', 'uri callback' => 'search_api_index_url', 'access callback' => 'search_api_entity_access', 'module' => 'search_api', 'exportable' => TRUE, 'entity keys' => array (...), 'fieldable' => FALSE, 'static cache' => TRUE, 'field cache' => TRUE, 'load hook' => 'search_api_index_load', 'bundles' => array (...), 'view modes' => array (...), 'translation' => array (...), 'base table field types' => array (...), 'schema_fields_sql' => array (...), 'token type' => 'search_api_index', 'configuration' => TRUE); protected $idKey = 'id'; protected $nameKey = 'machine_name'; protected $statusKey = 'status'; protected $defaultLabel = FALSE; public $status = '2'; public $module = 'd7_search'; public $rdf_mapping = array () }, 1 => 1000, 2 => 14795, 3 => FALSE, 4 => array ('sandbox' => array ('limit' => 14795, 'batch_size' => 1000, 'progress' => 0), 'results' => array ('indexed' => 0, 'not indexed' => 0, 'drush' => FALSE), 'finished' => 1, 'message' => ''))) includes/batch.inc:284
  9. _search_api_batch_indexing_callback($index = class SearchApiIndex { protected $datasource = class SearchApiEntityDataSourceController { protected $entityInfo = array ('path' => 'node/%node', 'metatags' => TRUE, 'field replacement' => array (...), 'efq bundle conditions' => TRUE, 'label' => 'Content', 'controller class' => 'NodeController', 'base table' => 'node', 'revision table' => 'node_revision', 'uri callback' => 'node_uri', 'fieldable' => TRUE, 'entity keys' => array (...), 'bundle keys' => array (...), 'bundles' => array (...), 'view modes' => array (...), 'translation' => array (...), 'static cache' => TRUE, 'field cache' => TRUE, 'load hook' => 'node_load', 'base table field types' => array (...), 'schema_fields_sql' => array (...), 'linkit' => TRUE, 'default path' => 'node/%node', 'token type' => 'node', 'bundle label' => 'Content type', 'xmlsitemap' => array (...), 'plural label' => 'Nodes', 'description' => 'Nodes represent the main site content items.', 'access callback' => 'entity_metadata_no_hook_node_access', 'creation callback' => 'entity_metadata_create_node', 'save callback' => 'node_save', 'deletion callback' => 'node_delete', 'revision deletion callback' => 'node_revision_delete', 'form callback' => 'entity_metadata_form_node', 'view callback' => 'entity_metadata_view_node', 'configuration' => FALSE, 'uuid' => TRUE, 'language callback' => 'entity_translation_language', 'label callback' => 'title_entity_label'); protected $idKey = 'nid'; protected $bundleKey = 'type'; protected $bundles = array (); protected $type = 'node'; protected $entityType = 'node'; protected $info = array ('name' => 'Content', 'datasource controller' => 'SearchApiEntityDataSourceController', 'entity_type' => 'node', 'module' => 'search_api'); protected $table = 'search_api_item'; protected $itemIdColumn = 'item_id'; protected $indexIdColumn = 'index_id'; protected $changedColumn = 'changed' }; protected $server_object = class SearchApiServer { public $id = '19'; public $name = 'Local Solr Server'; public $machine_name = 'local_solr_server'; public $description = ''; public $class = 'search_api_solr_service'; public $options = array ('clean_ids' => TRUE, 'site_hash_form' => array (...), 'site_hash' => FALSE, 'scheme' => 'http', 'host' => 'localhost', 'port' => '8983', 'path' => '/solr/d7', 'http_user' => '', 'http_pass' => '', 'excerpt' => 0, 'retrieve_data' => 0, 'highlight_data' => 0, 'skip_schema_check' => 0, 'solr_version' => '', 'http_method' => 'GET', 'log_query' => 1, 'log_response' => 0); public $enabled = '1'; protected $proxy = class SearchApiSolrService { protected $connection_class = 'SearchApiSolrConnection'; protected $solr = class SearchApiSolrConnection { ... }; protected $fieldNames = array (...); protected $fields = NULL; protected $commitScheduled = TRUE; protected $request_handler = NULL; protected $server = ...; protected $options = array (...) }; protected $entityType = 'search_api_server'; protected $entityInfo = array ('label' => 'Search server', 'controller class' => 'EntityAPIControllerExportable', 'metadata controller class' => FALSE, 'entity class' => 'SearchApiServer', 'base table' => 'search_api_server', 'uri callback' => 'search_api_server_url', 'access callback' => 'search_api_entity_access', 'module' => 'search_api', 'exportable' => TRUE, 'entity keys' => array (...), 'fieldable' => FALSE, 'static cache' => TRUE, 'field cache' => TRUE, 'load hook' => 'search_api_server_load', 'bundles' => array (...), 'view modes' => array (...), 'translation' => array (...), 'base table field types' => array (...), 'schema_fields_sql' => array (...), 'token type' => 'search_api_server', 'configuration' => TRUE); protected $idKey = 'id'; protected $nameKey = 'machine_name'; protected $statusKey = 'status'; protected $defaultLabel = FALSE; public $status = '3'; public $module = 'd7_search'; public $rdf_mapping = array () }; protected $callbacks = NULL; protected $processors = NULL; protected $added_properties = NULL; protected $fields = array (1 => array (0 => array (...))); protected $fulltext_fields = array (); public $id = '10'; public $name = 'Site Index'; public $machine_name = 'site_index'; public $description = NULL; public $server = 'local_solr_server'; public $item_type = 'node'; public $options = array ('index_directly' => 1, 'cron_limit' => '1000', 'fields' => array ('author' => array (...), 'body:summary' => array (...), 'body:value' => array (...), 'changed' => array (...), 'created' => array (...), 'field_article_tags' => array (...), 'field_careers_address_line_1' => array (...), 'field_careers_address_line_2' => array (...), 'field_careers_email' => array (...), 'field_careers_empl_cat' => array (...), 'field_careers_job_location' => array (...), 'field_careers_job_ref_no' => array (...), 'field_careers_job_title' => array (...), 'field_careers_last_name' => array (...), 'field_careers_region' => array (...), 'field_careers_telephone' => array (...), 'field_careers_type_of_job' => array (...), 'field_categories_downloads' => array (...), 'field_description' => array (...), 'field_location' => array (...), 'field_news_html_summary:value' => array (...), 'field_part_no_' => array (...), 'field_sc_address_line_1' => array (...), 'field_sc_address_line_2' => array (...), 'field_telephone' => array (...), 'field_town' => array (...), 'language' => array (...), 'promote' => array (...), 'search_api_language' => array (...), 'status' => array (...), 'sticky' => array (...), 'title' => array (...), 'type' => array (...)), 'data_alter_callbacks' => array ('search_api_alter_bundle_filter' => array (...), 'search_api_alter_node_access' => array (...), 'search_api_alter_node_status' => array (...), 'search_api_alter_language_control' => array (...), 'search_api_alter_add_viewed_entity' => array (...), 'search_api_alter_add_url' => array (...), 'search_api_alter_add_aggregation' => array (...), 'search_api_alter_add_hierarchy' => array (...)), 'processors' => array ('search_api_case_ignore' => array (...), 'content_taxonomy_autocomplete_moderated_terms' => array (...), 'search_api_html_filter' => array (...), 'search_api_transliteration' => array (...), 'search_api_tokenizer' => array (...), 'search_api_stopwords' => array (...))); public $enabled = '1'; public $read_only = '0'; protected $entityType = 'search_api_index'; protected $entityInfo = array ('label' => 'Search index', 'controller class' => 'EntityAPIControllerExportable', 'metadata controller class' => FALSE, 'entity class' => 'SearchApiIndex', 'base table' => 'search_api_index', 'uri callback' => 'search_api_index_url', 'access callback' => 'search_api_entity_access', 'module' => 'search_api', 'exportable' => TRUE, 'entity keys' => array ('id' => 'id', 'label' => 'name', 'name' => 'machine_name', 'revision' => '', 'bundle' => '', 'translations' => 'translations'), 'fieldable' => FALSE, 'static cache' => TRUE, 'field cache' => TRUE, 'load hook' => 'search_api_index_load', 'bundles' => array ('search_api_index' => array (...)), 'view modes' => array (), 'translation' => array ('entity_translation' => array (...)), 'base table field types' => array ('id' => 'serial', 'name' => 'varchar', 'machine_name' => 'varchar', 'description' => 'text', 'server' => 'varchar', 'item_type' => 'varchar', 'options' => 'text', 'enabled' => 'int', 'read_only' => 'int', 'status' => 'int', 'module' => 'varchar'), 'schema_fields_sql' => array ('base table' => array (...)), 'token type' => 'search_api_index', 'configuration' => TRUE); protected $idKey = 'id'; protected $nameKey = 'machine_name'; protected $statusKey = 'status'; protected $defaultLabel = FALSE; public $status = '2'; public $module = 'd7_search'; public $rdf_mapping = array () }, $batch_size = 1000, $limit = 14795, $drush = FALSE, $context = array ('sandbox' => array ('limit' => 14795, 'batch_size' => 1000, 'progress' => 0), 'results' => array ('indexed' => 0, 'not indexed' => 0, 'drush' => FALSE), 'finished' => 1, 'message' => '')) includes/batch.inc:284
 10. search_api_index_items($index = class SearchApiIndex { protected $datasource = class SearchApiEntityDataSourceController { protected $entityInfo = array ('path' => 'node/%node', 'metatags' => TRUE, 'field replacement' => array (...), 'efq bundle conditions' => TRUE, 'label' => 'Content', 'controller class' => 'NodeController', 'base table' => 'node', 'revision table' => 'node_revision', 'uri callback' => 'node_uri', 'fieldable' => TRUE, 'entity keys' => array (...), 'bundle keys' => array (...), 'bundles' => array (...), 'view modes' => array (...), 'translation' => array (...), 'static cache' => TRUE, 'field cache' => TRUE, 'load hook' => 'node_load', 'base table field types' => array (...), 'schema_fields_sql' => array (...), 'linkit' => TRUE, 'default path' => 'node/%node', 'token type' => 'node', 'bundle label' => 'Content type', 'xmlsitemap' => array (...), 'plural label' => 'Nodes', 'description' => 'Nodes represent the main site content items.', 'access callback' => 'entity_metadata_no_hook_node_access', 'creation callback' => 'entity_metadata_create_node', 'save callback' => 'node_save', 'deletion callback' => 'node_delete', 'revision deletion callback' => 'node_revision_delete', 'form callback' => 'entity_metadata_form_node', 'view callback' => 'entity_metadata_view_node', 'configuration' => FALSE, 'uuid' => TRUE, 'language callback' => 'entity_translation_language', 'label callback' => 'title_entity_label'); protected $idKey = 'nid'; protected $bundleKey = 'type'; protected $bundles = array (); protected $type = 'node'; protected $entityType = 'node'; protected $info = array ('name' => 'Content', 'datasource controller' => 'SearchApiEntityDataSourceController', 'entity_type' => 'node', 'module' => 'search_api'); protected $table = 'search_api_item'; protected $itemIdColumn = 'item_id'; protected $indexIdColumn = 'index_id'; protected $changedColumn = 'changed' }; protected $server_object = class SearchApiServer { public $id = '19'; public $name = 'Local Solr Server'; public $machine_name = 'local_solr_server'; public $description = ''; public $class = 'search_api_solr_service'; public $options = array ('clean_ids' => TRUE, 'site_hash_form' => array (...), 'site_hash' => FALSE, 'scheme' => 'http', 'host' => 'localhost', 'port' => '8983', 'path' => '/solr/d7', 'http_user' => '', 'http_pass' => '', 'excerpt' => 0, 'retrieve_data' => 0, 'highlight_data' => 0, 'skip_schema_check' => 0, 'solr_version' => '', 'http_method' => 'GET', 'log_query' => 1, 'log_response' => 0); public $enabled = '1'; protected $proxy = class SearchApiSolrService { protected $connection_class = 'SearchApiSolrConnection'; protected $solr = class SearchApiSolrConnection { ... }; protected $fieldNames = array (...); protected $fields = NULL; protected $commitScheduled = TRUE; protected $request_handler = NULL; protected $server = ...; protected $options = array (...) }; protected $entityType = 'search_api_server'; protected $entityInfo = array ('label' => 'Search server', 'controller class' => 'EntityAPIControllerExportable', 'metadata controller class' => FALSE, 'entity class' => 'SearchApiServer', 'base table' => 'search_api_server', 'uri callback' => 'search_api_server_url', 'access callback' => 'search_api_entity_access', 'module' => 'search_api', 'exportable' => TRUE, 'entity keys' => array (...), 'fieldable' => FALSE, 'static cache' => TRUE, 'field cache' => TRUE, 'load hook' => 'search_api_server_load', 'bundles' => array (...), 'view modes' => array (...), 'translation' => array (...), 'base table field types' => array (...), 'schema_fields_sql' => array (...), 'token type' => 'search_api_server', 'configuration' => TRUE); protected $idKey = 'id'; protected $nameKey = 'machine_name'; protected $statusKey = 'status'; protected $defaultLabel = FALSE; public $status = '3'; public $module = 'd7_search'; public $rdf_mapping = array () }; protected $callbacks = NULL; protected $processors = NULL; protected $added_properties = NULL; protected $fields = array (1 => array (0 => array (...))); protected $fulltext_fields = array (); public $id = '10'; public $name = 'Site Index'; public $machine_name = 'site_index'; public $description = NULL; public $server = 'local_solr_server'; public $item_type = 'node'; public $options = array ('index_directly' => 1, 'cron_limit' => '1000', 'fields' => array ('author' => array (...), 'body:summary' => array (...), 'body:value' => array (...), 'changed' => array (...), 'created' => array (...), 'field_article_tags' => array (...), 'field_careers_address_line_1' => array (...), 'field_careers_address_line_2' => array (...), 'field_careers_email' => array (...), 'field_careers_empl_cat' => array (...), 'field_careers_job_location' => array (...), 'field_careers_job_ref_no' => array (...), 'field_careers_job_title' => array (...), 'field_careers_last_name' => array (...), 'field_careers_region' => array (...), 'field_careers_telephone' => array (...), 'field_careers_type_of_job' => array (...), 'field_categories_downloads' => array (...), 'field_description' => array (...), 'field_location' => array (...), 'field_news_html_summary:value' => array (...), 'field_part_no_' => array (...), 'field_sc_address_line_1' => array (...), 'field_sc_address_line_2' => array (...), 'field_telephone' => array (...), 'field_town' => array (...), 'language' => array (...), 'promote' => array (...), 'search_api_language' => array (...), 'status' => array (...), 'sticky' => array (...), 'title' => array (...), 'type' => array (...)), 'data_alter_callbacks' => array ('search_api_alter_bundle_filter' => array (...), 'search_api_alter_node_access' => array (...), 'search_api_alter_node_status' => array (...), 'search_api_alter_language_control' => array (...), 'search_api_alter_add_viewed_entity' => array (...), 'search_api_alter_add_url' => array (...), 'search_api_alter_add_aggregation' => array (...), 'search_api_alter_add_hierarchy' => array (...)), 'processors' => array ('search_api_case_ignore' => array (...), 'content_taxonomy_autocomplete_moderated_terms' => array (...), 'search_api_html_filter' => array (...), 'search_api_transliteration' => array (...), 'search_api_tokenizer' => array (...), 'search_api_stopwords' => array (...))); public $enabled = '1'; public $read_only = '0'; protected $entityType = 'search_api_index'; protected $entityInfo = array ('label' => 'Search index', 'controller class' => 'EntityAPIControllerExportable', 'metadata controller class' => FALSE, 'entity class' => 'SearchApiIndex', 'base table' => 'search_api_index', 'uri callback' => 'search_api_index_url', 'access callback' => 'search_api_entity_access', 'module' => 'search_api', 'exportable' => TRUE, 'entity keys' => array ('id' => 'id', 'label' => 'name', 'name' => 'machine_name', 'revision' => '', 'bundle' => '', 'translations' => 'translations'), 'fieldable' => FALSE, 'static cache' => TRUE, 'field cache' => TRUE, 'load hook' => 'search_api_index_load', 'bundles' => array ('search_api_index' => array (...)), 'view modes' => array (), 'translation' => array ('entity_translation' => array (...)), 'base table field types' => array ('id' => 'serial', 'name' => 'varchar', 'machine_name' => 'varchar', 'description' => 'text', 'server' => 'varchar', 'item_type' => 'varchar', 'options' => 'text', 'enabled' => 'int', 'read_only' => 'int', 'status' => 'int', 'module' => 'varchar'), 'schema_fields_sql' => array ('base table' => array (...)), 'token type' => 'search_api_index', 'configuration' => TRUE); protected $idKey = 'id'; protected $nameKey = 'machine_name'; protected $statusKey = 'status'; protected $defaultLabel = FALSE; public $status = '2'; public $module = 'd7_search'; public $rdf_mapping = array () }, $limit = 1000) sites/all/modules/search_api/search_api.module:3137
 11. search_api_index_specific_items($index = class SearchApiIndex { protected $datasource = class SearchApiEntityDataSourceController { protected $entityInfo = array ('path' => 'node/%node', 'metatags' => TRUE, 'field replacement' => array (...), 'efq bundle conditions' => TRUE, 'label' => 'Content', 'controller class' => 'NodeController', 'base table' => 'node', 'revision table' => 'node_revision', 'uri callback' => 'node_uri', 'fieldable' => TRUE, 'entity keys' => array (...), 'bundle keys' => array (...), 'bundles' => array (...), 'view modes' => array (...), 'translation' => array (...), 'static cache' => TRUE, 'field cache' => TRUE, 'load hook' => 'node_load', 'base table field types' => array (...), 'schema_fields_sql' => array (...), 'linkit' => TRUE, 'default path' => 'node/%node', 'token type' => 'node', 'bundle label' => 'Content type', 'xmlsitemap' => array (...), 'plural label' => 'Nodes', 'description' => 'Nodes represent the main site content items.', 'access callback' => 'entity_metadata_no_hook_node_access', 'creation callback' => 'entity_metadata_create_node', 'save callback' => 'node_save', 'deletion callback' => 'node_delete', 'revision deletion callback' => 'node_revision_delete', 'form callback' => 'entity_metadata_form_node', 'view callback' => 'entity_metadata_view_node', 'configuration' => FALSE, 'uuid' => TRUE, 'language callback' => 'entity_translation_language', 'label callback' => 'title_entity_label'); protected $idKey = 'nid'; protected $bundleKey = 'type'; protected $bundles = array (); protected $type = 'node'; protected $entityType = 'node'; protected $info = array ('name' => 'Content', 'datasource controller' => 'SearchApiEntityDataSourceController', 'entity_type' => 'node', 'module' => 'search_api'); protected $table = 'search_api_item'; protected $itemIdColumn = 'item_id'; protected $indexIdColumn = 'index_id'; protected $changedColumn = 'changed' }; protected $server_object = class SearchApiServer { public $id = '19'; public $name = 'Local Solr Server'; public $machine_name = 'local_solr_server'; public $description = ''; public $class = 'search_api_solr_service'; public $options = array ('clean_ids' => TRUE, 'site_hash_form' => array (...), 'site_hash' => FALSE, 'scheme' => 'http', 'host' => 'localhost', 'port' => '8983', 'path' => '/solr/d7', 'http_user' => '', 'http_pass' => '', 'excerpt' => 0, 'retrieve_data' => 0, 'highlight_data' => 0, 'skip_schema_check' => 0, 'solr_version' => '', 'http_method' => 'GET', 'log_query' => 1, 'log_response' => 0); public $enabled = '1'; protected $proxy = class SearchApiSolrService { protected $connection_class = 'SearchApiSolrConnection'; protected $solr = class SearchApiSolrConnection { ... }; protected $fieldNames = array (...); protected $fields = NULL; protected $commitScheduled = TRUE; protected $request_handler = NULL; protected $server = ...; protected $options = array (...) }; protected $entityType = 'search_api_server'; protected $entityInfo = array ('label' => 'Search server', 'controller class' => 'EntityAPIControllerExportable', 'metadata controller class' => FALSE, 'entity class' => 'SearchApiServer', 'base table' => 'search_api_server', 'uri callback' => 'search_api_server_url', 'access callback' => 'search_api_entity_access', 'module' => 'search_api', 'exportable' => TRUE, 'entity keys' => array (...), 'fieldable' => FALSE, 'static cache' => TRUE, 'field cache' => TRUE, 'load hook' => 'search_api_server_load', 'bundles' => array (...), 'view modes' => array (...), 'translation' => array (...), 'base table field types' => array (...), 'schema_fields_sql' => array (...), 'token type' => 'search_api_server', 'configuration' => TRUE); protected $idKey = 'id'; protected $nameKey = 'machine_name'; protected $statusKey = 'status'; protected $defaultLabel = FALSE; public $status = '3'; public $module = 'd7_search'; public $rdf_mapping = array () }; protected $callbacks = NULL; protected $processors = NULL; protected $added_properties = NULL; protected $fields = array (1 => array (0 => array (...))); protected $fulltext_fields = array (); public $id = '10'; public $name = 'Site Index'; public $machine_name = 'site_index'; public $description = NULL; public $server = 'local_solr_server'; public $item_type = 'node'; public $options = array ('index_directly' => 1, 'cron_limit' => '1000', 'fields' => array ('author' => array (...), 'body:summary' => array (...), 'body:value' => array (...), 'changed' => array (...), 'created' => array (...), 'field_article_tags' => array (...), 'field_careers_address_line_1' => array (...), 'field_careers_address_line_2' => array (...), 'field_careers_email' => array (...), 'field_careers_empl_cat' => array (...), 'field_careers_job_location' => array (...), 'field_careers_job_ref_no' => array (...), 'field_careers_job_title' => array (...), 'field_careers_last_name' => array (...), 'field_careers_region' => array (...), 'field_careers_telephone' => array (...), 'field_careers_type_of_job' => array (...), 'field_categories_downloads' => array (...), 'field_description' => array (...), 'field_location' => array (...), 'field_news_html_summary:value' => array (...), 'field_part_no_' => array (...), 'field_sc_address_line_1' => array (...), 'field_sc_address_line_2' => array (...), 'field_telephone' => array (...), 'field_town' => array (...), 'language' => array (...), 'promote' => array (...), 'search_api_language' => array (...), 'status' => array (...), 'sticky' => array (...), 'title' => array (...), 'type' => array (...)), 'data_alter_callbacks' => array ('search_api_alter_bundle_filter' => array (...), 'search_api_alter_node_access' => array (...), 'search_api_alter_node_status' => array (...), 'search_api_alter_language_control' => array (...), 'search_api_alter_add_viewed_entity' => array (...), 'search_api_alter_add_url' => array (...), 'search_api_alter_add_aggregation' => array (...), 'search_api_alter_add_hierarchy' => array (...)), 'processors' => array ('search_api_case_ignore' => array (...), 'content_taxonomy_autocomplete_moderated_terms' => array (...), 'search_api_html_filter' => array (...), 'search_api_transliteration' => array (...), 'search_api_tokenizer' => array (...), 'search_api_stopwords' => array (...))); public $enabled = '1'; public $read_only = '0'; protected $entityType = 'search_api_index'; protected $entityInfo = array ('label' => 'Search index', 'controller class' => 'EntityAPIControllerExportable', 'metadata controller class' => FALSE, 'entity class' => 'SearchApiIndex', 'base table' => 'search_api_index', 'uri callback' => 'search_api_index_url', 'access callback' => 'search_api_entity_access', 'module' => 'search_api', 'exportable' => TRUE, 'entity keys' => array ('id' => 'id', 'label' => 'name', 'name' => 'machine_name', 'revision' => '', 'bundle' => '', 'translations' => 'translations'), 'fieldable' => FALSE, 'static cache' => TRUE, 'field cache' => TRUE, 'load hook' => 'search_api_index_load', 'bundles' => array ('search_api_index' => array (...)), 'view modes' => array (), 'translation' => array ('entity_translation' => array (...)), 'base table field types' => array ('id' => 'serial', 'name' => 'varchar', 'machine_name' => 'varchar', 'description' => 'text', 'server' => 'varchar', 'item_type' => 'varchar', 'options' => 'text', 'enabled' => 'int', 'read_only' => 'int', 'status' => 'int', 'module' => 'varchar'), 'schema_fields_sql' => array ('base table' => array (...)), 'token type' => 'search_api_index', 'configuration' => TRUE); protected $idKey = 'id'; protected $nameKey = 'machine_name'; protected $statusKey = 'status'; protected $defaultLabel = FALSE; public $status = '2'; public $module = 'd7_search'; public $rdf_mapping = array () }, $ids = array (0 => '47504', 1 => '47505', 2 => '47506', 3 => '47507', 4 => '47508', 5 => '47509', 6 => '47510', 7 => '47511', 8 => '47512', 9 => '47513', 10 => '47514', 11 => '47515', 12 => '47516', 13 => '47517', 14 => '47518', 15 => '47519', 16 => '47520', 17 => '47521', 18 => '47522', 19 => '47523', 20 => '47524', 21 => '47525', 22 => '47526', 23 => '47527', 24 => '47528', 25 => '47529', 26 => '47530', 27 => '47531', 28 => '47532', 29 => '47533', 30 => '47534', 31 => '47535', 32 => '47536', 33 => '47537', 34 => '47538', 35 => '47539', 36 => '47540', 37 => '47541', 38 => '47542', 39 => '47543', 40 => '47544', 41 => '47545', 42 => '47546', 43 => '47547', 44 => '47548', 45 => '47549', 46 => '47550', 47 => '47551', 48 => '47552', 49 => '47553', 50 => '47554', 51 => '47555', 52 => '47556', 53 => '47557', 54 => '47558', 55 => '47559', 56 => '47560', 57 => '47561', 58 => '47562', 59 => '47563', 60 => '47564', 61 => '47565', 62 => '47566', 63 => '47567', 64 => '47568', 65 => '47569', 66 => '47570', 67 => '47571', 68 => '47572', 69 => '47573', 70 => '47574', 71 => '47575', 72 => '47576', 73 => '47577', 74 => '47578', 75 => '47579', 76 => '47580', 77 => '47581', 78 => '47582', 79 => '47583', 80 => '47584', 81 => '47585', 82 => '47586', 83 => '47587', 84 => '47588', 85 => '47589', 86 => '47590', 87 => '47591', 88 => '47592', 89 => '47593', 90 => '47594', 91 => '47595', 92 => '47596', 93 => '47597', 94 => '47598', 95 => '47599', 96 => '47600', 97 => '47601', 98 => '47602', 99 => '47603', 100 => '47604', 101 => '47605', 102 => '47606', 103 => '47607', 104 => '47608', 105 => '47609', 106 => '47610', 107 => '47611', 108 => '47612', 109 => '47613', 110 => '47614', 111 => '47615', 112 => '47616', 113 => '47617', 114 => '47618', 115 => '47619', 116 => '47620', 117 => '47621', 118 => '47622', 119 => '47623', 120 => '47624', 121 => '47625', 122 => '47626', 123 => '47627', 124 => '47628', 125 => '47629', 126 => '47630', 127 => '47631', 128 => '47632', 129 => '47633', 130 => '47634', 131 => '47635', 132 => '47636', 133 => '47637', 134 => '47638', 135 => '47639', 136 => '47640', 137 => '47641', 138 => '47642', 139 => '47643', 140 => '47644', 141 => '47645', 142 => '47646', 143 => '47647', 144 => '47648', 145 => '47649', 146 => '47650', 147 => '47651', 148 => '47652', 149 => '47653', 150 => '47654', 151 => '47655', 152 => '47656', 153 => '47657', 154 => '47658', 155 => '47659', 156 => '47660', 157 => '47661', 158 => '47662', 159 => '47663', 160 => '47664', 161 => '47665', 162 => '47666', 163 => '47667', 164 => '47668', 165 => '47669', 166 => '47670', 167 => '47671', 168 => '47672', 169 => '47673', 170 => '47674', 171 => '47675', 172 => '47676', 173 => '47677', 174 => '47678', 175 => '47679', 176 => '47680', 177 => '47681', 178 => '47682', 179 => '47683', 180 => '47684', 181 => '47685', 182 => '47686', 183 => '47687', 184 => '47688', 185 => '47689', 186 => '47690', 187 => '47691', 188 => '47692', 189 => '47693', 190 => '47694', 191 => '47695', 192 => '47696', 193 => '47697', 194 => '47698', 195 => '47699', 196 => '47700', 197 => '47701', 198 => '47702', 199 => '47703', 200 => '47704', 201 => '47705', 202 => '47706', 203 => '47707', 204 => '47708', 205 => '47709', 206 => '47710', 207 => '47711', 208 => '47712', 209 => '47713', 210 => '47714', 211 => '47715', 212 => '47716', 213 => '47717', 214 => '47718', 215 => '47719', 216 => '47720', 217 => '47721', 218 => '47722', 219 => '47723', 220 => '47724', 221 => '47725', 222 => '47726', 223 => '47727', 224 => '47728', 225 => '47729', 226 => '47730', 227 => '47731', 228 => '47732', 229 => '47733', 230 => '47734', 231 => '47735', 232 => '47736', 233 => '47737', 234 => '47738', 235 => '47739', 236 => '47740', 237 => '47741', 238 => '47742', 239 => '47743', 240 => '47744', 241 => '47745', 242 => '47746', 243 => '47747', 244 => '47748', 245 => '47749', 246 => '47750', 247 => '47751', 248 => '47752', 249 => '47753', 250 => '47754', 251 => '47755', 252 => '47756', 253 => '47757', 254 => '47758', 255 => '47759', ...)) sites/all/modules/search_api/search_api.module:1544
 12. search_api_server_tasks_check($server = class SearchApiServer { public $id = '19'; public $name = 'Local Solr Server'; public $machine_name = 'local_solr_server'; public $description = ''; public $class = 'search_api_solr_service'; public $options = array ('clean_ids' => TRUE, 'site_hash_form' => array (), 'site_hash' => FALSE, 'scheme' => 'http', 'host' => 'localhost', 'port' => '8983', 'path' => '/solr/d7', 'http_user' => '', 'http_pass' => '', 'excerpt' => 0, 'retrieve_data' => 0, 'highlight_data' => 0, 'skip_schema_check' => 0, 'solr_version' => '', 'http_method' => 'GET', 'log_query' => 1, 'log_response' => 0); public $enabled = '1'; protected $proxy = class SearchApiSolrService { protected $connection_class = 'SearchApiSolrConnection'; protected $solr = class SearchApiSolrConnection { protected $options = array (...); protected $base_url = 'http://localhost:8983/solr/d7/'; protected $update_url = 'http://localhost:8983/solr/d7/update?wt=json'; protected $http_auth = NULL; protected $stream_context = NULL; protected $luke = array (...); protected $stats = NULL; protected $system_info = NULL; protected $soft_commit = TRUE }; protected $fieldNames = array (); protected $fields = NULL; protected $commitScheduled = TRUE; protected $request_handler = NULL; protected $server = ...; protected $options = array ('clean_ids' => TRUE, 'site_hash_form' => array (...), 'site_hash' => FALSE, 'scheme' => 'http', 'host' => 'localhost', 'port' => '8983', 'path' => '/solr/d7', 'http_user' => '', 'http_pass' => '', 'excerpt' => 0, 'retrieve_data' => 0, 'highlight_data' => 0, 'skip_schema_check' => 0, 'solr_version' => '', 'http_method' => 'GET', 'log_query' => 1, 'log_response' => 0) }; protected $entityType = 'search_api_server'; protected $entityInfo = array ('label' => 'Search server', 'controller class' => 'EntityAPIControllerExportable', 'metadata controller class' => FALSE, 'entity class' => 'SearchApiServer', 'base table' => 'search_api_server', 'uri callback' => 'search_api_server_url', 'access callback' => 'search_api_entity_access', 'module' => 'search_api', 'exportable' => TRUE, 'entity keys' => array ('id' => 'id', 'label' => 'name', 'name' => 'machine_name', 'revision' => '', 'bundle' => '', 'translations' => 'translations'), 'fieldable' => FALSE, 'static cache' => TRUE, 'field cache' => TRUE, 'load hook' => 'search_api_server_load', 'bundles' => array ('search_api_server' => array (...)), 'view modes' => array (), 'translation' => array ('entity_translation' => array (...)), 'base table field types' => array ('id' => 'serial', 'name' => 'varchar', 'machine_name' => 'varchar', 'description' => 'text', 'class' => 'varchar', 'options' => 'text', 'enabled' => 'int', 'status' => 'int', 'module' => 'varchar'), 'schema_fields_sql' => array ('base table' => array (...)), 'token type' => 'search_api_server', 'configuration' => TRUE); protected $idKey = 'id'; protected $nameKey = 'machine_name'; protected $statusKey = 'status'; protected $defaultLabel = FALSE; public $status = '3'; public $module = 'd7_search'; public $rdf_mapping = array () }) sites/all/modules/search_api/search_api.module:1567
 13. SearchApiServer->deleteItems($ids = 'all', $index = NULL) sites/all/modules/search_api/search_api.module:1373
 14. SearchApiSolrService->deleteItems($ids = 'all', $index = NULL) sites/all/modules/search_api/includes/server_entity.inc:346
 15. SearchApiSolrConnection->deleteByQuery($rawQuery = '*:*', $timeout = *uninitialized*) sites/all/modules/search_api_solr/includes/service.inc:829
 16. SearchApiSolrConnection->update($rawPost = '<delete><query>*:*</query></delete>', $timeout = 3600) sites/all/modules/search_api_solr/includes/solr_connection.inc:832
 17. SearchApiSolrConnection->sendRawPost($url = 'http://localhost:8983/solr/d7/update?wt=json', $options = array ('data' => '<delete><query>*:*</query></delete>', 'timeout' => 3600)) sites/all/modules/search_api_solr/includes/solr_connection.inc:771
 18. SearchApiSolrConnection->makeHttpRequest($url = 'http://localhost:8983/solr/d7/update?wt=json', $options = array ('data' => '<delete><query>*:*</query></delete>', 'timeout' => 3600, 'method' => 'POST', 'headers' => array ('Content-Type' => 'text/xml; charset=UTF-8'))) sites/all/modules/search_api_solr/includes/solr_connection.inc:609
 19. drupal_http_request($url = 'http://localhost:8983/solr/d7/update?wt=json', $options = array ('data' => '<delete><query>*:*</query></delete>', 'timeout' => 3600, 'method' => 'POST', 'headers' => array ('Content-Type' => 'text/xml; charset=UTF-8'))) sites/all/modules/search_api_solr/includes/solr_connection.inc:640
 20. fwrite(resource(140879) of type (stream), 'POST /solr/d7/update?wt=json HTTP/1.0\r\nContent-Type: text/xml; charset=UTF-8\r\nUser-Agent: Drupal (+http://drupal.org/)\r\nHost: localhost:8983\r\nContent-Length: 35\r\n\r\n<delete><query>*:*</query></delete>') includes/common.inc:951

Backtrace and local vars:

Most calls are from these functions below (~100.000 calls in 120 seconds), sorted by number of calls:

92870 PHP function-return: checkResponse() in sites/all/modules/search_api_solr/includes/solr_connection.inc:536
92870 PHP execute-entry in: sites/all/modules/search_api_solr/includes/solr_connection.inc:659
92869 PHP function-return: makeHttpRequest() in sites/all/modules/search_api_solr/includes/solr_connection.inc:628
92869 PHP execute-entry in: sites/all/modules/search_api_solr/includes/solr_connection.inc:658
92868 PHP function-entry:  SearchApiSolrConnection checkResponse() in sites/all/modules/search_api_solr/includes/solr_connection.inc:536
92868 PHP execute-entry in: sites/all/modules/search_api_solr/includes/solr_connection.inc:536
92865 PHP function-return: sendRawPost() in sites/all/modules/search_api_solr/includes/solr_connection.inc:603
92864 PHP function-return: update() in sites/all/modules/search_api_solr/includes/solr_connection.inc:761
92864 PHP function-return: scheduleCommit() in sites/all/modules/search_api_solr/includes/service.inc:2348
92864 PHP function-entry:  SearchApiSolrService scheduleCommit() in sites/all/modules/search_api_solr/includes/service.inc:2348
92864 PHP execute-entry in: sites/all/modules/search_api_solr/includes/service.inc:2348
92864 PHP execute-entry in: sites/all/modules/search_api/search_api.module:1367
92863 PHP function-return: deleteItems() in sites/all/modules/search_api_solr/includes/service.inc:804
92863 PHP function-return: deleteByQuery() in sites/all/modules/search_api_solr/includes/solr_connection.inc:829
92863 PHP execute-entry in: sites/all/modules/search_api_solr/includes/solr_connection.inc:761
92863 PHP execute-entry in: sites/all/modules/search_api_solr/includes/service.inc:804
92863 PHP execute-entry in: sites/all/modules/search_api_solr/includes/service.inc:500
92863 PHP execute-entry in: sites/all/modules/search_api/includes/server_entity.inc:127
92862 PHP function-return: ensureProxy() in sites/all/modules/search_api/includes/server_entity.inc:127
92862 PHP function-return: deleteItems() in sites/all/modules/search_api/includes/server_entity.inc:343
92862 PHP function-entry:  SearchApiSolrService deleteItems() in sites/all/modules/search_api_solr/includes/service.inc:804
92862 PHP function-entry:  SearchApiSolrService connect() in sites/all/modules/search_api_solr/includes/service.inc:500
92862 PHP function-entry:  SearchApiSolrConnection update() in sites/all/modules/search_api_solr/includes/solr_connection.inc:761
92862 PHP function-entry:  SearchApiSolrConnection sendRawPost() in sites/all/modules/search_api_solr/includes/solr_connection.inc:603
92862 PHP function-entry:  SearchApiSolrConnection makeHttpRequest() in sites/all/modules/search_api_solr/includes/solr_connection.inc:628
92862 PHP function-entry:  SearchApiServer ensureProxy() in sites/all/modules/search_api/includes/server_entity.inc:127
92862 PHP function-entry:  SearchApiServer deleteItems() in sites/all/modules/search_api/includes/server_entity.inc:343
92862 PHP execute-entry in: sites/all/modules/search_api_solr/includes/solr_connection.inc:628
92862 PHP execute-entry in: sites/all/modules/search_api_solr/includes/solr_connection.inc:603
92862 PHP execute-entry in: sites/all/modules/search_api/includes/server_entity.inc:343
92861 PHP function-return: connect() in sites/all/modules/search_api_solr/includes/service.inc:500
92860 PHP function-entry:  SearchApiSolrConnection deleteByQuery() in sites/all/modules/search_api_solr/includes/solr_connection.inc:829
92860 PHP execute-entry in: sites/all/modules/search_api_solr/includes/solr_connection.inc:830
92860 PHP execute-entry in: sites/all/modules/search_api_solr/includes/solr_connection.inc:829
92860 PHP execute-entry in: sites/all/modules/search_api_solr/includes/service.inc:806

I've found out that in search_api_task table I've a lot of tasks to delete the items, not sure how these records get there:

`search_api_item` VALUES (47513,10,1458497284);
`search_api_task` VALUES (47513,'local_solr_server','deleteItems',NULL,'s:3:\"all\";');
`search_api_task` VALUES (147513,'local_solr_server','deleteItems',NULL,'s:3:\"all\";');
`search_api_task` VALUES (247513,'local_solr_server','deleteItems',NULL,'s:3:\"all\";');
`search_api_task` VALUES (347513,'local_solr_server','deleteItems',NULL,'s:3:\"all\";');
`search_api_task` VALUES (447513,'local_solr_server','deleteItems',NULL,'s:3:\"all\";');
`search_api_task` VALUES (475130,'local_solr_server','deleteItems',NULL,'s:3:\"all\";');
`search_api_task` VALUES (475131,'local_solr_server','deleteItems',NULL,'s:3:\"all\";');
`search_api_task` VALUES (475132,'local_solr_server','deleteItems',NULL,'s:3:\"all\";');
`search_api_task` VALUES (475133,'local_solr_server','deleteItems',NULL,'s:3:\"all\";');
`search_api_task` VALUES (475134,'local_solr_server','deleteItems',NULL,'s:3:\"all\";');
`search_api_task` VALUES (475135,'local_solr_server','deleteItems',NULL,'s:3:\"all\";');
`search_api_task` VALUES (475136,'local_solr_server','deleteItems',NULL,'s:3:\"all\";');
`search_api_task` VALUES (475137,'local_solr_server','deleteItems',NULL,'s:3:\"all\";');
`search_api_task` VALUES (475138,'local_solr_server','deleteItems',NULL,'s:3:\"all\";');
`search_api_task` VALUES (475139,'local_solr_server','deleteItems',NULL,'s:3:\"all\";');
`search_api_task` VALUES (547513,'local_solr_server','deleteItems',NULL,'s:3:\"all\";');
`search_api_task` VALUES (647513,'local_solr_server','deleteItems',NULL,'s:3:\"all\";');
`search_api_task` VALUES (747513,'local_solr_server','deleteItems',NULL,'s:3:\"all\";');
`search_api_task` VALUES (847513,'local_solr_server','deleteItems',NULL,'s:3:\"all\";');
`search_api_task` VALUES (947513,'local_solr_server','deleteItems',NULL,'s:3:\"all\";');
`search_api_task` VALUES (1047513,'local_solr_server','deleteItems',NULL,'s:3:\"all\";');
`search_api_task` VALUES (1147513,'local_solr_server','deleteItems',NULL,'s:3:\"all\";');
`search_api_task` VALUES (1247513,'local_solr_server','deleteItems',NULL,'s:3:\"all\";');
`search_api_task` VALUES (1347513,'local_solr_server','deleteItems',NULL,'s:3:\"all\";');
`search_api_task` VALUES (1447513,'local_solr_server','deleteItems',NULL,'s:3:\"all\";');
`search_api_task` VALUES (1475130,'local_solr_server','deleteItems',NULL,'s:3:\"all\";');
`search_api_task` VALUES (1475131,'local_solr_server','deleteItems',NULL,'s:3:\"all\";');
`search_api_task` VALUES (1475132,'local_solr_server','deleteItems',NULL,'s:3:\"all\";');
`search_api_task` VALUES (1475133,'local_solr_server','deleteItems',NULL,'s:3:\"all\";');
`search_api_task` VALUES (1475134,'local_solr_server','deleteItems',NULL,'s:3:\"all\";');
`search_api_task` VALUES (1475135,'local_solr_server','deleteItems',NULL,'s:3:\"all\";');
`search_api_task` VALUES (1475136,'local_solr_server','deleteItems',NULL,'s:3:\"all\";');
`search_api_task` VALUES (1475137,'local_solr_server','deleteItems',NULL,'s:3:\"all\";');
`search_api_task` VALUES (1475138,'local_solr_server','deleteItems',NULL,'s:3:\"all\";');
`search_api_task` VALUES (1475139,'local_solr_server','deleteItems',NULL,'s:3:\"all\";');
`search_api_task` VALUES (1547513,'local_solr_server','deleteItems',NULL,'s:3:\"all\";');
`search_api_task` VALUES (1647513,'local_solr_server','deleteItems',NULL,'s:3:\"all\";');
`search_api_task` VALUES (1747513,'local_solr_server','deleteItems',NULL,'s:3:\"all\";');
`search_api_task` VALUES (1847513,'local_solr_server','deleteItems',NULL,'s:3:\"all\";');
`search_api_task` VALUES (1947513,'local_solr_server','deleteItems',NULL,'s:3:\"all\";');

$ drush sqlq "SELECT COUNT(*) FROM search_api_task"
5654261

5mln items to remove?;/

Workaround:

drush sqlq "SELECT COUNT(*) FROM search_api_task"

The batch worked! But how this happened, I don't know.

Btw. Shouldn't search_api_task table be processed on cron run, not on site indexing?

I've implemented this workaround:

/**
 * Truncate search_api_task table in order to clean old tasks Search API tasks.
 */
function MYMODULE_update_7101() {
  db_truncate('search_api_task');
}

so it'll clear the SAPI tasks during the deployment, but still no idea if that will happen again.

Btw. Shouldn't search_api_task table be processed on cron run, not on site indexing?

The problem is that the "tasks" stored in that table should have already been executed, and we can't really know whether their order is relevant. So, doing any other operation before the pending tasks have been finished might result in erroneous behavior. (Think of trying to index before addIndex() was called.)

So, unfortunately I'm quite sure that the code is correct there, even though it can lead to problems, as we see here.
As to why there are so many tasks, I can't really say. But, confusingly, in your case these aren't tasks for deleting specific items, but all items from the server. Why it tried to do that 5 M times is beyond me.
Maybe try to add a debugging line to the code where it creates the task and log a backtrace, or something?
Also, see my other comments in this issue: disabled servers are evil and should be avoided.

Just to add my case to this, for me it was this exactly:

Do you maybe have an unreachable Solr server set up, or something?

Solr stopped working properly after a Java update, and I ended up with 2.5 million entries in my search_api_task table. This ended up prevented reindexing via the admin, drush, and cron.php, as the server would run out of memory trying to process the tasks.

It took awhile to debug, and I didn't find this thread until after I'd tracked down the issue; while that's on me, I do wonder if some sort of status message on the View Server page indicating that there are a very large number of entries in the search_api_task table could help. Because PHP was either failing with a fatal error or having its tasks killed for pushing up against the limit of the VPS's resources, I wasn't finding useful information in logs.

Thanks for all your work on this! My experience with Solr via the Search API has otherwise been a positive one.

Same as gregsullivan the SOLR server wasn't reachable and end up inserting millions of rows in the table "search_api_task" and ends up using all of my cpu on the mysql daemon.
That ends up slowing down the login process at a point we had to wait more that a minute to be able to log in...
Even after reconfiguring the solr to be reachable it was still slow, not as bad though.
I had to truncate the search_api_task table and it worked much faster

I think we've solved this a lot better now in Drupal 8, but some improvements would surely also be possible in Drupal 7.
However, since I'm currently fully occupied (in my contrib time) with the D8 port, I probably won't be able to work on this in the foreseeable future. But I'd be happy to give pointers, if someone else wants to write a patch.

@drunken_monkey: Pointers please! :) Perhaps you could summarize how this was improved in D8 to guide us coding any D7 improvements.

Been hit by this too: Solr unavailable for a couple of days on a site indexing 146297 taxonomy terms led to 19393509 items in search_api_task. Causes fatal memory time outs on every page, as the index was set to update immediately.

Truncating table and reindexing has allowed the site to survive, but would be good to work out a way to prevent this kind of behaviour.

@drunken_monkey: Pointers please! :) Perhaps you could summarize how this was improved in D8 to guide us coding any D7 improvements.

In Drupal 8, we never just execute all the tasks (at least outside of tests), but instead just to a maximum of 100 tasks – see \Drupal\search_api\Task\ServerTaskManager::execute(). If there are more than 100 tasks, this will always report failure to execute the tasks, but will still execute and remove 100 of them – so, steadily, all tasks will be gone.

There is also the possibility on the server's "View" tab to execute all pending tasks for the server at once, in a batch (\Drupal\search_api\Form\ServerStatusForm::buildForm() and \Drupal\search_api\Task\TaskManager::setTasksBatch()). This also gives users a chance to do something about large numbers of tasks.

However, for over a million tasks, this might still not be enough. So, maybe, additionally we should provide a button (both in D7 and D8) to just delete all tasks? With a strict warning that this might mess up the search?

I hope this helps!
One last note: If you're looking at the D8 code, you have to be aware that we extended the tasks system a bit there, so that there aren't just server tasks, but also index tasks and tasks for individual datasources (and anything else that might want to create tasks). So the corresponding D7 code would probably be a bit simpler.

Is there any kind of checking going on about duplication in the tasks? The table was too big for me to query or export so I couldn't tell, but my instinct tells me there weren't 19m unique activities: most seemed to be deletions, and I suspect at least some might have been duplicated when the processing found that Solr was unavailable.

Is there any kind of checking going on about duplication in the tasks? The table was too big for me to query or export so I couldn't tell, but my instinct tells me there weren't 19m unique activities: most seemed to be deletions, and I suspect at least some might have been duplicated when the processing found that Solr was unavailable.

In theory, we only create one task for such an event (e.g., item deletion), and delete all tasks which we attempted to execute. So even if the deletion also fails during task execution, and thus generates another task, the old task should be removed, and no duplication should occur.

However, once there are too many tasks and the function executing tasks runs out of time or memory, new tasks for ones whose execution failed will still be created, but the old ones not removed (since that would only happen at the end of the function – which will never be reached, since PHP runs out of time/memory before that) – so, yes, then you'd have duplicates. Furthermore, even if tasks are successfully executed, they wouldn't be removed from the pending tasks anymore (but no new ones are created for them in this case).

So, thinking about that, I guess we'll have to categorize this as a bug after all.

@ janusman: Any progress with a patch?
I'm still too caught up in D8 for the moment, but if you bump this issue after the Search API D8 version becomes stable, I can also work on the patch. (Will still take a few months until then, though.)

Hi all,

I'm also having this issue, do you have any pointers for the back port?

Thank you

Julian

The attached patch should at least implement the 100 tasks limit, which will help to avoid crashing anything. Indexing will still not work while there are tasks remaining, but at least they'll get fewer over time, not more.

The code for having an "Execute tasks" local action is mostly in \Drupal\search_api\Task\TaskManager::setTasksBatch() (and the methods mentioned there).
For D7, the simplest process batch callback would probably be just calling search_api_server_tasks_check() untill it returns TRUE. (Would not have a progress indicator unless you manually check the task count between each call, but should otherwise work well enough.)

Then you'll just need to add a local action through search_api_menu() – either one directly on the overview, or for each server individually (could also be a button, in the latter case, to keep with the UI pattern we already use) – that comes with an access check that only shows it if there are actual tasks to be executed.
Bonus would be also porting the "pending tasks" check in search_api_requirements(), to notify admins that there are tasks they could execute.

#20 Worked for me. Was experiencing out of memory issue (even with a 1GB limit).

Great to hear, thanks a lot for reporting back!
Committed the patch. Back to "Active" for the batch.

#20 worked for us as well. We were seeing a large number of remaining entries in search_api_task after indexing. Thanks Drunken Monkey =).

Initial attempt at backporting the d8 functionality of exposing a method to execute pending tasks for a given server. Additionally, providing warnings on the status report page if pending tasks exist. Kept with the current UI pattern over a local action / task (server page).

Thanks a lot for taking this on, looks great already!
This revision also adds an option for executing pending tasks for all servers, makes the batch a bit more robust and prevents tasks from being executed on a disabled server, plus a few style changes.

Please test/review!

If you want to add a Drush command, too, go ahead! Does sound like a good idea.
Otherwise, though, we can just remove that @todo and create a follow-up issue.
In theory, I don't see any problem with running this via Drush.

Drush command for executing pending tasks with an optional server argument added. Let me know if you're after an interdiff.

Great job, thanks!
Yes, an interdiff would have been nice, since you not only changed the Drush file, but since lots of people are lazy that way, I already have a script setup which just takes me five extra seconds. So no big problem.

Just three remarks regarding the code:

+++ b/search_api.module
@@ -1476,6 +1492,112 @@ function search_api_server_tasks_check(SearchApiServer $server = NULL) {
+ * @param Booelan $drush

It's bool, not Booelan, for type hints.

+++ b/search_api.drush.inc
@@ -467,6 +480,30 @@ function drush_search_api_clear($index_id = NULL) {
+  if ($server_id && $servers = search_api_drush_get_server($server_id)) {
+    $server = reset($servers);
+  }

I think if an invalid server was passed we should just abort, not execute for all servers (as this seems to be doing).

+++ b/search_api.module
@@ -1476,6 +1492,112 @@ function search_api_server_tasks_check(SearchApiServer $server = NULL) {
+  if ($drush) {
+    drush_backend_batch_process();

We can just use function_exists() here and avoid the extra param.

Revised patch attached, please tell me if it still works for you!

Oops …

Could maybe someone review and confirm this works for them? Then we could commit this.

Ahh I'm all over the place atm; sorry I didn't get a chance to test until today. Works wonderfully for us! Thanks heaps Drunken Monkey =).

Alright, thanks a lot for testing!
Committed.