Index all queued content only indexes as many items as 1 cron run would do

This is a MAJOR bug.

Symptom:

If there are more *updated* items in the apache solr indexing queue than will be indexed by one cron run, the "index all queued content" option will only index the number of items that will be indexed by one cron run.

How to reproduce:

- You have indexed 100 documents from Drupal.
- Your Apachesolr settings say you should index 50 items per cron run.
- Force an update of the apache solr index by setting the apachesolr_index_entities_node.changed column beyond the last update for 90 of your already indexed items.
- The Apachesolr status page will now say 90 items remain for indexing.
- If you attempt to "index all queued content", only 50 items are actually sent to Solr.
- The other 40 items will never be reindexed.

Why does this happen?

The update algorithm for selecting items only uses last entity_id or last changed date (apachesolr_index_get_entities_to_index). After the equivalent of one cron run, the last changed date is changed to the date of the last indexed item (apachesolr_index_entities). If several entries share the same timestamp (not at all unthinkable in big custom bulk operations) you risk not getting your data indexed and you have no error messages to tell the story.

Proposted solution

I propose adding a dirty-bit column to the apachesolr_index_entities_node table, named "pending". If set to 1, update, if set to 0, leave alone. The function apachesolr_index_entities could then run a bulk db_update on all rows that were successfully indexed, setting the dirty bit to 0.

You lose efficiency with the db_update, but gain efficiency in the function that selects items to reindex.

I have myself implemented a custom solution (1 changed and 3 added lines of code) that doesn't demand changes in the database or API. It's much less efficient and not worthy of publication. I can mail it to anyone interested, though :)

Remi