Node revision status can easily become out of sync with moderation history status.

We've been getting reports of content not becoming published correctly from our users.
Node's would appear to workbench moderation as published, but they would appear to core as un-published (a pink background on node pages in many themes)

I found a hand-full of problem nodes by running the following query:
"all nodes where workbench current status = 1 but the node_revision status is 0 for that vid"

SELECT * FROM workbench_moderation_node_history 
  INNER JOIN node ON node.vid = workbench_moderation_node_history.vid 
  INNER JOIN node_revision ON node_revision.vid = workbench_moderation_node_history.vid 
  WHERE
  workbench_moderation_node_history.published = 1 AND 
  workbench_moderation_node_history.current = 1 AND 
  node_revision.status = 0;

Nodes can get in to this state because (in our case) whenever a node is published, it gets indexed by apache solr - and whilst this happens successfully 99% of the time, sometimes it would bomb out and cause a timeout, leaving the node in this messed up state where node_revsion.status = 0 for all revisions, while workbench_moderation_node_history at the current node would have a status of 1.

This can happen because of how workbench moderation performs it's final node publish node_save().
It happens in workbench_moderation_store (during request shutdown), and if this is never called, then the node_revision table has had it's status set to 0 for all revisions of the node in workbench_moderation_moderate. This is done with a db_update query:

// If this revision is to be published, the new moderation record should be
  // the only one flagged 'published' in both
  // {workbench_moderation_node_history} AND {node_revision}
  if ($new_revision->published) {
    $query = db_update('workbench_moderation_node_history')
      ->condition('nid', $node->nid)
      ->fields(array('published' => 0))
      ->execute();
    $query = db_update('node_revision')
      ->condition('nid', $node->nid)
      ->fields(array('status' => 0))
      ->execute();
  }

We'll fix the problem with our apache solr instance - and make sure it requests time out before the php process can timeout, but there are any number of other ways this could happen (This problem is also touched on in #1966630 - where a watchog call was causing the request to end before the shutdown phase).

Does anybody have any ideas about how to make this more robust against this type of failure?

We're currently dealing with the issue on a case-by-case basis and would rather not implement some kind of automatic bandage solution unless we really have no other options.