Indexing with „Index items immediately“ indexes old entity state

Well, of course the item will have to be re-indexed first (usually on the next cron run) for this to take effect. But then, it should work as expected.
In case it is a security requirement that this is updated immediately, enabling the index's „Index immediately“ option should work (though it will slow your performance for a lot of nodes/changes).

Also, #1182912: Rules action for indexing entities and/or action for marking them as needing indexing would allow you to only index the item immediately if it is unpublished, and otherwise wait for cron.

Hm, OK, then this seems to be some kind of bug, or usage problem. This should definitely work.
Have you checked whether waiting for a few minutes (more than 120 seconds, Solr's autocommit delay) after unpublishing helps? Then maybe just the committing doesn't work.
Otherwise, please check whether search_api_entity_update() and search_api_track_item_change() are properly called when publishing/unpublishing the node,and maybe try to debug what happens then.
In principle, items should always be marked as "dirty" (needing to be re-indexed) when they are edited, and (un)publishing of course counts, too.

@drunken_monkey, I'm having the same problem.

Here's my workflow. Nodes are created non-published and then are purchased via paypal. Once purchased, they are published.

My problem is, after they're published, they're not showing up on my site.

I debug'd some of the functions in question. Here's my conclusion. If you could tell me exactly how to mark these are "dirty" i might be able to hack everything together to get it working.

Here's the trace:

search_api_entity_update() is getting called as expected upon an update.

search_api_track_item_change() is getting called, but it's not flagging (in the database) `search_api_item.changed` as changed.

Reason is:

    // ...

    else {                                                                                               
      // For indexes with the index_directly option set, index the items right
      // away.
      $indexed = array();
      try {
        $indexed = search_api_index_specific_items($index, $item_ids);
        // JL: This finds the node as expected, it's already in the index. 
      }
      catch (SearchApiException $e) {
        watchdog('search_api', $e->getMessage(), NULL, WATCHDOG_ERROR);
      }
      dpm(array("UPDATING INDEX" => $indexed, "items" => $item_ids, "type" => $type));
      // JL: Here's the problem
      // JL: $indexed has the NID in there. $item_ids has the NID in there. The counts are the same
      // JL: This code is skipped, not changing the dirty flag in the database I assume
      if (count($indexed) < count($item_ids)) {                                                     
        // If indexing failed for some items, mark those as dirty.
        $diff = array_diff($item_ids, $indexed); 
        search_api_get_datasource_controller($type)->trackItemChange($diff, array($index));
      }
    }
    // ...

So I changed the code to this:

      // ...
      if (1) { 
        // If indexing failed for some items, mark those as dirty.
        search_api_get_datasource_controller($type)->trackItemChange($item_ids, array($index)); 
      }
      // ...                    

This marks the appropriate items as dirty in the database.

But then you run cron, and while the row in `search_api_item` is no longer set to "dirty", it still doesn't show up on my site. Something else is happening.

This code is only executed when the index's „Index immediately“ option is set. And if this is the case, then we of course shouldn't mark the items as changed, as they have already been indexed again in their latest form.

Ah, OK, I think I've found the bug here! When „Index items immediately“ is enabled, items are immediately indexed during the hook_entity_update() invocation – when the entity's new version isn't yet saved! From my understanding I would have thought that entity_load(), because of its internal static caching, would already return the changed version, but it's evident that this is not the case (maybe some recent change in core I'm not aware of?). Therefore, while the item is seemingly correctly indexed, what is indexed is really only its old version.

It's weird I didn't realize this before, as I have the option always enabled on my testing sites, but it's easily reproducable. And therefore a much more urgent problem than initially thought, as this also affects access checks. Upgrading priority accordingly.

OK, as for how to fix this … There is, regrettably, no hook_entity_postsave() (or similar), so I guess we don't have many other options than using drupal_register_shutdown_function() to execute some function at the end of the page request which really indexes the changed items.

Any other ideas or comments?

entity_load with $reset = TRUE?

I've run into this problem with Drupal Commerce. I believe it happened after some entity locking code was added (don't quote me on this).

Here's some related issue queues perhaps

http://drupal.org/node/1060002#comment-4084810
http://drupal.org/node/1030140

You might want to ask @rszrama about this, as he's done a lot of work with EntityAPI and may have a solution.

Also maybe this is relative? From a recent Drupal Commerce release:

We've implemented our first step toward "pessimistic entity locking." This only governs orders right now and will prevent an order from being loaded and saved asynchronously, which could have led to data loss. Developers interacting with the cart line item API directly may encounter db_transaction() errors that have been resolved in Drupal core. However, you must update to Drupal 7.x-dev to get the fix, as it was committed after Drupal 7.7. Once Drupal 7.8 is out, this should no longer be an issue.

Quote from here: http://drupal.org/node/1245834

Relates to this issue: http://drupal.org/node/1243306

Sounds similar? Not exactly too sure what was changed in Drupal 7.8. Maybe you need to look into some locking.

I hope I'm pointing in the right direction :)

There's also this bug I've run into where entity_revisions are getting added into their _revision table with the initial revision_id as 0...and then after, it gets updated.

http://drupal.org/node/1212482

entity_load with $reset = TRUE?

Do you think this might be the problem or the solution?
Currently, we use FALSE and I think that should be right in this case. A quick test showed that changing this doesn't help, in any case.

And no, this doesn't have to do with transactions, and the others seem also unrelated. In the end, we just can't index the item during hook_entity_update() – even if we could work around this issue: with other modules maybe modifying the item afterwards we'd have no guarantee that we really index the data that gets saved.

So unless we get some other idea, we'll have to delegate that to a shutdown function (and hope this doesn't lead to icky results when redirecting directly to a search view …).

(Also, please use proper linking when referencing issues: [#NID] for issues, [#NID-COMMENTNR] for comments (where COMMENTNR is the number in the issue, starting at 1, not the cid).)

Strange. The modified entity should be in static cache? Are you using entitycache, and could it be causing problems like this?

Yup, strange. And no, no additional entity cache. Just try it out, it should be reproducable for you, too. (If it isn't, we might get an idea of what changed there.)

Could you perhaps pass the $entity from hook_entity_update to your function in stead of passing the id and then loading it?

That's what we previously did. However, since #1064884: Add support for indexing non-entities was committed, this got abstracted so we now have to pass the ID. Changing this would be possible, I guess, but would be an unnecessary API change.

Furthermore, as said, now that I think about it this was broken to begin with. We have to index what gets stored, not what we get at one point in the update hook. Remembering the update and indexing the item later (then also using $reset = TRUE to be on the safe side, I guess) is, in the absence of a postsave hook, really the only option I can see to do this reliably.

I'm using Flag and Rules to create a toggle to publish and unpublish content. It's interesting that the Views filter for the Search API field for node status is doing opposites. I toggle the node to be unpublished it does unpublish as it should but the view shows the unpublished node in a search list. If I toggle the node to be published it works as it should except it no longer shows in the Search API view list. However, if I click the node edit tab and save with the publish check box unmarked as it was after toggling the node to be unpublished the Search API view list no longer will show that node.

My guess is that, with Rules triggering an action like unpublish, the node is being indexed by Search API before it's being updated in the database.

EDIT: It works fine when the Index Immediately check box is turned off in the index settings.

My guess is that, with Rules triggering an action like unpublish, the node is being indexed by Search API before it's being updated in the database.

As explained, it will happen no matter how the node is edited (at least as far as I can tell).

Any quick and dirty patch I can do to get this working. I have a site I need to get live and it doesn't really work if my nodes don't show up after purchased.

I don't mind wiping the entire cache as this is a low traffic site and I'm really just using Search API for the facets.

If you don't need the „Index items immediately“ functionality, just deactivate it and things should work fine. (You'll have to re-index the data now, of course, to be sure no stale data remains there.)
Otherwise you could do some hack to pass the entity from hook_entity_update() to search_api_index_specific_items(), or just circumvent the whole thing by indexing the entity right from hook_entity_update().
The proper patch shouldn't be too hard either: registering a shutdown function, saving the items (index + item ID) to index in some static cache, indexing the items in the shutdown function. You could also just try to code that.
Also, as I'm determined to do a stable release as soon as possible, and this is one of the blockers, you can probably rely on me fixing this in the next week or so.

The attached patch works for me. Please test and review!

Or do you think we should additionally pass $reset = TRUE when loading the entities for indexing?

Is patch #20 going to be committed to dev?

I've applied it but it didn't resolve my problem. I ended up $index->reindex(-1);'ing my content after every save to get the items to show up.

Is patch #20 going to be committed to dev?

If people review/test it and are OK with it, then yes. Just try it out and see if it works for you, to help.

I've applied it but it didn't resolve my problem. I ended up $index->reindex(-1);'ing my content after every save to get the items to show up.

OK, that's another workaround I guess.
Still, weird that this didn't fix the problem, I don't really know what could go wrong then.
Does just refreshing maybe help, indicating that they are just indexed after some displayed list is already built?
Otherwise, could you please check whether the indexing is triggered, and what entity version (old or new one) is used there?

Update: I actually looked into my code and it appeared the patch was not actually applied.

I applied it along with the Views Relationship & Entity API patches and now things are working as expected.

Thank you.

OK, then I guess we can commit this.
Yay, another release blocker resolved!