Allow bundle setting to be changed on existing indexes

I tried the same but is not working for me, after adding new bundle and saving i cannot see any results on searches using views search.

Attaching a quick reroll because for some reason the patch posted in the original issue is not working with drush make for me; I don't see any particular difference, though... just trying a rerolled patch.

Ah, it was because -dev and 7.x-1.16 have diverged and the patch in the OP doesn't apply to 1.16 :)

Carry on!

Hiding the patches that I uploaded; the latter of which will patch 7.x-1.16.

Btw, I was working on an index where I wanted to add one more bundle, and using this patch and adding the bundle, then either queueing all content for reindexing, or dropping the index and reindexing still resulted in an index without that bundle present. Before I had 175 items in the index (there are 186 total nodes including the bundle that wasn't in the index), and after it still only indexed 175 items.

It looks like there might be something more to adding a bundle (or removing one).

So, it was just a matter of needing to do the following steps after applying the patch and changing the bundle settings:

1. Disable the changed index: drush sapi-dis [index_machine_name]
2. Enable the changed index: drush sapi-en [index_machine_name]
3. Clear the changed index: drush sapi-c [index_machine_name]
4. Reindex the changed index: drush sapi-i [index_machine_name]

Simply clearing and reindexing (or queuing all content for reindexing) didn't work. I also had to disable/re-enable the index.

So, it was just a matter of needing to do the following steps after applying the patch and changing the bundle settings:

Yes, that's what I thought. I don't know why it worked for the OP, but from my understanding I think there's more to it. Internally, Search API has a table of all items for all indexes, and nodes of excluded bundles won't even be in there – re-indexing can't help you there.

You can see in the Drupal 8 version, though, that this is indeed possible, you just have to properly react to changes for existing indexes – i.e., find out which bundles got added or removed and then add/remove the entities from those bundles to/from the search_api_item table as appropriate. The only reason this wasn't done in Drupal 7 is because I didn't have enough time and this was still better than what we had before. But if someone provides a patch that properly implements this (optimally even with some tests), I'd gladly review and commit it.
However, as it is, I don't think it's usable enough.

We're running Solr, and I can confirm that we had to follow the steps described by @geerlingguy in #8.

Removing #disabled allowed us to add the bundle, but there were no results after reindexing. The only way we were able to see the new bundle in search results is by disabling/re-enabling the changed index, and then clearing and reindexing again.

The steps outlined by @geerlingguy also worked for me. In addition to re-indexing, I also had to re-enable a search page that was disabled by disabling my index.

@drunken monkey, is it possible to accept the patch if it will contain the instruction of rebuilding the index properly?

@drunken monkey, is it possible to accept the patch if it will contain the instruction of rebuilding the index properly?

No, that's not enough, sorry.
When manual rebuilding is required, you might as well just delete the index and create a new one with the correct settings.

Oh. It will require a ton of settings to be re-clicked.

I'd have to agree with @maximpodorov. We have hundreds of settings in the index. Deleting and creating a new one is not realistic. I'm very glad we were able to get #8 to work, even if it was a manual process.

You can do it programmatically:

1. Load index.
2. Delete it.
3. Mark it as new.
4. Change the datasource settings.
5. Save it again.

So, this can be done upon submitting the form with index settings, if bundle set is modified, right? Will you accept such a patch? :)

So, this can be done upon submitting the form with index settings, if bundle set is modified, right? Will you accept such a patch? :)

I'd be glad to review and commit such a patch, yes!
And as said, you can just look at the D8 version of the datasource (that code will be moved soon, though, and made a bit more complicated, so better use that version I linked to) for how to implement this.

Here's the D8 code, before it disappears:

  public function setConfiguration(array $new_config) {
    $old_config = $this->getConfiguration();
    $new_config += $this->defaultConfiguration();
    $this->configuration = $new_config;

    // We'd now check the items of which bundles need to be added to or removed
    // from tracking for this index. However, if the index is not tracking at
    // all (i.e., is disabled) currently, we can skip all that.
    if (!$this->index->status() || !$this->index->hasValidTracker()) {
      return;
    }

    if ($this->hasBundles() && $old_config != $new_config) {
      $old_bundles = array_flip($old_config['bundles']['selected']);
      $new_bundles = array_flip($new_config['bundles']['selected']);

      // First, check if the "default" setting changed and invert the set
      // bundles for the old config, so the following comparison makes sense.
      if ($old_config['bundles']['default'] != $new_config['bundles']['default']) {
        $old_bundles = array_diff_key($this->getBundles(), $old_bundles);
      }

      // Now, go through all the bundles and start/stop tracking for them
      // accordingly.
      $newly_selected = array_diff_key($new_bundles, $old_bundles);
      $newly_unselected = array_diff_key($old_bundles, $new_bundles);
      if ($new_config['bundles']['default']) {
        $actions['trackItemsDeleted'] = $newly_selected;
        $actions['trackItemsInserted'] = $newly_unselected;
      }
      else {
        $actions['trackItemsDeleted'] = $newly_unselected;
        $actions['trackItemsInserted'] = $newly_selected;
      }

      $actions = array_filter($actions);
      foreach ($actions as $action => $bundles) {
        if ($entity_ids = $this->getBundleItemIds(array_keys($bundles))) {
          $this->getIndex()->$action($this->getPluginId(), $entity_ids);
        }
      }
    }
  }

So it seems the logic would need to go in SearchApiEntityDataSourceController::configurationFormSubmit(), right?

So, making some headway on it, but I'm not sure about the actions loop - the D8 code has references to several methods that don't exist in the D7 codebase.

Good start, thanks! Please see my additional corrections. (Not sure about the array_diff_key() change, but I think the bundles should be in the values in this case, not in the keys. See the beginning of the method.)

Still to do:
- Implement the special case of an empty selection (needs to be interpreted as "all"). We could also set the field to "required" and default to all, but that would mess up existing sites, so I don't want to do that.
- Implement getBundleItemIds() – should be easy, though, just base it on the code in startTrackingFallback(). In D8, we also had to take the languages into account, but luckily we don't have that problem here. (Unlucky for people with multilingual sites, but way easier to maintain.)
- Add a test to ensure this actually works as intended.

Would the following help?
- Change the text to "This setting cannot be changed for enabled indexes."
- Make the checkboxes editable if the index is disabled.

This would force people to disable the index before they change these settings.

Would the following help?
- Change the text to "This setting cannot be changed for enabled indexes."
- Make the checkboxes editable if the index is disabled.

Oh, yes, that would be a great interim solution. Very easy to implement, and would still do the job. Not very well, of course, but still undoubtedly an improvement. Thanks a lot!

The attached patch would implement this, please verify it works for you and I can commit it!
Thanks again!

Thanks @drunken monkey, tested the patch from #25 and it worked without any problems.

Great, thanks for testing!
Committed.

Leaving at "Needs work" for the other, better patch from #23.

Thanks @drunken monkey, tested the patch from #25 and it worked

Posting this here as this is the main article that comes up when searching around this subject. It's also highly relevant. This will allow you to modify the datasource and re-index everything in one nice db update.

function my_module_update_7105(&$sandbox) {
  $index = search_api_index_load('my_index_machine_name');
  $datasource = $index->datasource();

  // First pass, clear out and update the existing index.
  if (!isset($sandbox['total'])) {
    // Clear all existing tracking items.
    $datasource->stopTracking(array($index));

    // Update the index datasource bundles. This is mirrored in Features but
    // needs to be done before revert.
    $index->options['datasource']['bundles'] = array(
      'article',
      'blog',
      'case_study',
      'structure',
      'news',
    );
    $index->save();

    // Start tracking all items using our new bundle settings.
    $datasource = $index->datasource();
    $datasource->startTracking(array($index));

    // Clear remote index data.
    $index->clear();

    // Fetch the total items that need to be tracked.
    $sandbox['total'] = count($datasource->getChangedItems($index));
    $sandbox['current'] = 0;
  }

  // Main pass, re-index 250 at a time.
  search_api_index_items($index, 250);
  $sandbox['current']++;

  $items = $datasource->getChangedItems($index);

  $sandbox['#finished'] = count($items) <= 0 ? TRUE : FALSE;

  if ($sandbox['#finished']) {
    return t('Sucessfully re-indexed the main site index.');
  }
}

A note here for D8 users: it's true that you can change the bundles that are indexed. However, if you deploy this change by importing the changed configuration to another environment then the records in the search_api_item table do not get correctly updated. I'm not yet sufficiently familiar with D8 to offer opinions as to the best way to solve this problem.