I was wondering if there has been any documentation or if others have found ways to index and search(view) external content. Currently I have external content indexed into Solr via the data import handler and have mimic my drupal content (field names etc). But I am failing to see any external content being displayed in a search view. The count of results accurately displays the both the internal and external content but the display only shows internal documents. I believe its being thrown out because the search api is trying to do something to verify that all the content types were loaded cause I a seeing a warning from the below code:

loadItemsMultiple(array $item_ids) {
    // Group the requested items by datasource. This will also later be used to
    // determine whether all items were loaded successfully.
    $items_by_datasource = array();
    foreach ($item_ids as $item_id) {
      list($datasource_id, $raw_id) = Utility::splitCombinedId($item_id);
      $items_by_datasource[$datasource_id][$raw_id] = $item_id;
    }

    // Load the items from the datasources and keep track of which were
    // successfully retrieved.
    $items = array();
    foreach ($items_by_datasource as $datasource_id => $raw_ids) {
      try {
        $datasource = $this->getDatasource($datasource_id);
        $datasource_items = $datasource->loadMultiple(array_keys($raw_ids));
        foreach ($datasource_items as $raw_id => $item) {
          $id = $raw_ids[$raw_id];
          $items[$id] = $item;
          // Remember that we successfully loaded this item.
          unset($items_by_datasource[$datasource_id][$raw_id]);
        }
      }
      catch (SearchApiException $e) {
        watchdog_exception('search_api', $e);
        // If the complete datasource could not be loaded, don't report all its
        // individual requested items as missing.
        unset($items_by_datasource[$datasource_id]);
      }
    }

    // Check whether there are requested items that couldn't be loaded.
    $items_by_datasource = array_filter($items_by_datasource);
    if ($items_by_datasource) {
      // Extract the second-level values of the two-dimensional array (that is,
      // the combined item IDs) and log a warning reporting their absence.
      $missing_ids = array_reduce(array_map('array_values', $items_by_datasource), 'array_merge', array());
      $args['%index'] = $this->label();
      $args['@items'] = '"' . implode('", "', $missing_ids) . '"';
      \Drupal::service('logger.channel.search_api')
        ->warning('Could not load the following items on index %index: @items.', $args);
      // Also remove those items from tracking so we don't keep trying to load
      // them.
//      foreach ($items_by_datasource as $datasource_id => $raw_ids) {
//        $this->trackItemsDeleted($datasource_id, array_keys($raw_ids));
//      }
    }

    // Return the loaded items.
    return $items;
  }

Any help or tips on this would be great. The only things I was able to find are the following, which I am not sure are valid anymore:
https://www.drupal.org/node/2717589
https://www.drupal.org/node/2682347

Comments

ndrake86 created an issue. See original summary.

ndrake86’s picture

So far, this is what I have in hopes this might help some else start and continue building an external datasource plugin for the search API. The properties are specific for what I was accessing but I do think they can be generalized. The problem I am currently having is the search API tries to load the original entity which of course doesn't exist in any drupal known datasource. Extending the ComplexDataType and setting the originalEntity would was the recommendation I found but I am little stuck on how to do that successfully.

code so far

<?php

use Drupal\Core\TypedData\ComplexDataInterface;
use Drupal\Core\TypedData\DataDefinition;
use Drupal\search_api\Datasource\DatasourcePluginBase;

/**
 * Datasource plugin for external content.
 *
 * @SearchApiDatasource(
 *   id = "external_data",
 *   label = @Translation("External Data Source"),
 *   description = @Translation("Loads external data to the drupal search index.")
 * )
 */
class ExternalDataSource extends DatasourcePluginBase {

  /**
   * {@inheritdoc}
   */
  public function getItemId(ComplexDataInterface $item) {
    return $item->getValue();
  }

  /**
   * {@inheritdoc}
   */
  public function getPropertyDefinitions() {
    $fields = $this->getFieldDefinitions();
    $properties = [];
    foreach ($fields as $field_id => $field_definition) {
      $properties[$field_id] = new DataDefinition($field_definition);
    }
    return $properties;
  }

  /**
   * Get all fields for the external data source.
   *
   * @return array
   *   Array of field definitions.
   */
  public function getFieldDefinitions() {
    return [
      'book_id' => [
        'label' => 'book_id',
        'description' => 'FT Book Id.',
        'type' => 'string',
        'prefix' => 's',
      ],
      'book_title' => [
        'label' => 'Book title',
        'description' => 'Title of the book',
        'type' => 'string',
      ],
      'book_content' => [
        'label' => 'Books content',
        'description' => 'Content about the book',
        'type' => 'string',
      ],
      'book_title_string' => [
        'label' => 'Book title string',
        'description' => 'String version of the book',
        'type' => 'string',
      ],
    ];
  }

  /**
   * {@inheritdoc}
   */
  public function load($id) {
    $items = $this->loadMultiple(array($id));
    return $items ? reset($items) : NULL;
  }

  /**
   * {@inheritdoc}
   */
  public function loadMultiple(array $ids) {
    return parent::loadMultiple($ids);
  }

  /**
   * {@inheritdoc}
   */
  public function getItemLabel(ComplexDataInterface $item) {
    return NULL;
  }

  /**
   * {@inheritdoc}
   */
  public function getItemLanguage(ComplexDataInterface $item) {
    return 'en';
  }

  /**
   * {@inheritdoc}
   */
  public function getItemBundle(ComplexDataInterface $item) {
    return $this->getPluginId();
  }

  /**
   * {@inheritdoc}
   */
  public function getItemUrl(ComplexDataInterface $item) {
    return NULL;
  }

  /**
   * {@inheritdoc}
   */
  public function getViewModes($bundle = NULL) {
    return array();
  }

  /**
   * {@inheritdoc}
   */
  public function getBundles() {
    return array(
      $this->getPluginId() => $this->label(),
    );
  }

  /**
   * {@inheritdoc}
   */
  public function viewItem(ComplexDataInterface $item, $view_mode, $langcode = NULL) {
    return array();
  }

  /**
   * {@inheritdoc}
   */
  public function viewMultipleItems(array $items, $view_mode, $langcode = NULL) {
    $build = array();
    foreach ($items as $key => $item) {
      $build[$key] = $this->viewItem($item, $view_mode, $langcode);
    }
    return $build;
  }

}
ndrake86’s picture

Category: Support request » Feature request
ndrake86’s picture

This is the source from which I started. Its rather old but is how I got this started -> https://www.drupal.org/node/2701729

ndrake86’s picture

drunken monkey’s picture

Maybe create your own class implementing ComplexDataInterface, wrap your results data in that and then set it as the original item in the Search API result items? That should do it.
Just providing all necessary fields' values in the result items should also work, but there might also be bugs regarding that.

ndrake86’s picture

@drunken monkey thanks for the reply, I had a feeling (based on some of the other similar issues) that extending the complexDataType class would be a potential solution. I did tried the second suggestion providing all the fields in the data set but it would always fail at getOriginalObject call or some reference to a search API specific field. There is also an issue with access checking that I am not sure how I can solve. Mainly wanted to comingle both external and internal (Drupal made) content. Which based on what I am seeing for access checks would also need to be taken care of. I am trying to work through extending the complexDataType but have been running into issues there. Continuing to work through. Again thanks for the suggestions, hopefully I can get this working at least in my specific case then can throw it back out to become more generalized.

cato’s picture

So @ndrake86 did you get it working? I'm doing something similar where we want to query an external index.

dcam’s picture

I've been working on a Solr datasource module for the past several weeks.* The sandbox project can be found on my GitHub repo. This project should be considered pre-alpha, unstable, and unsecure. Please only set it up in a development environment.

I have recently been successful in displaying external Solr data in Drupal using this module. Differences in server configurations and Search API index configurations may cause issues with displaying your data. Information about setting up the module, warnings about using it, and a list of known issues are all included in the README file.

*I didn't find this issue when I looked for info about datasource creation. It is gratifying to see @drunken_monkey recommend the same procedure for developing it that I figured out on my own.

mkalkbrenner’s picture

Status: Active » Fixed

I mark this issue as fixed to clean up the queue.
BTW I would love to see dcam's module as a sandbox project on drupal.org.

dcam’s picture

mkalkbrenner’s picture

great!

drunken monkey’s picture

Status: Fixed » Needs work

Feel free to mark as "fixed" again if you're sure, but I think this issue should remain open, with the goal of eventually getting this functionality built-in as part of this module (or at least as a sub-module in this project).

mkalkbrenner’s picture

https://www.drupal.org/node/2618414#comment-12052047
It seems like dcam development goes in the right direction. I like the idea to have that additional module for that use-case.

mkalkbrenner’s picture

Version: 8.x-1.x-dev » 8.x-2.x-dev
Component: Code » &amp;quot;Any Schema&amp;quot; backend
Status: Needs work » Fixed

Status: Fixed » Closed (fixed)

Automatically closed - issue fixed for 2 weeks with no activity.